Ninja Traders...lets Build a Better Mousetrap

[darthtrader2.0 10]

This thread is inspired by david22's post about neoticker in another thread and building internals for non NYSE markets...

I know this is neoticker's game and this is my last ditch attempt before I just buy neoticker...

I blew away one of my machines the other day and played around with the neoticker demo today but I just don't like the program.

So the first question is, has anyone here ever messed around with multi instrument strategy plots in Ninja? I know currently it is possible to construct some form of the tick client side from the datafeed with onmarketdata() and strategyplot() but I do not know if Ninja/our machines/memory can handle creating the SPX in real time..I'm pretty sure not...

NT7 is promised to have multiple instrument indicators so I'm on hold with neoticker until I see what that can do.

How many instruments that will support in real time is the key...While I do not expect to be able to calculate the S&P cash market from the individual stocks(even though this would rule because it would gives us a real time, tick precise PREM) there should be a work around as far as pruning the list down from 500.

A simple correlation between the move of security X and the S&P cash over a good sample size should do the trick there with a cutoff point, I suspect the list is much smaller than what we probly think.

The other reason then I want to stick with ninja is its not that hard to override plot()..OHLC and candles are simply a terrible way to visualize the data of TICK...we already know the bounds of the data, a distribution at N time makes alot more sense.

[ryker 10]

This thread is inspired by david22's post about neoticker in another thread and building internals for non NYSE markets...

I know this is neoticker's game and this is my last ditch attempt before I just buy neoticker...

I blew away one of my machines the other day and played around with the neoticker demo today but I just don't like the program.

So the first question is, has anyone here ever messed around with multi instrument strategy plots in Ninja? I know currently it is possible to construct some form of the tick client side from the datafeed with onmarketdata() and strategyplot() but I do not know if Ninja/our machines/memory can handle creating the SPX in real time..I'm pretty sure not...

NT7 is promised to have multiple instrument indicators so I'm on hold with neoticker until I see what that can do.

How many instruments that will support in real time is the key...While I do not expect to be able to calculate the S&P cash market from the individual stocks(even though this would rule because it would gives us a real time, tick precise PREM) there should be a work around as far as pruning the list down from 500.

A simple correlation between the move of security X and the S&P cash over a good sample size should do the trick there with a cutoff point, I suspect the list is much smaller than what we probly think.

The other reason then I want to stick with ninja is its not that hard to override plot()..OHLC and candles are simply a terrible way to visualize the data of TICK...we already know the bounds of the data, a distribution at N time makes alot more sense.

 

Hi darth,

Using the strategyplot to display markets internals is quite easy to do and also using it to calculate and display some indicators based on a few instruments is quite easy but if the list is too big NT won't handle it (I can't say what is the limit though...).

I think you're right by saying that we could bring down the number but it would require some quantitative analysis which would be probably better to do outside NT (you can plug NT to a statistical software).

 

Do you use some statistical software?

[darthtrader3.0beta 10]

Hi darth,

Using the strategyplot to display markets internals is quite easy to do and also using it to calculate and display some indicators based on a few instruments is quite easy but if the list is too big NT won't handle it (I can't say what is the limit though...).

I think you're right by saying that we could bring down the number but it would require some quantitative analysis which would be probably better to do outside NT (you can plug NT to a statistical software).

 

Do you use some statistical software?

 

Hey ryker,

Can you define what is too big as far as ninja goes here as far as your experience? I've mostly been waiting to see what they do in 7 as far as the the multi instrument indicator stuff...i'm sure that won't be much different than in strategies...

If its 100 vs 500 that is cool..if its 500 vs 3...not so good...

Statistical software wise I've messed around with R but never found anything that wasn't easier to do with Excel...Excel is probly robust enough, certainly for this kind of thing..

Output to the Ninja window with commas is kind of a pain for large datasets to save/paste/import into excel but certainly impossible and easier than any other solution I've found at my level of programming....

[ryker 10]

I've never done any testing in NT regarding what is the maximum number of instruments we can add in the strategy. But it all depends on your actual setup and also on your computer.

 

I'll run some tests this week by adding a few instruments in a blank strategy and then running some simple calculations on them.

 

As for NT7, I don't think the multi instrument indicator would be too much different thant the multi instrument strategy but NT7 will support multiple cores so if you have a powerful CPU you should be able to have a bigger list than in NT6.5.

 

I saw NeoBreath and it looks quite cool, I don't think that NT7 would have something similar and the multi instrument indicator won't be probably as quick as NeoBreath (although I'd like to be wrong here ).

[darthtrader3.0beta 10]

Oh I had not heard that that NT7 will support multiple cores..thats fantastic...

I'm going to try to build TIKI over the thanksgiving holiday. I guess the problem with that is I'm sure ninja can handle 30 custom series like that if its just doing addition and subtraction. I would think at some point memory becomes a bigger issue than actual CPU useage but I could be totally off there.

As far as neobreadth, isn't the the ease of that have to do with the fact that neoticker handles the list of single equities behind the scenes?

We could potentially do something like that pulling from a text file with a for loop in Initialize():

 

for (blah blah)

Add("X", PeriodType.Minute, 1);

[darthtrader3.0beta 10]

Well I messed around with this on the weekend and just loaded it up.

Bad news is my calculations make no sense and the plot is bogus but the good news is Ninja seems to be handling 30 manually added instruments like cakewalk..

In intialize I just did:

Add("AA", PeriodType.Minute, 1);

Add("AXP", PeriodType.Minute, 1);

Add("BA", PeriodType.Minute, 1);

Add("BAC", PeriodType.Minute, 1);

ect, ect..

 

then in onbarupdate (set to calculate on close false)

for(x = 0;x < 29; x++)

{

if(Closes[0][x] > LastTransactedPrice)

{Direction = 1;}

 

ect ect

So 30 instruments work and a for loop works, this should be doable. I couldn't figure out how to do this by overriding onmarketdata() but hopefully this will be good enough. Once I get the logic right I'll see if I can crash it with the S&P100.

[BlowFish 10]

Does ninja generate a bar update event if any data series changes or just if the first one does?

[darthtrader3.0beta 10]

Does ninja generate a bar update event if any data series changes or just if the first one does?

 

Well I think I just got TIKI going just now and spitting out the DOW stock data to the ninja output window its updating all the data series extremely fast. I'm not sure if its on the first dataseries or on all them though, I just loaded up SPY so that even if it needs a tick from the main dataseries to update, SPY is more than fast enough.

This is certainly better than the 5 second lag I get on DTN.

Going to drink some coffee and grunt through adding the S&P 100 right now.

 

Bump: Lame...well it appears Ninja has a 50 instrument limit as fas as add() goes.

Above this the strategy loads but doesn't switch on.

very disappointing considering how smoothly its handling 50 instruments.

[atto 20]

Interesting.. what if you ran like 3 strategies.. two "collect" the stock data, and one totals them?

[darthtrader3.0beta 10]

Well nevermind, the ninja guys said that its probly an issue with my dataprovider in that I can only use 50 streams at once.

This might be worth looking into as on my average computer, 50 instruments is only using 25% of the cpu.

 

That 3 strategies idea is interesting but I don't think ninja can communicate between strategies currently? Either way though I guess this is a function of the dataprovider and computer, there is no limit on the software..

If anyone wants to mess with this here is the code for the S&P 100..

 

protected override void Initialize()

{

 

Add("AA", PeriodType.Minute, 1);

Add("AAPL", PeriodType.Minute, 1);

Add("ABT", PeriodType.Minute, 1);

Add("AEP", PeriodType.Minute, 1);

Add("AES", PeriodType.Minute, 1);

Add("AIG", PeriodType.Minute, 1);

Add("ALL", PeriodType.Minute, 1);

Add("AMGN", PeriodType.Minute, 1);

Add("AVP", PeriodType.Minute, 1);

Add("AXP", PeriodType.Minute, 1);

Add("BA", PeriodType.Minute, 1);

 

Add("BAC", PeriodType.Minute, 1);

Add("BAX", PeriodType.Minute, 1);

Add("BHI", PeriodType.Minute, 1);

Add("BK", PeriodType.Minute, 1);

Add("BMY", PeriodType.Minute, 1);

Add("BNI", PeriodType.Minute, 1);

Add("C", PeriodType.Minute, 1);

Add("CAT", PeriodType.Minute, 1);

Add("CBS", PeriodType.Minute, 1);

Add("CI", PeriodType.Minute, 1);

 

Add("CL", PeriodType.Minute, 1);

Add("CMCSA", PeriodType.Minute, 1);

Add("COF", PeriodType.Minute, 1);

Add("COP", PeriodType.Minute, 1);

Add("COV", PeriodType.Minute, 1);

Add("CPB", PeriodType.Minute, 1);

Add("CSCO", PeriodType.Minute, 1);

Add("CVS", PeriodType.Minute, 1);

Add("CVX", PeriodType.Minute, 1);

Add("DD", PeriodType.Minute, 1);

 

Add("DELL", PeriodType.Minute, 1);

Add("DIS", PeriodType.Minute, 1);

Add("DOW", PeriodType.Minute, 1);

Add("EMC", PeriodType.Minute, 1);

Add("EP", PeriodType.Minute, 1);

Add("ETR", PeriodType.Minute, 1);

Add("EXC", PeriodType.Minute, 1);

Add("F", PeriodType.Minute, 1);

Add("FDX", PeriodType.Minute, 1);

Add("GD", PeriodType.Minute, 1);

 

Add("GE", PeriodType.Minute, 1);

Add("GOOG", PeriodType.Minute, 1);

Add("GS", PeriodType.Minute, 1);

Add("HAL", PeriodType.Minute, 1);

Add("HD", PeriodType.Minute, 1);

Add("HIG", PeriodType.Minute, 1);

Add("HNZ", PeriodType.Minute, 1);

Add("HON", PeriodType.Minute, 1);

Add("HPQ", PeriodType.Minute, 1);

Add("IBM", PeriodType.Minute, 1);

 

Add("INTC", PeriodType.Minute, 1);

Add("IP", PeriodType.Minute, 1);

Add("JNJ", PeriodType.Minute, 1);

Add("JPM", PeriodType.Minute, 1);

Add("KFT", PeriodType.Minute, 1);

Add("KO", PeriodType.Minute, 1);

Add("MA", PeriodType.Minute, 1);

Add("MCD", PeriodType.Minute, 1);

Add("MDT", PeriodType.Minute, 1);

Add("MER", PeriodType.Minute, 1);

 

Add("MMM", PeriodType.Minute, 1);

Add("MO", PeriodType.Minute, 1);

Add("MRK", PeriodType.Minute, 1);

Add("MS", PeriodType.Minute, 1);

Add("MSFT", PeriodType.Minute, 1);

Add("NOV", PeriodType.Minute, 1);

Add("NSC", PeriodType.Minute, 1);

Add("NYX", PeriodType.Minute, 1);

Add("ORCL", PeriodType.Minute, 1);

Add("OXY", PeriodType.Minute, 1);

 

Add("PEP", PeriodType.Minute, 1);

Add("PFE", PeriodType.Minute, 1);

Add("PG", PeriodType.Minute, 1);

Add("PM", PeriodType.Minute, 1);

Add("QCOM", PeriodType.Minute, 1);

Add("RF", PeriodType.Minute, 1);

Add("RTN", PeriodType.Minute, 1);

Add("S", PeriodType.Minute, 1);

Add("SLB", PeriodType.Minute, 1);

Add("SLE", PeriodType.Minute, 1);

 

Add("SO", PeriodType.Minute, 1);

Add("T", PeriodType.Minute, 1);

Add("TGT", PeriodType.Minute, 1);

Add("TWX", PeriodType.Minute, 1);

Add("TXN", PeriodType.Minute, 1);

Add("TYC", PeriodType.Minute, 1);

Add("UNH", PeriodType.Minute, 1);

Add("UPS", PeriodType.Minute, 1);

Add("USB", PeriodType.Minute, 1);

Add("UTX", PeriodType.Minute, 1);

 

Add("VZ", PeriodType.Minute, 1);

Add("WB", PeriodType.Minute, 1);

Add("WFC", PeriodType.Minute, 1);

Add("WMB", PeriodType.Minute, 1);

Add("WMT", PeriodType.Minute, 1);

Add("WY", PeriodType.Minute, 1);

Add("WYE", PeriodType.Minute, 1);

Add("XOM", PeriodType.Minute, 1);

Add("XRX", PeriodType.Minute, 1);

 

 

 

Total = 0;

Add(StrategyPlot(0));

StrategyPlot(0).Plots[0].Pen.Color = Color.Blue;

StrategyPlot(0).PanelUI = 2;

CalculateOnBarClose = false;

}

protected override void OnBarUpdate()

{

//

 

if (BarsInProgress != 0)

return;

Total = 0;

for(x = 1;x < 50; x++)

{

// Print(Closes[x][0]);

if(Closes[x][0] > Closes[x][1])

{Total++;}

if(Closes[x][0] < Closes[x][1])

{Total--;}

 

}

StrategyPlot(0).Value.Set(Total);

}

Edited December 1, 2008 by darthtrader3.0beta

[BlowFish 10]

its updating all the data series extremely fast. I'm not sure if its on the first dataseries or on all them though,

 

My thinking was that if it generates an event on every stream then you could get rid of the loop and just update the stream (subtract old value and add new value) of the series that generated the event. Hope I am makin sense just woke up from my afternoon nap

[darthtrader3.0beta 10]

My thinking was that if it generates an event on every stream then you could get rid of the loop and just update the stream (subtract old value and add new value) of the series that generated the event. Hope I am makin sense just woke up from my afternoon nap

 

Hmm I'm not totally sure what you mean..Ninja is a bit confusing on this stuff because if calculateonbarclose is true, then close() is like the close of a candle stick...if its false, then close() is just an incoming tick..but then if you want to know if the tick transacted at the bid or the ask you have to override onmarketdata()..I dont think what I did is truely tick precise for each of the stock dataseries because that loop is only firing on each tick of the primary instrument, but like I said on how heavy SPY is, this should be more than enough speed.

All the loop is doing in what I posted is marching through the added dataseries and looking to see if the last tick of each dataseries is higher or lower than 1 tick back of that dataseries to add up up vs downticks. I'm not sure it would really be more effecient to not loop as there is really very little calculations going on. Its really just 50 comparisons and 50 additions or subtractions on each tick.

I would love to try to get the PREM like this someday but with the multiplication to compute the weighting on each stock for the index, that might be unrealistic computing power wise on a tick basis.

[ryker 10]

It would be probably better to write this in OnMarketData I think and not in OnOrderUpdate.

 

As for the loop, I think you should get rid of it, cause as the OnOrderUpdate is fired every new tick, you'll end up adding or substracting 50 times for one tick (hope this makes sense?). You only want to increment total on a new tick.

 

I'm not sure on the calcul that needs to be done. Is it for every new tick, it tick up then total++, if tick down then total--?

Uptick if the ask is hit, and downtick if it's the bid.

 

Is that correct?

Bump:

I think that something like the following should do what you want:

/// <summary>

/// Called on each incoming real time market data event

/// </summary>

protected override void OnMarketData(MarketDataEventArgs e)

{

if (e.MarketDataType != MarketDataType.Last || e.MarketData.Ask == null || e.MarketData.Bid == null)

return;

if (e.Price >= e.MarketData.Ask.Price) { total++; UVOL += e.Volume; }

if (e.Price <= e.MarketData.Bid.Price) { total--; DVOL += e.Volume; }

}

 

Remove everything in OnBarUpdate, and only keep it for StrategyPlot, then keep also initialize.

Also, I've added calculation for UVOL and DVOL (but not sure if this is correct though).

PS: I haven't tested this yet, I've just written it quickly.

Edited December 1, 2008 by ryker

[darthtrader3.0beta 10]

I think it would be better to do with OnMarketData but I don't completely understand it..I guess I wasn't looking at it abstractly enough as I was thinking that it was more linked to market data on the primary instrument as opposed to any market data on any data series in the strategy...

The calculation is as you said, for every new tick(of each stock that has been loaded as an additional instrument), if tick up then total++, if tick down then total--...The primary instrument is really just a place holder for the strategy and has nothing to do with the calculations. I'll mess around with it tomarrow and see what happens.

One thing I'm not sure about though is things didn't look right until I reset the total to zero on each tick, then ran the loop..so basically taking an uptick/downtick snap shot on each tick of SPY for what the summed uptick/downticks looked like on the underlieing instruments.

Volume could be really interesting for doing like an S&P TICK delta indicator but I don't see how you can get away with not weighting things. 1000 shares of XOM at the ask needs to have greater meaning than a 1000 shares at the ask of Disney, since the 1000 shares of XOM has greater meaning as far as calculating the index itself.

How would you go about calculating the cash index from the the 500 S&P stocks? I was thinking of eventually trying to have a Struct for each stock with Price, Bid Volume, Ask Volume and the index weighting..then 500 custom dataseries for holding the structs...if that was possible you would literally be able to do anything you want as far as breadth goes but I'm sure that is not the optimal way to do things by a long shot..

Edited December 2, 2008 by darthtrader3.0beta

[BlowFish 10]

The way I would calculate the cash is have an arrays of 1..500 of last prices of each constituent and an array of 1..500 of weights. When a new event is generated because one of the constituents has a new tick I would add the difference between the last price and the new price multiplied by the weight to the index. Then move the new price into the last price array. This relies on the event that triggers things passing an 'index' to which data series triggered the event so you can get at the appropriate array element.

[darthtrader3.0beta 10]

Sounds like a plan...

The only part that I'm not sure about is the event triggering the index. Add() is going to create 500 seperate dataseries no matter what and as long as there is some get function to get the index of the data series that had an event, then the remaining calculations should be straight forward.

good stuff..What I posted about structs doesn't make any sense, I guess you could just even use a 4d array and store price, bid volume, ask volume, weight..then you have everything.

One thing to keep in mind with all this is NT7 might have a different way of handling OnMarketData because currently you can't backfill anything you do with that while NT7 is going to have backfill for bid/ask volume, ect...Not sure how they are going to handle this. Losing everything if you reload the chart midday is the biggest reason I haven't bothered to get too into OnMarketData. Thats always annoyed me greatly.

[ryker 10]

I think you can use BarsInProgress as your index here.

 

You need to construct an array with the weight associated to each index and access it using BarsInProgress. You can store lastPrice as well as suggested by BlowFish to calculate the index value.

 

It should not be too hard to implement adding some code in the version of OnMarketData I wrote above.

 

But you're right though, that if you refresh your chart you'll lose everything... I'm not sure that NT7 would bring backfill but at least you'll have backwards compatibility (from what they said on their forum) so you should be able to continue to use your code with NT7.

[BlowFish 10]

The exiting thing (for me) that NT7 is supposed to have tick by tick chart backfill refresh for all data streams regardless of timeframe. Opens up a whole range of possibilities similar to NeoTickers tick precise technology.

[darthtrader3.0beta 10]

I think you can use BarsInProgress as your index here.

 

Ahh yea, I see that would work. How would you do that though? I have a hard time not thinking in loops, the only way I can see to do that would be in on market data like:

for(i = x, x < 499,x++)

if(BarsInProgress == x)

do something

return;

 

Have you guys ever ran across documentation for all the methods in ninja? It seems like there should be some kind of Get.BarsInProgress() method but I've never found that level of documentation or maybe I'm just missing a bigger overall concept.

 

The big things I've read about for NT7 are the tick backfill for everything, multi instrument and time frame indicators, plotting the same instrument on one chart(still not sure if this means plotting the same instrument in price chart 1, that would be sick). I did read the other day though the beta is pushed back to the end of Q1 09, which kind of sucks but oh well.

 

Bump: Here is another question maybe you guys would know the general programming concept for..

I would love to eventually override plot to produce a kind of 5 or 15 minute distribution profile for TICK, like the volume profile or market profile of TICK values but it plots a new one every 5 or 15 minutes. I think that would be a far better way to see how the TICK is going off as opposed to candles or line on close.

I've dug into the volume profile indicator enough to get the general idea of how they pulled that monster off..

What I don't understand though is how you would store things to chop it up into different time intervals.

Volume profile uses a sorted dictortionary/hash table, so would you just use a custom data series of sorted dictionarys and set the chart to whatever time period you want and override the plot?

I guess in a more general sense I don't understand how to start these kind of calculations new for each bar..Another easier example would be say you wanted to find the average price for a 5 minute bar on a tick by tick basis..So keep a running total of the value of each tick divided by a running count of ticks..

If you do this in OnMarketData...how would you tell it to reset the tick value and counter to zero when the next bar starts?

[ryker 10]

About the BarsInProgress, it holds an int value, so it is the index you're looking for here.

If you do it this way:

for(i = x, x < 499,x++)

if(BarsInProgress == x)

do something

return;

 

You'll end up doing too many operations for nothing (but it also depends on what you're trying to achieve here).

 

Let's say you have an array of weight called myArray, then you need to do that:

/// <summary>

/// Called on each incoming real time market data event

/// </summary>

protected override void OnMarketData(MarketDataEv entArgs e)

{

if (e.MarketDataType != MarketDataType.Last || e.MarketData.Ask == null || e.MarketData.Bid == null)

return;

if (e.Price >= e.MarketData.Ask.Price)

{ total++;

UVOL += e.Volume * myArray[barsInProgress]; }

if (e.Price <= e.MarketData.Bid.Price)

{ total--;

DVOL += e.Volume * myArray[barsInProgress]; }

}

 

PS: It's just an example, I'm not entirely sure this is the correct way of calculating UVOL & DVOL.

 

For your question, I think you'll need to create a custom data series of arrays (if possible?) that holds the values you want and then plot that (but I'm not sure I understood what you try to do here).

 

Use if (FirstTickOfBar) to reset your values in OnOrderUpdate with a 5min serie and calculateonbarclose set to false.

[darthtrader3.0beta 10]

haha nice...I wasn't even thinking that BarsInProgress is the index itself, so no reason to search for it.

if (FirstTickOfBar) was what I was looking for as far as a general concept goes for those other questions...I think I'm going to try to build a plot of the intrabar POC/PVP like what Market Delta does.. I always thought that was handy information. I'm pretty sure custom data series can hold an array of any other data structure..How to do this is all right there in the Volume Profile indicator that comes with Ninja, very slick programming in that thing.

 

My only real complaint about Ninja is while their forum is amazing for what they do support, discussions like this quickly falter with the "we don't support this" post that will eventually come. Thanks a bunch for letting me toss the ball around with someone who knows alot more than I...

[darthtrader3.0beta 10]

Well this has nothing to do with TICK but could be interesting, I believe the logic is correct.

This chart is basically a "moving average" that is just connecting the average price of a 5 minute bar to the next average price of a 5 minute bar tick wise.

One thing that sticks out on any chart is that on a move most the bar is under the average price line. Could be usefull as far as a filter to not get in if price is on the wrong side of the average line if thats the direction your looking to trade. Doing this with an intrabar vwap might also be quite interesting.

#region Variables

// Wizard generated variables

private int myInput0 = 1; // Default setting for MyInput0

private double Counter = 1;

private double AvgPrice = 0;

private DataSeries myDataSeries;

// User defined variables (add any user defined variables below)

#endregion

protected override void Initialize()

{

Add(new Plot(Color.FromKnownColor(KnownColor.Orange), PlotStyle.Line, "Plot0"));

myDataSeries = new DataSeries(this);

CalculateOnBarClose = false;

Overlay = false;

PriceTypeSupported = false;

}

 

/// <summary>

/// Called on each bar update event (incoming tick)

/// </summary>

protected override void OnBarUpdate()

{

 

if (FirstTickOfBar)

{

AvgPrice = 0;

Counter = 0;

}

 

AvgPrice += Close[0];

Counter++;

myDataSeries.Set(AvgPrice / Counter);

Plot0.Set(myDataSeries[0]);

}

[ryker 10]

It looks like that your average price is the close of the bar (would make sense if this indicator has been applied on historical data I think)?

Have you tried doing this on an intraday vwap price yet?

[darthtrader3.0beta 10]

Yea I don't know, I'm kind of confused after spending some time with this thing.

While the bar is moving live the average price wiggles around, but I've only seen a few bars when the average price line closes that its not at the close of the bar...I'm talking maybe 1% of the time??

Maybe it has something to do with using close[0]?

 

I mean this logic is correct right?

AvgPrice += Close[0];

Counter++;

AvgPrice / Counter;

 

I tried getting the average vwap going but it didn't seem any different than the average...I don't know if that was a problem with my logic or if intra bar the volume weighting just doesn't have enough time to really weight things differently than the straight out average before a new bar.

[ryker 10]

The logic looks fine but I think you should use this indicator from start of day till end of day without any refresh. Otherwise on historical data your code will return Close[0] / 1.