Zenfire and DTN Feed Different?

[FulcrumTrader 10]

Whilst UB contributed some valuable thoughts and insights some of the stuff he wrote, particularly about data feeds, was complete hyperbole. I pulled him on a couple of occasions but did not press the matter as the good stuff far out weighed the more. I always thought it a shame that some one that clearly was well informed with much to contribute on certain topics could talks such nonsense about others.

 

I guess you are talking about things like CQG when you talk of higher end feeds? Probably another good option.

 

From the last few posts it would seem safe to conclude that the problem issue is potentially with any live data feed and infrastructure/capacity limitations. However the original poster suggests that this is not the case and that IQ.Feed matches live and historically. This is odd as Zenfire is fairly highly compressed, it seems to get more ticks in fewer packets so you would expect it to suffer less from data loss. Also your tests would suggest that it matches. Strange.

 

My comments about UB were in relation to the higher end feeds, and their costs, that are available these days.

 

Just to be clear, the Zenfire/Rithmic feed is NOT always matching my DTN.IQ feed since the CME data granularity increase. Some of this imo will not be resolved until Zenfire/Rithmic add a ticker plant type capability.....even when Zenfire/Rithmic send data to a sophisticated user with more than adequate infrastructure, there is still at times data loss so that is not optimal. Backfill with DTN.IQ feed is a must at this point imo for Zenfire/Rithmic feed users who need clean bid/ask data.

[FulcrumTrader 10]

What is DTN.IQ? Data fee only? Which chart software can use their data?

 

And how is CQG compare to zenfire and DTN.IQ's data completeness/accuracy during real time?

 

CQG has always been known for very robust data feed so their feed should be as good as DTN.IQ feed for bid/ask data work.....also, they provide TFlow volume studies that rely on a proper bid/ask data stream so their feed should be good.

 

CQG TFlow Charts and Studies

[O66 10]

Just to be clear, the Zenfire/Rithmic feed is NOT always matching my DTN.IQ feed since the CME data granularity increase. Some of this imo will not be resolved until Zenfire/Rithmic add a ticker plant type capability.....even when Zenfire/Rithmic send data to a sophisticated user with more than adequate infrastructure, there is still at times data loss so that is not optimal. Backfill with DTN.IQ feed is a must at this point imo for Zenfire/Rithmic feed users who need clean bid/ask data.

 

Thanks.

I was afraid for this.

Imo Zenfire isnt usable for trading where bidxask info is needed.

(you dont want to backfill every few minutes)

DTN Iq.feed is for now the solution for feeding my IRT charts.

[O66 10]

after 4.5 hour live feed with Zenfire, chart looks like this:

image 1

 

 

after backfill with DTN:

image2

 

Cumulative delta on image 1 is about 33000

Image 2 is showing 28000 cumulative delta which is less

 

but since im plotting delta here more is not always better.

I assume the backfilled image2 is correct

somewhere during the last 4.5 hours i lost 5000 negative cumulative delta with Zenfire feed

 

I have a dual internet connection (adsl and cable) and am using a dual wan router with fail over.

Of course it is possible that i lost connection for a second but that doesnt explain that other traders at the other end of the world have the same chart (before backfill) in Ninja feeded with zenfire.

 

My conclusion is that it is not my charting package, its not my connection. It must be the feed.

The backfill option for zenfire as mentioned in multicharts forum is interesting

Hopefully it will be available soon so that we can test and compare.

Edited December 23, 2009 by O66

[FulcrumTrader 10]

066 -

 

It does seem the only verified and simplistic route to go at this time for Investor RT or Ninjatrader users, that do bid/ask Delta Volume work, is to use DTN.IQ feed. I can see how having to do DTN.IQ feed "backfill" during the day while using Zenfire/Rithmic or TT Fix Adapter feed would be a pain in the butt.

[mikka 10]

Hi,

 

I may not be understanding this issue correctly but will the DTN.IQ feed, real-time feed, not suffer the same issues i.e the problem is the way that continuously changing data is transmitted (UDP). Is it the actual 'backfill' that is resolving the problem.

[FulcrumTrader 10]

Hi,

 

I may not be understanding this issue correctly but will the DTN.IQ feed, real-time feed, not suffer the same issues i.e the problem is the way that continuously changing data is transmitted (UDP). Is it the actual 'backfill' that is resolving the problem.

 

Correct....DTN.IQ puts out feed in a different way than a typical broker feed and uses built in mechanisms to verify the data flow. My DTN.IQ feed always matches a CME daily run of data....I do not ever have problems with DTN.IQ feed for bid/ask work.

Edited December 27, 2009 by FulcrumTrader

[Ceschiatti 10]

Hi,

 

I may not be understanding this issue correctly but will the DTN.IQ feed, real-time feed, not suffer the same issues i.e the problem is the way that continuously changing data is transmitted (UDP). Is it the actual 'backfill' that is resolving the problem.

 

Correct....DTN.IQ puts out feed in a different way than a typical broker feed and uses built in mechanisms to verify the data flow. My DTN.IQ feed always matches a CME daily run of data....I do not ever have problems with DTN.IQ feed for bid/ask work.

 

Fulcrumtrader, which feed do you use for realtime data?

 

 

Sent from my iPhone using Tapatalk

[FulcrumTrader 10]

My DTN.IQ feed always matches a CME daily run of data....I do not ever have problems with DTN.IQ feed for bid/ask work.

 

As you can see.....DTN.IQ feed.

[MoreYummy 10]

So after a realtime bid/ask fill during intraday using DTN.iq, you would get an exact chart as using CME's chart at the end of day?

[FulcrumTrader 10]

If I check my DTN.IQ bid/ask run at the end of day it will match an uncoalesced CME run from a more direct source (I will leave it at that).

[thariman 10]

Yes.....I have mentioned before that I have verified DTN.IQ feed.....it is the ONLY regular feed that I use and have been able to verify.

 

I do think that CQG probably has exceptional feed too....Esignal does not have clean bid/ask data last I checked and I am not sure why. Tradestations feed is a mess so I will not even go into that one.....LOL!

 

Anyone trying to use OEC or Transact feed will definitely have to "backfill" imo with DTN.IQ to make sure they have clean bid/ask data runs each day. If you use Investor RT or Marketdelta with DTN.IQ feed you are set up perfect for bid/ask data work.

 

Hi Fulcrum if you have access to CME data could you find out which one is correct/wrong

It have different bid/ask inside bid flag between the two and different total volume.

Left side is Esignal, right side is Zen-Fire both using Ninja trader T&S

 

 

Thank you

 

Tony

[FulcrumTrader 10]

Tony,

 

Sending you a PM....I will just say that it is very odd to me that these data feeds can't get there data output straight. I know DTN.IQ feed spent a bunch of money upgrading their data infrastructure this year, so maybe that is why they are getting it right and others like Esignal have some errors.

[BlowFish 10]

If I check my DTN.IQ bid/ask run at the end of day it will match an uncoalesced CME run from a more direct source (I will leave it at that).

 

Just out of interest why are you so reticent to mention the data source? The sceptic in me immediately thinks that it may not stand up to scrutiny . In any case It makes it impossible for anyone to judge the veracity of your claims. I can't help wondering whether the data you used and/or your testing methodology might be flawed (I am not saying it is). For example I have observed inconsistencies with Zen/Gom both pre and post CME changes. From a pretty recent post of yours in the OFA thread you where advising pretty much the same thing about Ninja/Zen as you have with DTN.IQ. "NinjaTrader w/Zenfire is good for CD work"

 

Anyway forgive me for banging on about this but I have often seen damn smart traders show a bit less rigour when it comes to the technical side of things.

[FulcrumTrader 10]

Just out of interest why are you so reticent to mention the data source? The sceptic in me immediately thinks that it may not stand up to scrutiny . In any case It makes it impossible for anyone to judge the veracity of your claims. I can't help wondering whether the data you used and/or your testing methodology might be flawed (I am not saying it is). For example I have observed inconsistencies with Zen/Gom both pre and post CME changes. From a pretty recent post of yours in the OFA thread you where advising pretty much the same thing about Ninja/Zen as you have with DTN.IQ. "NinjaTrader w/Zenfire is good for CD work"

 

Anyway forgive me for banging on about this but I have often seen damn smart traders show a bit less rigour when it comes to the technical side of things.

 

Real simple.....DTN.IQ is matching CME data runs and I have NOW FOUND Zenfire/Rithmic feed runs that are NOT matching and that is a problem. I have ALSO since had verified what is causing the Rithmic/Zenfire feed data drops since I was myself using Zenfire feed with my NT charting (for back up and remote trading work). Now I also can't use NT/Zenfire for any CD work which is disappointing to me (hopeful that will get resolved soon with feed enhancements through Zenfire and platform enhancements with NT7...we will see). New FACTS have shown me for the time being I can no longer use NT/Zenfire for any CD work....I will be very happy if this set up can be used again in the future.

 

In the end, do your own due diligence and go through what it takes to verify a feed and find one that you trust. I am not expecting anyone to rely on my info....just putting out the info to make sure everyone is aware of potential problems with the various feeds for those who need CLEAN bid/ask data.

[taotree 10]

Real simple.....DTN.IQ is matching CME data runs and I have NOW FOUND Zenfire/Rithmic feed runs that are NOT matching and that is a problem. I have ALSO since had verified what is causing the Rithmic/Zenfire feed data drops since I was myself using Zenfire feed with my NT charting (for back up and remote trading work).

 

You mention you have verified what is causing the issue? Can you share that information?

 

Any chance it's related to something I discovered while using the API... there are cases happening regularly where the data events are coming out of order. I grab the exchange timestamp to store data and since I use a counter just in case I get multiple items with the same timestamp, I discovered the out of order data right away and had to compensate for it. I haven't investigated to see what "damage" it might do to analysis, though.

[FulcrumTrader 10]

You mention you have verified what is causing the issue? Can you share that information?

 

Any chance it's related to something I discovered while using the API... there are cases happening regularly where the data events are coming out of order. I grab the exchange timestamp to store data and since I use a counter just in case I get multiple items with the same timestamp, I discovered the out of order data right away and had to compensate for it. I haven't investigated to see what "damage" it might do to analysis, though.

Yes you are seeing some of the issues first hand.....what we verified was several issues with both the platform and at times the feed causing data drops. We also detected some data drops with Rithmic connected to Investor RT charting, so there are multiple problems that at anytime can cause these data drops of bid/ask data intraday (with the various broker supplied feeds).

[SNYP40A1 10]

Taotree,

I have actually noticed that frequently, the date event time stamps are arriving out of order. I think this is probably related to Zenfire's distribution mechanism. Each packet can take a different path through the internet and since they are only separated by milliseconds during fast action, they could easily arrive out of order. For historical data, I sort everything first.

 

FulcrumTrader, are you still seeing an issue with Zenfire (guess it's only been 10 days)? Below is a capture of the YM on Feb. 2 from Zenfire. Does anyone see what is missing from this data:

 

1265099523474,682000,ASK, 1,10109.0

1265099523845,376000,ASK, 2,10109.0

1265099524369,371000,ASK, 4,10109.0

1265099524577,589000,ASK, 3,10109.0

1265099525936,762000,BID, 3,10108.0

1265099526054,823000,TRADE,1,10109.0

1265099526054,823000,VOLUME,5787,0.0

1265099526054,911000,TRADE,1,10109.0

1265099526054,911000,VOLUME,5788,0.0

1265099526055,410000,TRADE,1,10109.0

1265099526055,410000,VOLUME,5789,0.0

1265099526058,675000,BID, 33,10101.0

1265099526060,252000,ASK, 21,10111.0

1265099526062,987000,TRADE,2,10109.0

1265099526062,987000,VOLUME,5791,0.0

1265099526063,878000,ASK, 18,10110.0

1265099526066,547000,BID, 17,10106.0

1265099526072,51000,BID,2 0,10105.0

1265099526081,717000,ASK, 2,10109.0

 

I can understand how the first 3 trades were executed at 10109.0, but I don't understand how the last 2 trade order was executed at that same level. We know there was an ask of 3 contracts 10109.0 at 1265099524577,589000. The 3 trades would have consumed those asks so it seems that there would be no bid/asks at 10109.0 after the group of 3 trades. There were no updates to 10109.0 so I don't understand how the last trade of 2 contracts could have executed. Here's another example that I posted on:

 

Here's a more concise example:

 

Time(ms),Time(ns),Type,Volume,Price

1265102922624,821000,TRADE,1,10134.0

1265102922624,821000,VOLUME,6744,0.0

1265102922624,821000,BID,0,10134.0

1265102922630,620000,TRADE,1,10134.0

1265102922630,620000,VOLUME,6745,0.0

1265102922631,30000,BID,25,10131.0

 

How did that second trade happen? The line immediately before shows 0 offers at 10134.

 

I spent about two hours looking for a description on Zenfire data reporting format and the closest description I could find was this:

 

Data Directly from the Market Data Platform

 

So I assume that CME sends out updates to bid/ask if they have been affected by new order, trades, or cancellations. Can someone explain the data above to me or has Zenfire started dropping data?

[FulcrumTrader 10]

Yes.....Rithmic/Zenfire feed is NOT working for proper BID/ASK volume studies. If you do your research and see how thorough DTN.IQ manages and time stamps their feed, you will then see why many of these broker supplied feeds can't touch the capability/reliability of DTN.IQ feed.

 

I do have an update on this feed situation from some recent testing. If you have a very good newer PC with very good internet (cable modem at a minimum) we have a client who is working with NT connected to DTN.IQ feed and he is NOT getting any data drops now. So a trader can then track the GomCD with NT connected to DTN.IQ feed.......the only capability you would not have is historical lookback (like you would have with DTN.IQ feed connected to Investor RT Pro for up to a 4 week backfill). If you wanted historical lookback in NT with DTN.IQ you would need to save your own data which can be done with the GomCD Recorder function. So this was good to see that one of my clients has been able to get NT to work now with DTN.IQ feed for Cumulative Delta tracking......cool!

[BlowFish 10]

I have not noticed any data drops from zenfire. I have a 20MB connection with a decent ISP not sure if that is considered good nowadays, toying with the idea of upgrading to 50mb can't really justify it beyond bragging rights.

 

A market sell getting matched to a market buy would explain what you observed Snyp.

 

Incidentally looks like Ninja are really dropping the ball on there handling of historic bid ask information. In a nutshell they are not preserving the correct sequence when storing the ifo for historic retrieval. A major blow for this type of analysis. There is a thread here here about it BidAsk Historical... - Page 4 - NinjaTrader Support Forum maybe if enough people 'vote' they may prioritise fixing this issue.

[SNYP40A1 10]

I didn't think a market buy could be matched to a market sell. I am a newbie, but are you sure that can happen? I thought market buy order are ALWAYS matched to the best ask price and market sells are always matched to the best bid price. I think this would be the fairest transaction price. Otherwise, if you have a 1 tick spread, one party has to pay the spread and the other does not. For that reason, I don't think market orders are ever matched directly, but let me know if I am wrong. I have been looking for over 5 hours from some documentation from the CME which describes in detail their order matching algorithms. If anyone has this please point me to it.

[BlowFish 10]

I didn't think a market buy could be matched to a market sell. I am a newbie, but are you sure that can happen? I thought market buy order are ALWAYS matched to the best ask price and market sells are always matched to the best bid price. I think this would be the fairest transaction price. Otherwise, if you have a 1 tick spread, one party has to pay the spread and the other does not. For that reason, I don't think market orders are ever matched directly, but let me know if I am wrong. I have been looking for over 5 hours from some documentation from the CME which describes in detail their order matching algorithms. If anyone has this please point me to it.

 

Now you mention it you might well be correct! As you say what price would they be matched at? I guess a trawl through the CME site might reveal the answer. You do sometimes get ticks between the best bid and best ask. How about a limit sell and a limit buy arriving at the same time and at the same price. It would not be unreasonable to expect them to be matched and filled without being displayed in the order book?

[FulcrumTrader 10]

Incidentally looks like Ninja are really dropping the ball on there handling of historic bid ask information. In a nutshell they are not preserving the correct sequence when storing the ifo for historic retrieval. A major blow for this type of analysis. There is a thread here here about it BidAsk Historical... - Page 4 - NinjaTrader Support Forum maybe if enough people 'vote' they may prioritise fixing this issue.

I myself just stay away from NT for now for any BID/ASK volume work until I see them enhance their platforms capabilities. Investor RT Pro for only $60 a month is a bargain to me and I can always have solid historical lookback capabilities with DTN feed.

[SNYP40A1 10]

Now you mention it you might well be correct! As you say what price would they be matched at? I guess a trawl through the CME site might reveal the answer. You do sometimes get ticks between the best bid and best ask. How about a limit sell and a limit buy arriving at the same time and at the same price. It would not be unreasonable to expect them to be matched and filled without being displayed in the order book?

 

As I mentioned, I spent over 5 hours looking over the CME site and the closest link I could find describing their data messaging format is the one I posted last night. Some of the time was productively spent, I learned more about the futures market, but I am very surprised that a specification describing how market activity (book and trades) is not linked somewhere on the front page of CME and especially Zen-Fire.

 

Can someone send me yesterday's market history (trades, bid/ask, volume) for any of FDAX, ES, YM, or NQ? I can format that datalog in the same format as mine, sort both by timestamp, and then compare to see what's different between the two. It would be great to compare against the DTN feed which is supposed to be superb.

 

Another note, I am not using NinjaTrader. I connect to Zen-Fire and log the data directly using their API. Although I am only monitoring 9 instruments, I have the bandwidth and CPU power to record at least the entire equities futures market. The CPU usage even during peak is negligible, something less than 5%. I am not sure if Zen-Fire uses UDP or TCP for their packet transmission protocol. Again, would be nice to have this someplace on their website. For those who don't have a background in data networks, UDP requires much less bandwidth compared to TCP. However, UDP can drop packets. I don't know if that's what's happening which is why I would like to compare against another feed. If someone can send me another datafeed history, even 1 hour of data during peak should be sufficient, I promise to post my findings here. I would also be happy to post a diff of the two datalogs.

 

How about a limit sell and a limit buy arriving at the same time and at the same price. It would not be unreasonable to expect them to be matched and filled without being displayed in the order book?

 

I believe these limit orders will still show up on the books. They may or may not get executed against one another depending on their price levels. The market is fair and will give each order the best available execution price. For a buy order, this will be the best current ASK price and for a sell order, this will be the best current bid price. Let's say the sell order comes in with the limit price set at the current ASK price. Then that sell order will go to the back of the best ask queue and will not get executed unless the other buy limit order is large enough to consume the entire best ask queue. The buy order obeys the same rules. However, if the buy order limit price is better than the current best bid, then it will get at least partially executed (may not get fully executed if it's the case that there are not enough orders on the ask at prices less than or equal to the limit price on the buy order). Obviously, if the new limit buy and new limit sell orders did not have overlapping or equal prices, then they would not get executed against each other. So I am pretty sure that all limit orders cause an update to the market books, but please correct me if I am wrong.

 

Another interesting case that could happen (but I don't think it does) would be if 2 market orders came into the market at the same time and the order quantity happened to be both equal and also an even number. In this case, the exchange could execute half of the transaction at the best bid price and half of the transaction at the best ask price. The execution price would average to directly in the middle between the best bid and best ask price and would be the fairest price to both parties. I don't think they do this because CME claims to match orders in less than 7 ms. Timestamp resolution must be somewhere in the microsecond range and it's probably very rare that two matching orders hit the market at exactly the same time. Seems like an interesting tradeoff between execution time and execution price.

[BlowFish 10]

I believe these limit orders will still show up on the books. They may or may not get executed against one another depending on their price levels.

 

I think they probably will if they are at the same price. For example 17 1/4 bid 17 3/4 asked if a limit order to buy 1 @ 17 1/2 and a limit order to sell 1 @ 17 1/2 came in at the same time they would be matched and filled. In the early days of electronic exchanges surprisingly few supported more than a couple of basic order types. Often a market order would be converted to a limit order well inside the market. I guess things are more sophisticated nowadays.

 

In your example if a limit buy and a limit sell both at 10109 both arriving simultaneously they would simply be matched and reported as trade @ 10109? If they where put on the book it would essentially be crossed with best bid 10109 and best ask 10109? Thats my guess anyway.

 

FT you are probably right about NT, currently it is a square peg and we have a round hole. I am getting some promising results with Zen and Multicharts though I need to do some further tests (purposely overloading MC) to be sure.With MC DTN.IQ is an option too.