taotree

I assume that's all he meant by order book--market depth to the same level as the exchange gives. For eMinis that's 10 levels, for other products typically 5 or whatever.

Oh, this is fun. So... First of all, let me say that when I looked at the other data it took me less than a couple minutes to find data that didn't line up. So either it does happen all over the place, or I just got really lucky. Sorry... not rigorous check yet since I'm still just exploring. So, I took a look at sample data direct from the CME. I spent a good deal longer finding ones that matched up--maybe 10 minutes? not sure, but then I found this: From: Top of Book 2010011509564006628780EES F100300900 201132252A M M100115 2010011509564006628780EES F100300157 201132002B M M100115 2010011509564006628790EES F100300038 201132002 100115 2010011509564006628800EES F100300900 201132252A M M100115 2010011509564006628800EES F100300120 201132002B M M100115 What this is showing is: First: ask = 900, bid = 157 Then a trade of 38 Then: ask = 900, bid = 120 Oops, we gained an extra one. So... there is at least one example of the CME's own data not lining up. So apparently the CME doesn't require them to match up. By the way... it's not that a 1 difference in the bid is going to make or break a strategy. It's that these discrepancies are making it difficult for me to assess the accuracy of a data feed. Another one: 2010011509564006628980EES F100300870 201132252A M M100115 2010011509564006628980EES F100300098 201132002B M M100115 2010011509564006628990EES F100300001 201132002 100115 2010011509564006629000EES F100300870 201132252A M M100115 2010011509564006629000EES F100300096 201132002B M M100115 And another: 2010011509564006629250EES F100300083 201132002A M M100115 2010011509564006629250EES F100301185 201131752B M M100115 2010011509564006629260EES F100300001 201132002 100115 2010011509564006629270EES F100300076 201132002A M M100115 2010011509564006629270EES F100301183 201131752B M M100115 Found those two quickly after the first... Okay... now time to ask CME directly about this and see if I can dig into what's going on. At this point, I'm going to assume then that probably the only possible way of assessing the accuracy of a data feed is to collect the data, purchase CME data for the same day, and run a check to ensure every tick was received in the same order and context.

By the way... It appears that you can get a direct internet feed to CME for $500/month but... does anyone know of a data distributor that just directly passes on CME data for less than $500/month? Then it would be easy to verify that we're getting the best data (can purchase some historical CME data directly and compare it to confirm). Comparing to data feeds that change the format and such is difficult because it requires manipulation and even interpretation.

I'm a little confused--I would expect historical data to be more accurate than real time data. I was collecting my own data for a while, but I now know that the feed I was using is inaccurate so... collecting data doesn't do me any good unless I know the data I'm getting is good. And as for "close enough"... that's what I'm struggling with. If this type of discrepancy "happens quite often" how do I assess the quality of data? If there are confusing things happening in even the best quality of data available (which I do not know if that's the case), how does one differentiate between the best, the "close enough" and the way off?

So... there have been various discussions about data feeds and I'd like to see if I can find a way to ensure I have a quality one. I've heard some recommendations but... Here's why I ask. Supposedly, I would expect that CQG data factory would be of high quality. This is historical data, so there are no questions about connection issues. I downloaded their sample and found this example: F.US.EPM08 20080104 1 0918 144375 B N N 28 F.US.EPM08 20080104 1 0918 144375 B N N 25 F.US.EPM08 20080104 1 0918 144400 A N N 34 F.US.EPM08 20080104 1 0918 144375 T N N 9 F.US.EPM08 20080104 1 0918 144375 T N N 5 F.US.EPM08 20080104 1 0918 144375 T N N 6 F.US.EPM08 20080104 1 0918 144325 B N N 8 F.US.EPM08 20080104 1 0918 144375 A N N 19 F.US.EPM08 20080104 1 0918 144325 B N N 5 Bid was at 25, then there were trades of 20 total and then bid went to 8. I asked them if that was normal and they said "what you see is normal and happens quite often". But we have 3 trade events in a row without any bid updates, and when we do get a bid update, it doesn't add up. I guess I might be able to get data from CME directly. http://www.cmegroup.com/market-data/datamine-historical-data/ It would be very interesting to see if they have the same issue of numbers not all adding up. What I'm getting at is... it seems incredibly difficult to find and confirm that data is accurate and I'm starting to wonder if it's possible at all.

I have looked at event data for zen-fire and can say that there are definitely things that don't add up. Something like bid is at 19, there's a trade at same bid price of 25 and next update is bid is at 43 at same price. Obviously, there should have been some updates in the middle in order for that to have happened but they didn't show up in the feed I received. Also, I can attest to data arriving out of sequence.

I think it's quite different. Explained below. Based on FulcrumTrader's statements, DTN.IQ does send all order book changes properly sequenced. Now... given the situation in question let's consider, what IS the proper sequence? A trade comes across that will take out all the rest of the bid. Is it correct to show the trade first, or the bid update first? And this goes for all the time--should the trade show up first, or the book change first? From a programming perspective... I would remove the bid before I processed the trade, otherwise there might be potential synchronization issues. I grab the amount off the bid to "hold" it while I process the order. Kind of like how someone puts an authorization on your credit card pre-sale and then charging it post-sale. Now, in the data feed, that low level of trade stuff does not need to be reflected, but... It seems quite reasonable to show book updates prior to transactions. It could go either way--so long as you are consistent and always do it one way or the other. I would be interested... in the low level data of dtn.iq--are book updates always given before transactions? So a trade of 10 at bid will be always immediately preceded by a reduction in the bid size by 10?

So... I wonder if there is a way to get this without paying the fee for Investor/RT?Any chance I could get this for Ninjatrader, I wonder... DTN says $60/month for basic service, how are you avoiding that?

Someone (I think on this thread) said that you could get delayed data from dtn for only $15/month. I haven't been able to find this information on their web site, though. Does anyone know where this is stated? Especially if it's possible to get this data through some type of API or download?

I thought you said TT Fix could get an uncoalesced version that was good?

Search some posts by FulcrumTrader as he has mentioned that he has analyzed zen-fire data and found it does have issues, and apparently Ninjatrader also has some issues with keeping up with data properly sometimes.

It depends on your platform. Ninjatrader for example, the bid/ask changes are sent independently of the trades. In many platforms where your indicator is only updated on trades and you call some API to get the inside bid/ask... in some cases (Tradestation, for example) that means you might want to forget about trying to assign bid/ask to trades because it's using snapshotted bid/ask data which are stale. In some platforms, that can be accurate as it sounds like Agekay mentioned he was using one like that in a previous post.

It's tracking each price and whether it was most recently a bid or an ask. So... when 123.22 went bid, it would be tracked as bid. When 123.21 went bid, it's tracked as bid. 123.22 is still in the dictionary as bid. So if a trade comes across at 123.22, it will be assigned as bid because that's the most recent value for that price.

I'm not interested in doing rebate trading exactly, but I am exploring a strategy that involves adding liquidity only (limit orders only) and is rather susceptible to commissions so... it seems that if I could find an instrument that offered rebates, it would be very interesting. Doing some searching, I find most references to rebate trading talking about stocks. Are there any futures or other non-stock, easy to day trade, instruments that offer commission rebates for adding liquidity?

This statement suffers from a non-sequitur falacy. The first (the API tells you) does not imply the second (no trades without BB/BA). All it means is that X_Trader or the exchange or wherever that data is coming from is using some type of algorithm for you. It would be rather interesting to dig into that and track down exactly what algorithm they're using.