taotree

I assume that's all he meant by order book--market depth to the same level as the exchange gives. For eMinis that's 10 levels, for other products typically 5 or whatever.

Oh, this is fun. So... First of all, let me say that when I looked at the other data it took me less than a couple minutes to find data that didn't line up. So either it does happen all over the place, or I just got really lucky. Sorry... not rigorous check yet since I'm still just exploring. So, I took a look at sample data direct from the CME. I spent a good deal longer finding ones that matched up--maybe 10 minutes? not sure, but then I found this: From: Top of Book 2010011509564006628780EES F100300900 201132252A M M100115 2010011509564006628780EES F100300157 201132002B M M100115 2010011509564006628790EES F100300038 201132002 100115 2010011509564006628800EES F100300900 201132252A M M100115 2010011509564006628800EES F100300120 201132002B M M100115 What this is showing is: First: ask = 900, bid = 157 Then a trade of 38 Then: ask = 900, bid = 120 Oops, we gained an extra one. So... there is at least one example of the CME's own data not lining up. So apparently the CME doesn't require them to match up. By the way... it's not that a 1 difference in the bid is going to make or break a strategy. It's that these discrepancies are making it difficult for me to assess the accuracy of a data feed. Another one: 2010011509564006628980EES F100300870 201132252A M M100115 2010011509564006628980EES F100300098 201132002B M M100115 2010011509564006628990EES F100300001 201132002 100115 2010011509564006629000EES F100300870 201132252A M M100115 2010011509564006629000EES F100300096 201132002B M M100115 And another: 2010011509564006629250EES F100300083 201132002A M M100115 2010011509564006629250EES F100301185 201131752B M M100115 2010011509564006629260EES F100300001 201132002 100115 2010011509564006629270EES F100300076 201132002A M M100115 2010011509564006629270EES F100301183 201131752B M M100115 Found those two quickly after the first... Okay... now time to ask CME directly about this and see if I can dig into what's going on. At this point, I'm going to assume then that probably the only possible way of assessing the accuracy of a data feed is to collect the data, purchase CME data for the same day, and run a check to ensure every tick was received in the same order and context.

By the way... It appears that you can get a direct internet feed to CME for $500/month but... does anyone know of a data distributor that just directly passes on CME data for less than $500/month? Then it would be easy to verify that we're getting the best data (can purchase some historical CME data directly and compare it to confirm). Comparing to data feeds that change the format and such is difficult because it requires manipulation and even interpretation.

I'm a little confused--I would expect historical data to be more accurate than real time data. I was collecting my own data for a while, but I now know that the feed I was using is inaccurate so... collecting data doesn't do me any good unless I know the data I'm getting is good. And as for "close enough"... that's what I'm struggling with. If this type of discrepancy "happens quite often" how do I assess the quality of data? If there are confusing things happening in even the best quality of data available (which I do not know if that's the case), how does one differentiate between the best, the "close enough" and the way off?

So... there have been various discussions about data feeds and I'd like to see if I can find a way to ensure I have a quality one. I've heard some recommendations but... Here's why I ask. Supposedly, I would expect that CQG data factory would be of high quality. This is historical data, so there are no questions about connection issues. I downloaded their sample and found this example: F.US.EPM08 20080104 1 0918 144375 B N N 28 F.US.EPM08 20080104 1 0918 144375 B N N 25 F.US.EPM08 20080104 1 0918 144400 A N N 34 F.US.EPM08 20080104 1 0918 144375 T N N 9 F.US.EPM08 20080104 1 0918 144375 T N N 5 F.US.EPM08 20080104 1 0918 144375 T N N 6 F.US.EPM08 20080104 1 0918 144325 B N N 8 F.US.EPM08 20080104 1 0918 144375 A N N 19 F.US.EPM08 20080104 1 0918 144325 B N N 5 Bid was at 25, then there were trades of 20 total and then bid went to 8. I asked them if that was normal and they said "what you see is normal and happens quite often". But we have 3 trade events in a row without any bid updates, and when we do get a bid update, it doesn't add up. I guess I might be able to get data from CME directly. http://www.cmegroup.com/market-data/datamine-historical-data/ It would be very interesting to see if they have the same issue of numbers not all adding up. What I'm getting at is... it seems incredibly difficult to find and confirm that data is accurate and I'm starting to wonder if it's possible at all.

I have looked at event data for zen-fire and can say that there are definitely things that don't add up. Something like bid is at 19, there's a trade at same bid price of 25 and next update is bid is at 43 at same price. Obviously, there should have been some updates in the middle in order for that to have happened but they didn't show up in the feed I received. Also, I can attest to data arriving out of sequence.

I think it's quite different. Explained below. Based on FulcrumTrader's statements, DTN.IQ does send all order book changes properly sequenced. Now... given the situation in question let's consider, what IS the proper sequence? A trade comes across that will take out all the rest of the bid. Is it correct to show the trade first, or the bid update first? And this goes for all the time--should the trade show up first, or the book change first? From a programming perspective... I would remove the bid before I processed the trade, otherwise there might be potential synchronization issues. I grab the amount off the bid to "hold" it while I process the order. Kind of like how someone puts an authorization on your credit card pre-sale and then charging it post-sale. Now, in the data feed, that low level of trade stuff does not need to be reflected, but... It seems quite reasonable to show book updates prior to transactions. It could go either way--so long as you are consistent and always do it one way or the other. I would be interested... in the low level data of dtn.iq--are book updates always given before transactions? So a trade of 10 at bid will be always immediately preceded by a reduction in the bid size by 10?

So... I wonder if there is a way to get this without paying the fee for Investor/RT?Any chance I could get this for Ninjatrader, I wonder... DTN says $60/month for basic service, how are you avoiding that?

Someone (I think on this thread) said that you could get delayed data from dtn for only $15/month. I haven't been able to find this information on their web site, though. Does anyone know where this is stated? Especially if it's possible to get this data through some type of API or download?

I thought you said TT Fix could get an uncoalesced version that was good?

Search some posts by FulcrumTrader as he has mentioned that he has analyzed zen-fire data and found it does have issues, and apparently Ninjatrader also has some issues with keeping up with data properly sometimes.

It depends on your platform. Ninjatrader for example, the bid/ask changes are sent independently of the trades. In many platforms where your indicator is only updated on trades and you call some API to get the inside bid/ask... in some cases (Tradestation, for example) that means you might want to forget about trying to assign bid/ask to trades because it's using snapshotted bid/ask data which are stale. In some platforms, that can be accurate as it sounds like Agekay mentioned he was using one like that in a previous post.

It's tracking each price and whether it was most recently a bid or an ask. So... when 123.22 went bid, it would be tracked as bid. When 123.21 went bid, it's tracked as bid. 123.22 is still in the dictionary as bid. So if a trade comes across at 123.22, it will be assigned as bid because that's the most recent value for that price.

I'm not interested in doing rebate trading exactly, but I am exploring a strategy that involves adding liquidity only (limit orders only) and is rather susceptible to commissions so... it seems that if I could find an instrument that offered rebates, it would be very interesting. Doing some searching, I find most references to rebate trading talking about stocks. Are there any futures or other non-stock, easy to day trade, instruments that offer commission rebates for adding liquidity?

This statement suffers from a non-sequitur falacy. The first (the API tells you) does not imply the second (no trades without BB/BA). All it means is that X_Trader or the exchange or wherever that data is coming from is using some type of algorithm for you. It would be rather interesting to dig into that and track down exactly what algorithm they're using.

If you're going to pay for IQFeed, why use a different feed for real-time? Why not use it for both real-time and historical?

Which data feeds have you worked with? Aside from the drop/sequence issues, Zenfire does give you sequential order book updates. And from what FulcrumTrader has said, I get the impression that DTN does give you that without the issues.

All that has nothing to do with my comment--it's all still valid. The only thing you pointed out is that Zenfire does not satisfy the requirement in my last statement. And to be accurate... Zenfire's out of order problem doesn't invalidate the timestamp--it's just passing that along from the exchange--it invalidates the sequence of arrival.

You don't have to know the timestamps at all to know the sequence of events. So long as the data arrives in your algorithm in the proper sequence, that's all you need. I use an id encoder algorithm: In my code, I keep track of the most recent timestamps. When a new event comes in, I check to see if that timestamp has already arrived, if so, I increment a counter and tack that on to create my id, otherwise I tack on a 0 value for the counter. That allows me to preserve the sequence precisely regardless of the timestamp precision, and every event has a unique id. Now, that assumes that the data flow from exchange through feed provider all preserve the sequence of events as they send it to you--but... no matter what you're always beholden to the accuracy of the data feed.

Yes. As I mentioned before, you're talking about the internals of how exchanges work, and I'm talking about the actual data that arrives at your trading platform and how to code an indicator to deal with it. As you say, the two don't always match.

Are you assuming a spread of only one tick? In that case, you are correct. However, many instruments have spreads of much greater than one tick. Speaking in terms of ticks, if bid is at 0 and ask is at 2 then a trade at 1 is at the midpoint.

There may be a difference between what is conceptually required by the market and exchange and what actually comes across the wire in the data feed. Any filtered data feed or snapshotted one (IB, Tradestation, TT, etc.) will definitely have this problem since you cannot get the actual last bid and ask--you can only get a snapshot value that may be stale data. I was using zenfire which is supposedly unfiltered, but there apparently have been cases of this issue with it as well. FulcrumTrader has confirmed that zenfire does drop data sometimes, so that may be why--so maybe on a high quality feed, this would not be an issue. But... I don't know.

My understanding is that is not the case--especially since the data stream is not perfect. And I know I have seen transactions occur on prices other than the last bid or ask I received from the data feed.

Someone asked me about this code and when I looked at it I noticed that it is missing the case where a transaction == mp, ie. when a transaction is at the midpoint between best bid and ask. I have read that at least one approach to that is to use whatever the direction was for the previous transaction for that case. Anyway... one would probably want to change the code to reflect that. Unless one wants to consider that if a transaction is exactly at the midpoint, then it's not directional at all and so ignoring it is what you want. Could argue either way I suppose.

You mention you have verified what is causing the issue? Can you share that information? Any chance it's related to something I discovered while using the API... there are cases happening regularly where the data events are coming out of order. I grab the exchange timestamp to store data and since I use a counter just in case I get multiple items with the same timestamp, I discovered the out of order data right away and had to compensate for it. I haven't investigated to see what "damage" it might do to analysis, though.