SNYP40A1

Thanks Nate and Blowfish, I appreciate the info. I actually posted a thread over at "that other place" and came to the conclusion that binary files are the absolute fastest way to store tick data. The more I thought about it, it's not that hard to write some code that will search among the binary files for the proper range that one is seeking. In fact, since the data will be stored in time order anyways, I don't see what value a database would add for what I am considering now. I can always go DB later if the need arises. I actually had read all those articles before you posted. If I went with a DB, it would probably be HDF5. Berkley DB supports concurrency (the concurrent version, data store version does not support concurrency at all) through internal locking. Most databases might work that way, but I don't want to ever have the writer blocked for a reader. Most important function of my tick datalogger is to log data. I was also concerned about the possibility of database corruption with HDF5. Unless the hard drives starts to fail, you can't really corrupt a binary file. So I may revisit this topic later, but for now, simple binary files seem to be the way to go for my current purposes. In any case, I appreciate the info!

Redundant Post, please delete.

I am currently logging tick data into binary files on one computer (Computer A). But I am looking for a database to store the data and furthermore, I want to be able to query Computer A to backfill my charting software on another computer, Computer B. After backfilling, I then want Computer A to relay all received ticks relevant to the instrument(s) being monitored by Computer B to be forwarded to Computer B. I know that it's not a good idea to relay data for a true automated HFT system. However, I am not doing HFT and that latency should be ok for now, but I'd like to keep it at a minimum. I am using Linux for both systems. Does anyone know of a good open-source database solution and method for relaying the ticks? Would master-slave database replication be the way to go? At this point, my database would be not much larger than a couple GBs, I could flush the database to binary files at the end of each week to keep it small if necessary.

I don't know of any feeds which supply the entire order book. They said that they don't filter their market depth data.

I posted about this exact issue a month or two ago on here. I spoke with IQ feed a week or two ago and they claimed two important things: 1. Timestamps come from the exchange -- the exchange provides the time stamps so that prevents market events from being *logged* out of order (but it's still possible for you to *receive* events out of chronological order...not a major problem, easy to sort out. 2. They claim that they never filter data (at least for CME, that's all I asked about). They report the complete datafeed from CME. I might be missing something, but that sounds as good as having a direct line to CME. That's probably why Fulcrum recommends IQ Feed, but then again, he's using a $2k/month feed so he's obviously getting something else too (and I'd love to know what's worth the extra $1800 / month :-)). Can someone please send me just 1 day of ES data (anytime after February) from your feed. I'd be especially interested in a sample from IQ Feed as that's the vendor I am considering, but also interested in comparing against other providers and I promise to document my findings here for the benefit of everyone.

Exactly. And I see holding assets in the US becoming a greater liability. It's probably ideal to store assets in multiple countries -- diversify. Just curious, if you buy a real estate in other countries, fully pay for the property and pay local property taxes on it, can the US government step in and seize? I suspect that in some countries they can and in others they cannot.

I started the same thread on another trading forum before this one was started and I got flamed pretty bad for suggesting that people should move to another country to avoid paying higher taxes. People over there really don't like the idea of other people not paying taxes. The best argument against not leaving the country is that it's still a pretty good place to live. We have the best higher education system in the world, that's undisputed. You don't have to work very hard to get by and have a nice life. You can even argue as some on here have that if you are paying taxes, that means you are making money -- be happy with what you have. That's all true, but at the end of the day it's still a business. And like all businesses, if they can get the same product from a different supplier for cheaper, then they should in the spirit of an open and competitive market. The problem with government, as I see it is that there's no competition. Combined with the fact that large organizations tend to sponsor corruption and the matter starts to become an issue of ethics for me, but that's a whole other debate. In any case, if you earn money as a US citizenship, you have to pay uncle Sam. I think the US and Philippines are the only countries that do this. Even if you leave the country and earn the money someplace else...you still have to pay taxes to the US (I think there might be a way you can exempt the first $90k though). What about moving to Panama or the Bahamas? Also, for me it's not just about taxes. I see the US becoming very hard-pressed for more money and with a growing population of unproductive people who can VOTE, I'm a bit concerned. Soon, all a president will have to do to get elected is campaign on redistributing the wealth and then...oh...already happened. So I think it might be wise to store money outside the country just to protect it from getting seized in the future. Question is, what other country would be safe (with a stable government) from having the US sieze it. Seems like Switzerland is no longer a safe haven. I'm not even talking about avoiding taxes on interest income here, just talking about storing the money in a place where the US can't seize it. Worst case, convert all your savings to raw gold and then bury them someplace out in the middle of nowhere... Anyone got a better idea? Edit: Oh, and if you do try to revoke your citizenship, there's an exit tax on all assets above $2 million. Before that, I think you still had to file taxes for something like 10 years after renouncing citizenship. That's pretty ridiculous, definitely like the mafia.

Nothing is hard if you know what you are doing...

I e-mailed two people at Zen-Fire and have not received comment. They both have responded to my queries before. I suspect that they are not saying anything until they have either determined the cause of the issue and/or the fix. I could be wrong, but their silence tells me that they know about the issue and are still working out the solution.

Anyone know if Zenfire or Rithmic has made any statement about this?

It was a general example and my point is obvious. If your connection cannot keep up with the stream, packets will be lost.

As I said before: If your computer / connection cannot keep up with the real time packet stream, there will be packet loss. Example, try receiving a 10 Mbps real-time stream on a 14.4 kbps dial-up connection. TCP can't perform magic.

If it's the case that the bid / ask always changes with each trade, those updates would have to be communicated anyway so it could save bandwidth. Is it possible to execute a trade without changing the bid/ask?

TCP/IP in itself will not guarantee that you receive every packet if it's the case that your computer / connection cannot keep up with the amount of data thrown at it. I think that's all he was trying to explain. It's only more robust than UDP because if it's the case that your system misses a packet not due to lack of bandwidth / processing, but something else, then the packets will be resent. For this reason, either way, you can't rely on when the packets were received for the timestamped, the feed itself must timestamp every tick at the exchange.

Yup, I think I have read everything on the CME site and I can't find anything describing what and how market events are reported. There's a continuation of this discussion someplace else, you can find it by googling "Ninja Zen no more unfiltered data?". I am pretty sure Fulcrum is posting there too under another alias ;-). The most relevant document that I found which describes what events are reported is this: http://www.cmegroup.com/globex/files/GlobexRefGd.pdf But there are still some cases like the one mentioned a page or two back that I am still unsure about.

I think we are looking at the same reference: http://www.cmegroup.com/globex/files/GlobexRefGd.pdf I just found that today. A market order with protection will be executed immediately and will not create a new order on the books like a limit order does. The stop order are essentially limit orders too, except that they remain hidden form the market until the trigger point is hit. I could be wrong but for the equity futures, I thought I remembered reading that there were no stop price bands. The only other two orders are minimum execution, which is essentially a market order that only executes if it can be filled immediately. Hidden quantity and request for cross which both appear to be limit orders.

That's a good question. They would definitely be matched and filled against each other. But would they be printed to the exchange? If they are not printed to the exchange, then that could explain the example about 10109 that I posted above. For completeness, I think the limit orders should update the books, reporting 1 new best bid and 1 new best ask both at 17 1/2, the trade should be reported, then the 17 1/2 levels should be updated back to 0 orders for both bid and ask and all of these actions would be reported under the same timestamp. This would show exactly what happened. But it would also generate 5 extra ticks that could be inferred if it is the case that overlapping limit orders are the only trades that can execute without affecting the book. If anyone knows the answer to this situation, please let us know. And even more important, if you know where these rules are posted, please let me know. I was not looking at the market during these times, but it seems that the only time a market order should be converted to a limit order is when there are no limit orders at any price level to match the trade. Although I am not sure how that could happen. What price would they assign to the new limit order?

As I mentioned, I spent over 5 hours looking over the CME site and the closest link I could find describing their data messaging format is the one I posted last night. Some of the time was productively spent, I learned more about the futures market, but I am very surprised that a specification describing how market activity (book and trades) is not linked somewhere on the front page of CME and especially Zen-Fire. Can someone send me yesterday's market history (trades, bid/ask, volume) for any of FDAX, ES, YM, or NQ? I can format that datalog in the same format as mine, sort both by timestamp, and then compare to see what's different between the two. It would be great to compare against the DTN feed which is supposed to be superb. Another note, I am not using NinjaTrader. I connect to Zen-Fire and log the data directly using their API. Although I am only monitoring 9 instruments, I have the bandwidth and CPU power to record at least the entire equities futures market. The CPU usage even during peak is negligible, something less than 5%. I am not sure if Zen-Fire uses UDP or TCP for their packet transmission protocol. Again, would be nice to have this someplace on their website. For those who don't have a background in data networks, UDP requires much less bandwidth compared to TCP. However, UDP can drop packets. I don't know if that's what's happening which is why I would like to compare against another feed. If someone can send me another datafeed history, even 1 hour of data during peak should be sufficient, I promise to post my findings here. I would also be happy to post a diff of the two datalogs. I believe these limit orders will still show up on the books. They may or may not get executed against one another depending on their price levels. The market is fair and will give each order the best available execution price. For a buy order, this will be the best current ASK price and for a sell order, this will be the best current bid price. Let's say the sell order comes in with the limit price set at the current ASK price. Then that sell order will go to the back of the best ask queue and will not get executed unless the other buy limit order is large enough to consume the entire best ask queue. The buy order obeys the same rules. However, if the buy order limit price is better than the current best bid, then it will get at least partially executed (may not get fully executed if it's the case that there are not enough orders on the ask at prices less than or equal to the limit price on the buy order). Obviously, if the new limit buy and new limit sell orders did not have overlapping or equal prices, then they would not get executed against each other. So I am pretty sure that all limit orders cause an update to the market books, but please correct me if I am wrong. Another interesting case that could happen (but I don't think it does) would be if 2 market orders came into the market at the same time and the order quantity happened to be both equal and also an even number. In this case, the exchange could execute half of the transaction at the best bid price and half of the transaction at the best ask price. The execution price would average to directly in the middle between the best bid and best ask price and would be the fairest price to both parties. I don't think they do this because CME claims to match orders in less than 7 ms. Timestamp resolution must be somewhere in the microsecond range and it's probably very rare that two matching orders hit the market at exactly the same time. Seems like an interesting tradeoff between execution time and execution price.

I didn't think a market buy could be matched to a market sell. I am a newbie, but are you sure that can happen? I thought market buy order are ALWAYS matched to the best ask price and market sells are always matched to the best bid price. I think this would be the fairest transaction price. Otherwise, if you have a 1 tick spread, one party has to pay the spread and the other does not. For that reason, I don't think market orders are ever matched directly, but let me know if I am wrong. I have been looking for over 5 hours from some documentation from the CME which describes in detail their order matching algorithms. If anyone has this please point me to it.

Taotree, I have actually noticed that frequently, the date event time stamps are arriving out of order. I think this is probably related to Zenfire's distribution mechanism. Each packet can take a different path through the internet and since they are only separated by milliseconds during fast action, they could easily arrive out of order. For historical data, I sort everything first. FulcrumTrader, are you still seeing an issue with Zenfire (guess it's only been 10 days)? Below is a capture of the YM on Feb. 2 from Zenfire. Does anyone see what is missing from this data: 1265099523474,682000,ASK, 1,10109.0 1265099523845,376000,ASK, 2,10109.0 1265099524369,371000,ASK, 4,10109.0 1265099524577,589000,ASK, 3,10109.0 1265099525936,762000,BID, 3,10108.0 1265099526054,823000,TRADE,1,10109.0 1265099526054,823000,VOLUME,5787,0.0 1265099526054,911000,TRADE,1,10109.0 1265099526054,911000,VOLUME,5788,0.0 1265099526055,410000,TRADE,1,10109.0 1265099526055,410000,VOLUME,5789,0.0 1265099526058,675000,BID, 33,10101.0 1265099526060,252000,ASK, 21,10111.0 1265099526062,987000,TRADE,2,10109.0 1265099526062,987000,VOLUME,5791,0.0 1265099526063,878000,ASK, 18,10110.0 1265099526066,547000,BID, 17,10106.0 1265099526072,51000,BID,2 0,10105.0 1265099526081,717000,ASK, 2,10109.0 I can understand how the first 3 trades were executed at 10109.0, but I don't understand how the last 2 trade order was executed at that same level. We know there was an ask of 3 contracts 10109.0 at 1265099524577,589000. The 3 trades would have consumed those asks so it seems that there would be no bid/asks at 10109.0 after the group of 3 trades. There were no updates to 10109.0 so I don't understand how the last trade of 2 contracts could have executed. Here's another example that I posted on: Here's a more concise example: Time(ms),Time(ns),Type,Volume,Price 1265102922624,821000,TRADE,1,10134.0 1265102922624,821000,VOLUME,6744,0.0 1265102922624,821000,BID,0,10134.0 1265102922630,620000,TRADE,1,10134.0 1265102922630,620000,VOLUME,6745,0.0 1265102922631,30000,BID,25,10131.0 How did that second trade happen? The line immediately before shows 0 offers at 10134. I spent about two hours looking for a description on Zenfire data reporting format and the closest description I could find was this: Data Directly from the Market Data Platform So I assume that CME sends out updates to bid/ask if they have been affected by new order, trades, or cancellations. Can someone explain the data above to me or has Zenfire started dropping data?

There's another forum and I can't give out the URL because it's probably against forum policy, but let's just say it's run by a Big guy named Mike who trades...if you become a VIP member, you can download historical data from their forums.

Actually, the longer I look at this problem, I don't see anything wrong with applying a moving average to the data and ignoring the non-uniform sampling issue. Run a moving average and at the given point of interest, and just take the last update of the MA. The reason that I think this approach is valid is because signal reconstruction, as long as the Nyquist Criterion applies, does not require uniform sampling -- I read that statement somewhere on comp.dsp, but I have not been able to verify it. Can anyone confirm?

I found this thread very relevant, especially the long post on the second page mentioning Jurik's MA: Designing Filters with non-uniform sampling | Comp.DSP | DSPRelated.com Still looking for my filter.

Non-uniform sampling. That's what I am looking for. I thought about doing least-squares analysis / linear regression, but I think that's probably getting too complicated...I guess all I am really trying to do at this point is plot the data. In Ninjatrader, it looks like what they do is simply group all the data into time intervals (1-minute, 5-minute, 233 tick, etc) and then plot the close of the given bar and hold that level until the next close. Essentially, they are doing zero-order hold. That's simple and it works, but seems like there must be an algorithm I could use and get a much smoother plot for very little effort.

Weighing by time rather than position in the time series would be more accurate. If I weighted only by position, then I could basically apply any moving average -- that would essentially be treating the data as uniform sampled. I could go even more complicated. I could weight the data by time as well as volume to reflect more of the instantaneous "force" of the market. Or weight by bid vs. ask bias, etc. Although, at that point, I have a new indicator beyond just plotting the data. For now I simply want to feed in the tick data and for any given point in time, get an estimate for what the price is at that point. What I plan to do now is simply take the last trade behind my desired sample point. Been a while since I took a signals and systems class, but I think this process is referred to as zero-order hold: Continuous/Discrete Conversions of LTI Models :: Operations on LTI Models (Control System Toolbox™) I am developing my own platform in Java -- at least the analysis part.