SNYP40A1

Thanks Nate and Blowfish, I appreciate the info. I actually posted a thread over at "that other place" and came to the conclusion that binary files are the absolute fastest way to store tick data. The more I thought about it, it's not that hard to write some code that will search among the binary files for the proper range that one is seeking. In fact, since the data will be stored in time order anyways, I don't see what value a database would add for what I am considering now. I can always go DB later if the need arises. I actually had read all those articles before you posted. If I went with a DB, it would probably be HDF5. Berkley DB supports concurrency (the concurrent version, data store version does not support concurrency at all) through internal locking. Most databases might work that way, but I don't want to ever have the writer blocked for a reader. Most important function of my tick datalogger is to log data. I was also concerned about the possibility of database corruption with HDF5. Unless the hard drives starts to fail, you can't really corrupt a binary file. So I may revisit this topic later, but for now, simple binary files seem to be the way to go for my current purposes. In any case, I appreciate the info!

Redundant Post, please delete.

I am currently logging tick data into binary files on one computer (Computer A). But I am looking for a database to store the data and furthermore, I want to be able to query Computer A to backfill my charting software on another computer, Computer B. After backfilling, I then want Computer A to relay all received ticks relevant to the instrument(s) being monitored by Computer B to be forwarded to Computer B. I know that it's not a good idea to relay data for a true automated HFT system. However, I am not doing HFT and that latency should be ok for now, but I'd like to keep it at a minimum. I am using Linux for both systems. Does anyone know of a good open-source database solution and method for relaying the ticks? Would master-slave database replication be the way to go? At this point, my database would be not much larger than a couple GBs, I could flush the database to binary files at the end of each week to keep it small if necessary.

I don't know of any feeds which supply the entire order book. They said that they don't filter their market depth data.

I posted about this exact issue a month or two ago on here. I spoke with IQ feed a week or two ago and they claimed two important things: 1. Timestamps come from the exchange -- the exchange provides the time stamps so that prevents market events from being *logged* out of order (but it's still possible for you to *receive* events out of chronological order...not a major problem, easy to sort out. 2. They claim that they never filter data (at least for CME, that's all I asked about). They report the complete datafeed from CME. I might be missing something, but that sounds as good as having a direct line to CME. That's probably why Fulcrum recommends IQ Feed, but then again, he's using a $2k/month feed so he's obviously getting something else too (and I'd love to know what's worth the extra $1800 / month :-)). Can someone please send me just 1 day of ES data (anytime after February) from your feed. I'd be especially interested in a sample from IQ Feed as that's the vendor I am considering, but also interested in comparing against other providers and I promise to document my findings here for the benefit of everyone.

Exactly. And I see holding assets in the US becoming a greater liability. It's probably ideal to store assets in multiple countries -- diversify. Just curious, if you buy a real estate in other countries, fully pay for the property and pay local property taxes on it, can the US government step in and seize? I suspect that in some countries they can and in others they cannot.

I started the same thread on another trading forum before this one was started and I got flamed pretty bad for suggesting that people should move to another country to avoid paying higher taxes. People over there really don't like the idea of other people not paying taxes. The best argument against not leaving the country is that it's still a pretty good place to live. We have the best higher education system in the world, that's undisputed. You don't have to work very hard to get by and have a nice life. You can even argue as some on here have that if you are paying taxes, that means you are making money -- be happy with what you have. That's all true, but at the end of the day it's still a business. And like all businesses, if they can get the same product from a different supplier for cheaper, then they should in the spirit of an open and competitive market. The problem with government, as I see it is that there's no competition. Combined with the fact that large organizations tend to sponsor corruption and the matter starts to become an issue of ethics for me, but that's a whole other debate. In any case, if you earn money as a US citizenship, you have to pay uncle Sam. I think the US and Philippines are the only countries that do this. Even if you leave the country and earn the money someplace else...you still have to pay taxes to the US (I think there might be a way you can exempt the first $90k though). What about moving to Panama or the Bahamas? Also, for me it's not just about taxes. I see the US becoming very hard-pressed for more money and with a growing population of unproductive people who can VOTE, I'm a bit concerned. Soon, all a president will have to do to get elected is campaign on redistributing the wealth and then...oh...already happened. So I think it might be wise to store money outside the country just to protect it from getting seized in the future. Question is, what other country would be safe (with a stable government) from having the US sieze it. Seems like Switzerland is no longer a safe haven. I'm not even talking about avoiding taxes on interest income here, just talking about storing the money in a place where the US can't seize it. Worst case, convert all your savings to raw gold and then bury them someplace out in the middle of nowhere... Anyone got a better idea? Edit: Oh, and if you do try to revoke your citizenship, there's an exit tax on all assets above $2 million. Before that, I think you still had to file taxes for something like 10 years after renouncing citizenship. That's pretty ridiculous, definitely like the mafia.

Nothing is hard if you know what you are doing...

I e-mailed two people at Zen-Fire and have not received comment. They both have responded to my queries before. I suspect that they are not saying anything until they have either determined the cause of the issue and/or the fix. I could be wrong, but their silence tells me that they know about the issue and are still working out the solution.

Anyone know if Zenfire or Rithmic has made any statement about this?

It was a general example and my point is obvious. If your connection cannot keep up with the stream, packets will be lost.

As I said before: If your computer / connection cannot keep up with the real time packet stream, there will be packet loss. Example, try receiving a 10 Mbps real-time stream on a 14.4 kbps dial-up connection. TCP can't perform magic.

If it's the case that the bid / ask always changes with each trade, those updates would have to be communicated anyway so it could save bandwidth. Is it possible to execute a trade without changing the bid/ask?

TCP/IP in itself will not guarantee that you receive every packet if it's the case that your computer / connection cannot keep up with the amount of data thrown at it. I think that's all he was trying to explain. It's only more robust than UDP because if it's the case that your system misses a packet not due to lack of bandwidth / processing, but something else, then the packets will be resent. For this reason, either way, you can't rely on when the packets were received for the timestamped, the feed itself must timestamp every tick at the exchange.