AgeKay

That's really funny because I just listened to a .NET Rocks interview today where the interviewee talked about that book and said it was wonderful. I also think that functional languages are the future since individual chips are not getting any faster but going multicore. Parallelism is a no brainer with functional languages. I read my data at O(1) [constant time for those who do know the Big O notation] no matter how big the file. You can't say that about databases and most other data storage solutions.

As BlowFish has correctly pointed out a few times, your file system would basically be your index. You just construct the file path from the market and date, which you use to directly access file you want to read.

I think your thinking is too theoretical. Realistically, you would not load 10 years of data in one go (and you could not even do that with a database either since you have only a limited amount of RAM), maybe one day of 500 markets. And yes, I do not plan to store data of the individual S&P stocks. If you store futures tick data only, then you'd be fine with a file based storage solution.

Well, your program would basically read it all in once on loading of the chart so that you can quickly move to any point in time of your chart. You never have to access it again for the day. And a database cannot be faster than the simple file read operation. Databases are not magically faster. They still somehow reside on the hard drive and have to read the data from disk first before you can do anything. What is not scalable about that? The number of files does not matter. I'd rather have millions of small files than a huge file that can could easily be corrupted.

BlowFish, I think you are making it way too complicated. All you need to do is read your data one trade at a time. Absolutely no need for databases and therefore no need for indexes (since there are no tables and no relational data). For example, put each trade of a day in one .csv file. Then read that file line by line (each line has all the information of one trade) and do what you need to do with the data. Reading a whole days worth of trades takes less than one second on my computer of which 66% is spent splitting the line in it's individual values using the build-in System.String.Split(delimiter) (I haven't optimized the code, I am sure you can make it much faster if you need it). Processing the trades into other things like bars takes milliseconds. And you could then even save those bars (seconds, minutes, hours, whatever) to a .csv file to be able to read in the future even faster.

Why is tick data in files not logical? What do you need a database for? If you were to really think long-term than you would realize that your favorite database technology will probably not exist in 10 years anymore. Either that or you would have to upgrade a few times or migrate to other products. I save each day of ticks in a file and have a hierarchical folder structure for the years, months and contracts. This is infinitely scalable and I could compress data I don't use that often while a database of 10 years of tick data would be massive and the sheer size of your tables will make your queries a lot slower.

I agree, I can't think of a reason one would want to do a query against a database that contains sequential data. I think saving them to file is the best solution and then read it in a stream since that is exactly how you receive real-time data. This also prevents you from doing backtesting mistakes where you know more about the future than you would in reality (e.g. accessing futures trades). In my current implementation I just have a list that only holds the last 2 trades so that I can never access more than the last trade and the trade before that, which has the additional benefits of small memory footprint, very fast insertion and access times. File based storage has the advantages of mobility, easy backup and can be accessed from all sorts of technologies. And again, you are not dependent on other products.

Totally agree. It's a shame that people that share valuable information are discouraged to post. I really liked to hear first hand comments from the developer of TickZoom. We are both obviously biased here, so let me comment on rolling your own software from the ground up. This is exactly what I do and it is not as much work as it might seem at first. But let me qualify this a bit. Ok, this is my 5th complete rewrite of my own trading software and I have learned a lot with regards to architecture and performance of trading applications. So I am building on past ideas and implementations but still completely rewriting my app since I wanted it to be 'clean' and free of legacy code that could force me into making suboptimal implementations. I am also using the latest and greatest in technology (best development environment and best APIs) that is available to me which also speeds up the process enormously. And I consider myself a pretty good developer. No, not if it is GDI which always runs in software. But with WPF is free and you get hardware acceleration for free which you simply can't beat with GDI, and it will only get better in the future since Microsoft is heavily investing in WPF. The WPF 2D Graphics API looks very similar to GDI+, but still easier to use since you don't have to worry about when to invalidate what since WPF does that all for you. So there is really no advantage of using GDI over WPF. Doing graphics with WPF is trivial compared to GDI. Doesn't really take that long. All you do is read the trades from disk and put them in your custom objects. Implementing a streaming list is trivial (I used generic lists before and just implemented a streaming list a few minutes ago. Took me less than 30min including testing that everything still works). I have never heard these libraries before so I guess you don't really need this. This is a good point, which actually supports my next point. This totally depends on how well designed the APIs of your broker are. It can be a piece of cake or it could be a nightmare. For example, TT's API is pretty straight forward. I actually don't trust other peoples code. This is not because I think I am better than other developers, but based on experience. You always end up regretting that you are dependent on someone else's code. Bugs, performance problems, missing flexibility, there is always a problem. Sure, you can go in and fix it if it's open source but I usually takes me less time to just implement the damn thing myself myself than read the entire source code and figure out how it works. Of course, you are always dependent on some API. I chose .NET in which Microsoft has heavily invested and they usually develop better code than me, you or any other open source project contributors. So in my point of view, the advantages of completely rolling your own is complete control over your software and independence from others. I essentially create my own perfect trading environment, everything is exactly like I want it to be, and no one can do that for you. I also decide what is most important to me right now, be it performance, that new chart/indicator, visuals, performance analysis of my own trading, etc...

I had fixed a few bugs myself in ZedGraph too, but the performance was still not acceptable for me. How is your performance if you zoom into your 10 million ticks and then scroll quickly to either side? I would be surprised if you get smooth scrolling. If so, did you do any performance optimizations inside ZedGraph too? I am willing to share a few gotchas/insides I have found about writing the most performant 2D Graphics in WPF if someone wants to go that route too. It's mostly based on measuring and comparing the performance of my own code and stepping through the .NET Framework Source Code. My current implementation is impossibly fast even though I am not even virtualizing yet. Oh, that's interesting. I somehow missed that when I checked out your website. Do you have screenshots of these charts? I've come up with a couple chart types you will not have seen anywhere but they are not based on bars and I am not willing to share them either ;-). What do you mean when you say "zoom speed testing"? Does not make sense to me. Your programmer, especially if you have more than 1, will cost you more than (low) 6 figures per year, so why not just buy the commercial system?

I just found out about DynamicDataDisplay - an open source charting library written in WPF. I have not tried it personally yet. Has anyone tried this? P.S.: I am completely dumbfounded by the performance of 2D Graphics in WPF (using DrawingVisuals). I have more than 20k drawings on my chart and when scrolling the chart quickly the drawings are rendered faster than my eyes can conceive. Absolutely amazing. And best of all, my CPU just sleeps because the GPU is doing all the work. And I am not even virtualizing the drawings yet.

I've looked at Dundas Charts. Look very good (visually). They have been around forever and seem to offer every chart under the sky. But I have not personally tried their charts. Using an external charting component, you always have to worry about flexibility and performance. For example, I was completely blown away by the flexibility of ZedGraph, but like many other charts, they are not very good for real time financial data. My ultimate trading software is basically something that helps me trade my personal strategy optimally. So it will look different for every strategy. Sure, it will use the same components underneath, but it would essentially be completely different for every strategy. I think it would be very hard to develop something that would suit everyone. This only works well for software platforms based on technical indicators (like NinjaTrader), which I am not a big fan of. I looked at DTN. You need to pay like $600 just to get access to their API (they say it's for "support"). You cannot even have a look at their API documentation before you buy it. To get access to the TT feed (through the X_Trader API), you need X_Trader Pro, which costs depending on your broker around $1250-$1500 per month. Zen-Fire would probably be cheaper but I don't know whether it is even available outside of NinjaTrader.

While this would obviously be great, I see two reasons why this is highly unlikely: 1. There is no incentive for the programmers unless they are getting paid. If they do create something really useful then they would probably just keep it to themselves. 2. Most programmers do not understand trading and most traders do not understand programming. This explains why all the trading platforms out there are so crappy. NinjaTrader is a an example of a trading platform written by programmers who do not understand trading (check out their "advanced" matching algorithm on the simulator, ridiculous). X_Trader is an example of a trading platform designed by a trader (Harris Brumsfield) who does not know what is actually possible with computers (limited charts and very limited in performance analysis). I have not used CQG, but I am sure you will get eye cancer from using it over a long period of time ;-) (their designers must be blind). I just checked TickZoom. They seem to use the open source ZedGraph component for charting. I have used this for my own software before too since it is extremely flexible and very well designed, but found it to be buggy and having poor performance when scrolling on a lot of data and or using it in real-time. So I am currently rewriting all of my charts in WPF 2D Graphics which are hardware accelerated out of the box since it uses DirectX (the API that games use for their drawing) underneath. You just can't beat the performance unless you writing directly the DirectX or the graphics card. It's really amazing. StockChartX looks good though. I would definitely use their WPF version though because of the hardware acceleration. BlowFish, can you please provide a link to "Dcraig over at that other forum". I'd like to check it out. I've googled for it, but didn't find anything.

Yes, I am sure. Just to make things clear. I mean time & sales when I mean tape. So market orders appear in the time & sales and limit orders can be seen in the order book, but you never know the size of individual limit orders. You're right. This has not much to do with the original thread, so let's stop this discussion.

I agree, I don't want to fight either. Spot Forex is very different from Futures since they are not traded on an exchange (Globex is an exception but the volume is not meaningful there). You couldn't even calculate Trade Intensity with currencies since not all trades of all traders are shown. Please ask your full time futures traders friends to explain to you how limit orders and market orders in futures markets work. And you don't need super fast quants to bite size an order, just a simple piece of code.

Isn't CurreneX for Forex only? We are talking about Futures markets here.

Viper, the complete post is false. Please stop spreading misinformation, there is already enough of it out there. Believe it or not, exchanges keep track of all limit orders and market orders and more information that you are probably not even aware of, which they then sell as historical data. You might not see this in real-time but they save the information.

:rofl: Oh, soooo funny. I was gonna ask Viper to just shut up since he apparently he doesn't have a clue how the market works. Viper, seriously, you're confusing a limit order with a stop market order. Limit orders and market orders are fundamentally different. That's why they have different names! :crap:

What kind of orders?

The distinction is limit orders vs. market orders. Limit orders do not appear on the tape, ever. You never know the actual size of the limit order that sits in the order book. One could place a 1000 contract order or ten orders of 100 contracts. Looks exactly the same. It could be one trader with an iceberg order or other traders that place their limit orders when the best bid or ask is taken out. You never actually know whether it is an iceberg order or not and how big it actually was since it is impossible to detect while automated trading that splits up a large market order in several smaller market orders is possible to detect. That's the difference.

No Viper, I think you are confusing the iceberg orders and what trade intensity is trying to detect. Iceberg orders are limit orders that where only part of the order is placed in the order book / L2 (thus only the "tip of the iceberg" is seen) and the orders are 'refilled' when they are traded until the desired size is executed. Trade Intensity detects market orders that were split up to disguise the actual size of a big market order.

daenz, I don't think trade intensity will work well with currencies since the idea behind it is to detect automated trades in stock index futures.

I would like to thank everyone on behalf of everyone for sharing information, indicators, charts and opinion so everyone can feel good about himself or herself, and so we can finally stop this childish "oh, see what i have done, you haven't done anything, so who are you to comment?". Seriously, this is not even funny anymore.

Guys, please stop this. There is really no sense in arguing on an anonymous forum. We don't want this to turn into ET.

I look at the time & sales.

Please make sure you know how the double auction market works before you make statements that are not true. If all the limit orders on the best ask are pulled then the price goes up without even any market order hitting the exchange. An extreme example that I have seen a few times is where all the bid or offers on 5 tick levels were pulled at the same time and price moved up or down 5 ticks without any market order hitting the market.