Data Feed with API for Order Book Analysis

[AgeKay 10]

I am looking for a data feed with an API that allows me to do order book analysis on Eurex. This requires a data feed with an API that allows one to enumerate every single order book update (including trades) in the order it occurred. An added plus would be exchange time stamps with high resolution (i.e. milliseconds or smaller).

 

To do this for Eurex, that means that the vendor either has to subscribe directly to the Enhanced Broadcast Solution (EBS) at Eurex [members only so unlikely that vendors subscribe to it] or to the CEF ultra+ Eurex feed from Deutsche Börse (this is basically just EBS + trade recovery) [for non-members such as vendors]. I could find only a handful of vendors on the website of Deutsche Börse that are connected to the CEF ultra+ Eurex feed (one of them is CQG).

 

Another important part is that the API must allow one to receive order book updates and trades in a synchronous manner so that the correct order of events can be ensured. Also no updates may be missed when the thread that the event handler is raised on is blocked (e.g. due to long processing) [i.e. the API instead queues those updates].

 

The only data vendor I've found so far that seems to meet both these requirements is CQG (waiting for confirmation from CQG). Please share your experience with their API if you have any.

 

Are there any other out there besides CQG that meet my requirements?

[gooni 10]

Zen-Fire should do this, or at least seems to based on my usage of the API - even though I never used it for the level 2 feed (order book).

 

Although I very strongly recommend that you don't block the API even if you don't want to process the events immediately (ie you need guaranteed synchronous processing). I used a thread to pull the events from the API and place them in a local memory queue (actually a Queue in C#) and then processed them from that queue. If the API itself does it, all good and well - I'm just saying that it's not a requirement and none that I know do it (most APIs are geared towards technologically advanced users, all of which have their own methods for synchronising threads and prefer to retain control over it).

 

Best part: Zen-Fire API access is free. Can't comment on quality of data though.

[AgeKay 10]

Thank you for your detailed response gooni. Zen-Fire is a very fast feed which is excellent for execution but based on Fulcrum's investigations its data not always complete.

 

Thank you also for the heads up for not blocking the API. I don't do that anyway and always queue all events to be processed on another thread. The reason for this being a requirement is to make sure that the API does not miss any data during fast markets because even if you immediately pull the data and queue it, you can miss data if the API does not queue unprocessed data. At least that was my experience with the X_Trader API because their feed is also optimized for execution like Zen-Fire. On the other hand, I never miss any data from IQFeed sockets API but it's not connected to the data feeds I mentioned above (it's connected to CEF Core from Deutsche Börse that delivers netted order book data).

[gooni 10]

No problems AgeKay, just wanted to be sure you understood the difference between the API blocking and your event thread queuing.

 

Regarding Zen-Fire, the suggestion from FulcrumTrader is that the order book updates might come out of sequence with regards to the trades themselves, causing a mild discrepancy that over time could build up when one is doing bid-ask cumulative delta work.

 

However, you need to ask yourself whether this affects you. If you look at, say, a Ninja DOM when connected to Zen-Fire, the frequency of updates is quite good. Is this guaranteed to be in the same order that is disseminated by the exchange? I'd be amazed if it was. But is it good enough for what you want to do, given that it's free and easy to program to?

 

As always, it might be more prudent trying what's free and easy before attempting to expensively solve a problem you may never have.

[AgeKay 10]

gooni, thanks again for your concern.

 

For the kind of analysis I want to do I actually do need a "perfect" feed with all order book updates that can be enumerated in the order they were received. I've done lots of research for this already and unfortunately haven't found anything cheap that meets my requirements. I am not surprised though considering that this is pretty high-end analysis that I want to do (think what algos do). I actually just got confirmation from CQG that they do meet my requirements and they cost about as much as X_Trader and their API is cheaper than TT's API.

 

I've already implemented the order book visualization which would show me "interesting" behaviour in the order book, unfortunately it wasn't as helpful even though it was connected to feeds that are considered "good" for trading because I was missing lots of data (I could actually see that things were not adding up). If I get the feed I am looking for, then I am literally not going to miss anything going on in the order book, which would enable me to see things that guys like UrmaBlume are looking for.

[BlowFish 10]

Zen-Fire should do this, or at least seems to based on my usage of the API - even though I never used it for the level 2 feed (order book).

 

Although I very strongly recommend that you don't block the API even if you don't want to process the events immediately (ie you need guaranteed synchronous processing). I used a thread to pull the events from the API and place them in a local memory queue (actually a Queue in C#) and then processed them from that queue. If the API itself does it, all good and well - I'm just saying that it's not a requirement and none that I know do it (most APIs are geared towards technologically advanced users, all of which have their own methods for synchronising threads and prefer to retain control over it).

 

Best part: Zen-Fire API access is free. Can't comment on quality of data though.

 

I just started with Zenfire and C#. I have ticks running into a grid last, BB & BA no 'level II' yet. I am kind of learning it all as I go along. I came to a similar conclusion.....have a single queue for ticks that raises events for interested methods. Zen is not thread safe but using a single queue and then multi threaded from there on seems like the right way to proceed. Time to read the multi thread chapters of 'CLR via C#'. I have to say it seems like a massively complex environment. One thing that occurs to me is that there is no way to know if a race condition can occur within the Zen API itself.

[AgeKay 10]

Time to read the multi thread chapters of 'CLR via C#'. I have to say it seems like a massively complex environment.

 

I find multi-threading one of the hardest things to do. Debugging a threading problem is nightmare but VS 2010 supposedly makes it much easier (still using VS 2008). The best tutorial of threading in C# is this. Try to keep it simple. Most of the time, all you need to do is to prevent a piece of code from being entered by two threads at the same time to ensure data you access is the same it was when you entered that piece of code. You can use the 'lock' keyword on any instance of a reference type to do that. All the other times, I just use a 'ConsumerThread' class I developed that consumes data on another thread.

 

One thing that occurs to me is that there is no way to know if a race condition can occur within the Zen API itself.

 

This can only happen if they raise events on different threads. If they always raise events on the same thread, then no race condition. Best to just ask them.

[gooni 10]

The best tutorial of threading in C# is this.

This is the tutorial I used when writing my Zen-Fire API connector (C#). Can't recommend it highly enough. If you'd like some code, PM me.

 

Since I come from a messaging/integration background, I naturally created a single 'hub' that would receive ticks from Zen-Fire, and I then distribute to interested 'tick sinks' (which are plugins). Some tick sinks are files, others are the console itself, others are trading programs etc. This saves having to rewrite sections of code and manage multiple connections.

 

By far the largest issue was reconnection after a connection failure. The API is meant to take care of it internally, but it simply doesn't always work. I had to resort to actually killing my daemon and restarting it. This works nicely

 

Note that Zen-Fire is definitely not thread-safe. All access to its events etc should be single threaded. This is especially important for GUI applications (which I try and avoid).

[BlowFish 10]

I seem to keep hijacking threads with this .net stuff. Perhaps I should start a 'roll your own with .net c#'.

 

Anyway coming from a background of linked lists pointers procedures etc. I am finding it quite hard to 'unlearn' and actually start thinking in OO. I have looked at (and read chunks of) various books in series that I have liked in the past. Most are heavy going (even the 'beginners' ones) with page counts in thousands. That site is refreshingly concise. Another one that I came across that clicked in a similar way is at brainbell C Sharp Tutorials I have just got to the multi-threading section so can't comment on that but have had several aha moments up to that point.

[AgeKay 10]

I don't mind you hijacking my thread. I've put this one hold as it seems that CQG doesn't meet one of my requirements after talking to them. I have to work on my own trading as that has been going downhill lately. .NET is massive and there is no one who knows everything about it, just many experts for one or two domains. I've been programming exclusively in .NET since its introduction (7-8 years ago) and I am still having problems catching up with all the new stuff they keep introducing (I am still using .NET 3.5/ C# 3 / VS 2008 and haven't had a chance too look much at the .NET 4 / C# 4 / VS 2010 stuff). Learning the C# language is the easy part as it's basically like any other OO languages plus very cool features that it stole and improved upon from other languages. The only thing that might take some time to wrap your head around is generics when there are multiple nested type parameters involved, but the concept is similar to templates in C++, just much better. The framework itself includes so much that it's unlikely that you'll learn (or even need) more than 10% of it. Just stick with the newest technologies for each domain. E.g. use WPF instead of Windows Forms, ASP.NET MVC2 instead of ASP.NET Web Controls, etc.

 

Also, get Resharper, I would refuse to develop software without it. Some people find CodeRush also very useful, but I've never really tried it.

[BlowFish 10]

Uncanny how you answer some of my questions before I asked them one of them being forms vs WPF. Everything screams WPF except forms just seem so much easier not to mention there are loads of open source libraries whereas the WPF ones seem to be about a grand and a half a pop.

 

Yeah it became apparent to me early on that it is pretty much all about the framework rather than the language however c# seems like the best tool to generate IL.

[AgeKay 10]

Win Forms were introduced with .NET 1.1 and only updated in .NET 2.0. It's been WPF since .NET 3.0 (before .NET 4.0 there was also a .NET 3.5 release) so you can't expect much support for Win Forms in the future. Silverlight is also basically just a cut down version of WPF so it pays off to know it. WPF is initially harder to understand because it does a lot of "magic". You should know that you don't even need to use XAML to use WPF. XAML is just transformed to code during compilation anyway. I am not a big fan of books when it comes to technology (because they get outdated so quickly) but you should definitely read "Windows Presentation Foundation Unleashed" by Adam Nathan if you want to use WPF, it's the only software development book I still own. I didn't really "get" WPF and how it does its "magic" until I read that book and it's the best software development book I've read in years.

[AgeKay 10]

Btw, here are all known C# constructs:

 

Kirill Osenkov : Updated C# all-in-one file

[Mirko 10]

I am looking for a good broker with API connection to connect to Tradestation to trade FDax and other stocks with system. I like to work with Zen-Fire and Nijatrader but I learned that I have to invest 1500 bucks for a multi broker version to work with API and a broker.

 

Does anybody have an idea how I can save the 1500 or is there no way out?

[natedredd10 10]

I've already implemented the order book visualization which would show me "interesting" behaviour in the order book, unfortunately it wasn't as helpful even though it was connected to feeds that are considered "good" for trading because I was missing lots of data (I could actually see that things were not adding up).

 

Before getting too deep into this, I think you should question a lot of assumptions you are making here first...

How can you be so sure the feed was the problem?

My thinking on the book has always been polluted by that action market theory guy that runs a trading room with some quote that "the book is just noise until a trade happens"...

While obviously the book is far from "noise" if we were all trading at equal time frames and with equal data feed latency, I do think he is correct that from his time frame and data latency, thinking of the book as anything but noise is foolish.

Max Dama on Automated Trading: Current Competitive Latency

If you look at that visualization, to even start to get a clear picture of the book its going to cost you 10-50k a month for a data feed....even though latency arbs are skimming off the top and distorting things and it seems obvious that zenfire themselves are certainly the slowest of the slow on there, no matter how fast they ship it to you once they get the data...

I've started shifting my thinking that this retail obsession over "true tick data", aggregated data?? BOOOOO...

Obviously execution speed matters a ton, but since the highest end is operating on microsecond time frames..you might be better off with aggregated data that has some millisecond aggregation built in to keep you out of this fools game of trying to beat guys that are so so far ahead of the curve and just picking off the most obvious rare setups.

[natedredd10 10]

Anyway coming from a background of linked lists pointers procedures etc. I am finding it quite hard to 'unlearn' and actually start thinking in OO. I have looked at (and read chunks of) various books in series that I have liked in the past.

 

Maybe you are getting hung up on programming your own classes? Or at least on how to override a class?

I'm so glad I'm a 1999, dot com, CS dropout...OOP makes sense to build huge programs like windows, linked list pointers make sense to handle computation on machines in 1985 because memory management mattered.

Now? Who cares...

To me its a real shame CS hasn't evolved to be "the study of writing algorithms for parallel computation" at this point. Hardware has practically brute forced to meaningless the science part of 1999 CS...Academia is obviously not the most efficient market though for such things.

Don't waste your time learning to "think" in OOP...its a total waste of time given the scale of the project you will ever work on in the markets.

You should really consider learning R or matlab...steal from better programmers/bigger shut ins than you will ever be..

Great quote from the hack the market blog i posted in the other thread on here:

"“Good programmers write good programs. Great programmers steal good programs.”"

 

I would change that to "great programmers steal great algorithms"...No one here is geeky enough to "take out" anyone who has wrote a matlab or R function...

Edited July 10, 2010 by natedredd10

[Tams 20]

"“Good programmers write good programs. Great programmers steal good programs.”"

 

(OT)

 

and...

 

Good traders don't scalp.

Great traders ... (fill in the blank).

 

LOL

[AgeKay 10]

Before getting too deep into this, I think you should question a lot of assumptions you are making here first...

 

You're making a lot more assumptions than I do. My visualization is based on the order matching principles of the exchange.

 

How can you be so sure the feed was the problem?

 

The data just didn't add up. I then asked data providers and they confirmed it.

 

If you look at that visualization, to even start to get a clear picture of the book its going to cost you 10-50k a month for a data feed....

 

like you knew anything about my visualization. A direct connection with the exchange would be 10k, not more. The problem with other feeds that provide the same data is not that they don't get it, but their APIs do not support what I wanted to do.

 

 

even though latency arbs are skimming off the top and distorting things and it seems obvious that zenfire themselves are certainly the slowest of the slow on there, no matter how fast they ship it to you once they get the data...

 

did you read that in a chat room too? Reasonable latency does not matter as long as you get the complete data in the right order. The human eye/brain takes much longer to process the data than any expected latency.

 

you might be better off with aggregated data that has some millisecond aggregation built in to keep you out of this fools game of trying to beat guys that are so so far ahead of the curve and just picking off the most obvious rare setups.

 

Making a comment like that and the earlier comment just shows how little research you have done yourself and how little knowledge and experience you have with the order book and order matching algorithms. Please refrain from posting your useless comments unless your ideas are original or YOU have made the experience yourself (i.e. not other sources).

[AgeKay 10]

I'm so glad I'm a 1999, dot com, CS dropout...

 

So you're saying that you haven't programmed for 10 years and you think you can comment on what technologies someone should use?

 

OOP makes sense to build huge programs like windows, linked list pointers make sense to handle computation on machines in 1985 because memory management mattered.

 

OOP always makes sense when you need to model something. You need to use the right tools for the right job. You also still need to know data structures like linked lists to decide which data structure is the best in terms of performance / memory for what you need to do. Btw, memory management still matters, it's just a lot easier nowadays.

 

To me its a real shame CS hasn't evolved to be "the study of writing algorithms for parallel computation" at this point. Hardware has practically brute forced to meaningless the science part of 1999 CS...Academia is obviously not the most efficient market though for such things.

 

The real shame is that you have no idea what you're talking about. Academia is always many years behind current technology. Concurrency is important today but what is the point of studying its algorithms? Microsoft and other companies have already solved the problem and you bet you they do it better than a professor could teach at a college. You just have to know how and when to use them - not how they work, this applies to pretty much any technology nowadays, including data structures.

 

No one here is geeky enough to "take out" anyone who has wrote a matlab or R function...

 

You confuse programmers with mathematicians.

[Kiwi 16]

Jeez guys ... I thought that of all the conversations here this would be one of the most civilized. Why can't you bring this passion to the scammers who turn up here trying to sell us stuff?

 

An interesting discussion despite the emotion though so thanks

[BlowFish 10]

Maybe you are getting hung up on programming your own classes? Or at least on how to override a class?

 

I don't believe so. (though I mist admit I am mixing in some of the issues with 'managed frameworks') I just find it counter-intuitive to 'think' in the OOP paradigm at all. Having written code for over 30 years I like to think I am capable of writing pretty tight, efficient and re-usable stuff. I am fortunate enough to have learnt tricks and techniques (plagiarised if you like ). from some pretty smart guys. They (we) where using many of the ideas adopted by oops way back then.The key difference was that the data and routines that acted on it where not encapsulated by the compiler (I should say assembler). In fact none of the principles where forced by the compiler. Hell often there was no operating system (let alone a frame work to manage your code). Straight jackets are pretty good for preventing certain patients from harming themselves, interestingly bad programmers will still find a way to gurt themselves despite the OOP straight jacket.

[BlowFish 10]

You should really consider learning R or matlab...steal from better programmers/bigger shut ins than you will ever be...

 

I agree with AK you assume far too much about everyone else's objectives and their ability to achieve them. There are people that are doing/have done stuff that you are obviously simply reading about. you will never fully discover what can or can not be done or the best way to do it simply reading about things.

 

I can't imagine why on earth one would want to learn R or matlab. With a view to what exactly? (I have looked at their source code to steal some very specific algorithms and guess what? a) they didn't have exactly what I required & b) I found their implementations of similar function where far from optimal!). I usually know exactly what I want to do before I look at tools (though I am not adverse to playing with stuff to see how it works). Having said that I would certainly not presume to tell people what they should be doing based on stuff I had simply played with.

[natedredd10 10]

I can't imagine why on earth one would want to learn R or matlab.

 

I can't imagine why you would not. Top of the food change prototype ideas there because of speed of getting things done and testing new shit.

Here is a R package for Ralph Vince's Leverage Space Portfolio Model...

R-Forge: Leverage Space Portfolio Modeler: Project Info

functions to explore his 2009 book...Maybe its worth something, maybe its not..

Makes way more sense to me than writing your own functions to explore such an idea.

If you want to wait for public C# or C++ code then give it 10-20 years.

Anything new and interesting will come out for you to rob in R or matlab first..

Here is a 100 tick boxplot chart of some tick data from 2004 ES...Maybe there is something there, maybe there is not..

Took more time to format the data because of my ignorance than the simple call to a function in matlab to plot the data...

Edited July 24, 2010 by natedredd10

[dam5h 10]

sage math

 

nuff said

[BlowFish 10]

I can't imagine why you would not.

 

Because I am not interested in modelling portfolio risk. If I was, there are much much easier ways that are more than adequate. I would caution anyone from dicking around with modelling tools unless they have a very clear and well defined purpose. One of the reasons that traders never get going (let alone fail) is that they do not work purposefully towards specific goals or the goals are not actually appropriate to advance them along the path to 'successful trading'

 

I do wonder if you really are familiar with these tools (to be honest it doesn't sound like it)? Your "waiting 10-20 years" comment is nonsense, R is open source so it's all available now and guess what? ... if you use a licensed copy of Matlab MathWorks will provide details of algorithms and even chunks of source code.

 

As I asked before I am interested in what you are using Matlab / R for?