Zenfire and DTN Feed Different?

[FulcrumTrader 10]

FT you are probably right about NT, currently it is a square peg and we have a round hole. I am getting some promising results with Zen and Multicharts though I need to do some further tests (purposely overloading MC) to be sure.With MC DTN.IQ is an option too.

MC seems like a charting platform with a lot of potential.....glad to see you are doing some testing!

 

I myself over the next few months will start doing some testing of the TradeVec platform and charting, as this group may be adding Cumulative Delta volume studies as a part of their product. If TradeVec adds Cumulative Delta Volume studies (as candlesticks!!!) I will do back flips.....LOL! :rofl:

 

A good broker friend of mine has been working with this group and what I am seeing so far is very promising! They actually have an informational webinar planned for this Tuesday right at 3:30 pm US Central time (at the Globex close time)........

 

TradeVec Webinar

 

The date in the link shows Wednesday but that has been set for Tuesday now.....so Tuesday at 3:30 pm is the TradeVec info webinar.

Edited February 14, 2010 by FulcrumTrader

[SNYP40A1 10]

I think they probably will if they are at the same price. For example 17 1/4 bid 17 3/4 asked if a limit order to buy 1 @ 17 1/2 and a limit order to sell 1 @ 17 1/2 came in at the same time they would be matched and filled.

 

...

 

In your example if a limit buy and a limit sell both at 10109 both arriving simultaneously they would simply be matched and reported as trade @ 10109? If they where put on the book it would essentially be crossed with best bid 10109 and best ask 10109? Thats my guess anyway.

 

That's a good question. They would definitely be matched and filled against each other. But would they be printed to the exchange? If they are not printed to the exchange, then that could explain the example about 10109 that I posted above. For completeness, I think the limit orders should update the books, reporting 1 new best bid and 1 new best ask both at 17 1/2, the trade should be reported, then the 17 1/2 levels should be updated back to 0 orders for both bid and ask and all of these actions would be reported under the same timestamp. This would show exactly what happened. But it would also generate 5 extra ticks that could be inferred if it is the case that overlapping limit orders are the only trades that can execute without affecting the book. If anyone knows the answer to this situation, please let us know. And even more important, if you know where these rules are posted, please let me know.

 

In the early days of electronic exchanges surprisingly few supported more than a couple of basic order types. Often a market order would be converted to a limit order well inside the market. I guess things are more sophisticated nowadays.

 

I was not looking at the market during these times, but it seems that the only time a market order should be converted to a limit order is when there are no limit orders at any price level to match the trade. Although I am not sure how that could happen. What price would they assign to the new limit order?

[BlowFish 10]

I seem to remember that Globex was one of the exchanges that did not support market orders natively in days gone by. My broker (IB) still simulates them. What Globex now provide (according to the reference guide) is market order with protection. I am not sure when they introduced this order type but think it must be fairy recently. Before this order type was available your broker would simulate a market order by placing a limit order well inside the market. Market order with protection is essentially a limit order (with a wide limit).

 

Market with Protection

Market orders at CME Group are

implemented using a “Market with

Protection” approach. Unlike a

conventional Market order, where

customers are at risk of having their

orders flled at extreme prices, Market

with Protection orders are flled within a

predefned range of prices (the protected

range). The protected range is typically the

current best bid or ofer, plus or minus

50 percent of the product’s No Bust Range.

If the entire order cannot be flled within

the protected range, the unflled quantity

remains on the book as a Limit order at

the limit of the protected range.

[SNYP40A1 10]

I seem to remember that Globex was one of the exchanges that did not support market orders natively in days gone by. My broker (IB) still simulates them. What Globex now provide (according to the reference guide) is market order with protection. I am not sure when they introduced this order type but think it must be fairy recently. Before this order type was available your broker would simulate a market order by placing a limit order well inside the market. Market order with protection is essentially a limit order (with a wide limit).

 

Market with Protection

Market orders at CME Group are

implemented using a “Market with

Protection” approach. Unlike a

conventional Market order, where

customers are at risk of having their

orders flled at extreme prices, Market

with Protection orders are flled within a

predefned range of prices (the protected

range). The protected range is typically the

current best bid or ofer, plus or minus

50 percent of the product’s No Bust Range.

If the entire order cannot be flled within

the protected range, the unflled quantity

remains on the book as a Limit order at

the limit of the protected range.

 

I think we are looking at the same reference:

 

http://www.cmegroup.com/globex/files/GlobexRefGd.pdf

 

I just found that today. A market order with protection will be executed immediately and will not create a new order on the books like a limit order does. The stop order are essentially limit orders too, except that they remain hidden form the market until the trigger point is hit. I could be wrong but for the equity futures, I thought I remembered reading that there were no stop price bands. The only other two orders are minimum execution, which is essentially a market order that only executes if it can be filled immediately. Hidden quantity and request for cross which both appear to be limit orders.

[bathrobe 10]

Hi,

 

I use CVD plotted in Investor R/T using Zen-fire and DTN MA for backfill. Because of this is my data different than IQ Feed?

 

Also, does anyone know if the CVD study in X_Study is accurate?

 

Chris

[BlowFish 10]

One last thing Snyp (I think we have done this to death!) there is a bit of info on matching here Introduction to CME Globex you need to use the pulldown menu and click matching algorithms. To be honest it doesn't go into huge detail and I expect you found it already anyway. I think Globex is 'vanilla' FIFO which has the least information.

[FulcrumTrader 10]

Hi,

 

I use CVD plotted in Investor R/T using Zen-fire and DTN MA for backfill. Because of this is my data different than IQ Feed?

 

Also, does anyone know if the CVD study in X_Study is accurate?

 

Chris

 

Best to just use DTN for your charting feed with Inv RT. CVD in XStudy is OK but you are not able to get candlesticks for the CD plot.

[SNYP40A1 10]

One last thing Snyp (I think we have done this to death!) there is a bit of info on matching here Introduction to CME Globex you need to use the pulldown menu and click matching algorithms. To be honest it doesn't go into huge detail and I expect you found it already anyway. I think Globex is 'vanilla' FIFO which has the least information.

 

Yup, I think I have read everything on the CME site and I can't find anything describing what and how market events are reported. There's a continuation of this discussion someplace else, you can find it by googling "Ninja Zen no more unfiltered data?". I am pretty sure Fulcrum is posting there too under another alias ;-). The most relevant document that I found which describes what events are reported is this:

 

http://www.cmegroup.com/globex/files/GlobexRefGd.pdf

 

But there are still some cases like the one mentioned a page or two back that I am still unsure about.

[paolfili 10]

Hi Fulcrum.

Can you help to see clearly in the question?

 

1)Now we assume Zenfire Bid/Ask sequence lies on the type of trade.(observing the API I can confirm the taotree suggestion on the broken order of the timestamps).

So GomCD is unusable from the ZenFire feed.("Ticker plant" absence)

2)The DTN.iq feed for NinjaTrader can be ok with GomCD.

But why there 's the need of a good newer PC with very good internet (cable modem at a minimum)?

 

The DTN.iq data is TCP flow (insted of the ZenFire data that is UDP and TCP only for orders) so I' don' t see the need of a performance PC and good internet. (I' ve got a 7M adsl).

The only problem is some latency in the arrival of the data while the order must be preserved also in a slow connection.

IMHO the only important thing is the correct syncronism in TRADE and BEST_BID/BEST_ASK flow.

 

Another different kind of problem is the weight and the manipulation of (big amount of) data from the NinjaTrader platform.

So the limit here can be not the liability of CumulativeDelta offered by GomCD , but the number of charts you can manage with NT.

 

Please,correct me, if I'm wrong.

 

Thanks

 

P.S.

I think NinjaTrader is (for me) a very good platform .

So is important to undestand where is the problem and see if NinjaTrader/DTN.iq

(without the 4 weeks historical from the InvestorRT/DTN.iq) can be a serious way to look a the Cumulative Delta (using GomCD).

 

 

 

 

Yes.....Rithmic/Zenfire feed is NOT working for proper BID/ASK volume studies. If you do your research and see how thorough DTN.IQ manages and time stamps their feed, you will then see why many of these broker supplied feeds can't touch the capability/reliability of DTN.IQ feed.

 

I do have an update on this feed situation from some recent testing. If you have a very good newer PC with very good internet (cable modem at a minimum) we have a client who is working with NT connected to DTN.IQ feed and he is NOT getting any data drops now. So a trader can then track the GomCD with NT connected to DTN.IQ feed.......the only capability you would not have is historical lookback (like you would have with DTN.IQ feed connected to Investor RT Pro for up to a 4 week backfill). If you wanted historical lookback in NT with DTN.IQ you would need to save your own data which can be done with the GomCD Recorder function. So this was good to see that one of my clients has been able to get NT to work now with DTN.IQ feed for Cumulative Delta tracking......cool!

[SNYP40A1 10]

TCP/IP in itself will not guarantee that you receive every packet if it's the case that your computer / connection cannot keep up with the amount of data thrown at it. I think that's all he was trying to explain. It's only more robust than UDP because if it's the case that your system misses a packet not due to lack of bandwidth / processing, but something else, then the packets will be resent. For this reason, either way, you can't rely on when the packets were received for the timestamped, the feed itself must timestamp every tick at the exchange.

[BlowFish 10]

TCP offers error correction, flow control and sequence numbering. Of course if your infrastructure is totally saturated the whole time you are stuffed, however if it is just when data is 'fast' things should be recovered in the correct sequence. Of course time stamps will be off if time stamped by the client but for delta type calcs it is sequencing that is important.

 

If Zen uses UDP that could easily explain why some see holes in the data.

[gooni 10]

There are plenty of comms protocols that add reliability and sequencing on top of UDP - in fact this is usually the preferred approach for high volume data distribution.

 

The beauty of IQFeed's approach is that they (apparently) embed the current bid and ask along with each trade, rather than requiring this to be rebuilt at the client end. For the vast majority of retail use, this is sufficient and quite significantly more reliable.

[FulcrumTrader 10]

There are plenty of comms protocols that add reliability and sequencing on top of UDP - in fact this is usually the preferred approach for high volume data distribution.

 

The beauty of IQFeed's approach is that they (apparently) embed the current bid and ask along with each trade, rather than requiring this to be rebuilt at the client end. For the vast majority of retail use, this is sufficient and quite significantly more reliable.

There you go sir.....you are absolutely correct.

 

It should also be obvious the game has changed since the CME increased the data output last November......so the need for a newer PC with good internet is just common sense. The information needed to play the "Cumulative Delta" game properly is out there so I will leave it at that.

[FulcrumTrader 10]

I am pretty sure Fulcrum is posting there too under another alias ;-). The most relevant document that I found which describes what events are reported is this:

 

http://www.cmegroup.com/globex/files/GlobexRefGd.pdf

 

But there are still some cases like the one mentioned a page or two back that I am still unsure about.

Yes, most know that I am AMT4SWA over at that wondrous kind to one another land of ET......LOL!

[SNYP40A1 10]

The beauty of IQFeed's approach is that they (apparently) embed the current bid and ask along with each trade, rather than requiring this to be rebuilt at the client end. For the vast majority of retail use, this is sufficient and quite significantly more reliable.

 

If it's the case that the bid / ask always changes with each trade, those updates would have to be communicated anyway so it could save bandwidth. Is it possible to execute a trade without changing the bid/ask?

[paolfili 10]

1)TCP is a lossless protocol.(wikipedia/google is your friend).

2)The timestamp issue a well know problem (for quants).

If you have a high quality link to Exchange you can timestamp the data and then spread

to your customers.(As every timestamp feed provider does)

 

 

 

 

TCP/IP in itself will not guarantee that you receive every packet if it's the case that your computer / connection cannot keep up with the amount of data thrown at it. I think that's all he was trying to explain. It's only more robust than UDP because if it's the case that your system misses a packet not due to lack of bandwidth / processing, but something else, then the packets will be resent. For this reason, either way, you can't rely on when the packets were received for the timestamped, the feed itself must timestamp every tick at the exchange.

Edited February 16, 2010 by paolfili

[paolfili 10]

The UDP design on ZenFire data is not reliable on high volume data distribution.

(from ZenFire desk).

Can you mention some reliable UDP protocols (in the sense of packet sequence number and frame reconstruction) that is better than TCP in the trade-off packet_lenght/information?

The "reliables" UDP protocols are used when (for some reason) you couldn' t use TCP connection.Otherways TCP is the way.

If you have a very high-quality link to the UDP data provider probably your packet loss

percentuage will be very very low.

But it depends from the kind and the number of symbols and type of data.

The same issue is for you "elaboration unit" (Pc, cluster,mainframe,etc..) .It' always a trade-off problem.

 

The main difference between ZenFire(Rithmic)/DTN.iq feed framework is the quantity of data.

Any Platform (NinjaTrader/Inverstor RT/etc) relies on API from the feed provider.

From what I' ve seen the DTN API can offer only BEST_BID/BEST_ASK, and cumulative VOLUME_BID/VOLUME_ASK.

Zenfire API can offer all level bid/ask change.This is a lot of data (more than DTN).

If the Zenfire book information will be reliable you could explore in very depth way the futures DOM (probably some quants are doing this now).

But probably the "price" for the possibility of this huge amount of data is the UDP design of the feed.

Another possibility is to re-organize the design of the feed.

(Any Zenfire guys hearing ...? )

 

 

There are plenty of comms protocols that add reliability and sequencing on top of UDP - in fact this is usually the preferred approach for high volume data distribution.

 

The beauty of IQFeed's approach is that they (apparently) embed the current bid and ask along with each trade, rather than requiring this to be rebuilt at the client end. For the vast majority of retail use, this is sufficient and quite significantly more reliable.

[BlowFish 10]

There are plenty of comms protocols that add reliability and sequencing on top of UDP - in fact this is usually the preferred approach for high volume data distribution.

 

The beauty of IQFeed's approach is that they (apparently) embed the current bid and ask along with each trade, rather than requiring this to be rebuilt at the client end. For the vast majority of retail use, this is sufficient and quite significantly more reliable.

 

Quite. The reason that people use TCP is because it is done for them rather than them having to handle session and transport stuff. Hell you can stuff everything into IP packets and handle it all yourself (thats what I'd do but I'd probably write it all in assembly language too )

 

In truth you don't even need bid and ask information you just need to know if the last trade was @bestbid or @bestask. This can be achieved with a pair of flags.

[BlowFish 10]

There you go sir.....you are absolutely correct.

 

It should also be obvious the game has changed since the CME increased the data output last November......so the need for a newer PC with good internet is just common sense. The information needed to play the "Cumulative Delta" game properly is out there so I will leave it at that.

 

Agree that a good connection makes sense Good does not need to be fast however. Most ISP's have off short outages and re syncing of the line, especially if your noise profile is optimised for speed. You would probably not notice normally. The PC is less important (though I would not run a 386 with Win 95 ).

 

Bandwidth and processing oomph have far outstripped any increase from the CME. Moores Law is not far off....tech stuff grows exponentially.

[paolfili 10]

You can do it using the OS call managment on IP (Stevens books docet...anyway I prefer C).

But TCP has several other goodies and is always an "observed object" (from the academic network community).

What about syn-flood like,Initial Sequence Number attack like ,etc in a your custom IP protocol?

Moreover the hardware (router/bridge) and software(kernel driver) are nowadays very tight to standardized protocols.

(Why reinventing the wheel?)

Changing approach to the problem I think the Future in the feed world are the FIX oriented protocols.

The present (for retails as we are) is for now learning the datafeed nuances.

 

 

Quite. The reason that people use TCP is because it is done for them rather than them having to handle session and transport stuff. Hell you can stuff everything into IP packets and handle it all yourself (thats what I'd do but I'd probably write it all in assembly language too )

 

In truth you don't even need bid and ask information you just need to know if the last trade was @bestbid or @bestask. This can be achieved with a pair of flags.

Edited February 16, 2010 by paolfili

[paolfili 10]

This is a really important issue.

What does it mean a good connection?

 

Do you think the re-syncing of the line is the most important parameter?

A re syncing of line determines the lack of some packets?

I think will be really interesting determining a reliable test (I can do my via hping scripting).

Can you suggest me any other meaningful test to do a network connection to

determine the goodness in high frequency trading context?

 

Thanks

 

Agree that a good connection makes sense Good does not need to be fast however. Most ISP's have off short outages and re syncing of the line, especially if your noise profile is optimised for speed. You would probably not notice normally. The PC is less important (though I would not run a 386 with Win 95 ).

 

Bandwidth and processing oomph have far outstripped any increase from the CME. Moores Law is not far off....tech stuff grows exponentially.

[SNYP40A1 10]

1)TCP is a lossless protocol.(wikipedia/google is your friend).

 

As I said before:

 

TCP/IP in itself will not guarantee that you receive every packet if it's the case that your computer / connection cannot keep up with the amount of data thrown at it.

 

If your computer / connection cannot keep up with the real time packet stream, there will be packet loss. Example, try receiving a 10 Mbps real-time stream on a 14.4 kbps dial-up connection. TCP can't perform magic.

[paolfili 10]

The ubiquitous *DSL is nowadays the standard.

Huge data flows (10Mbps) are well beyond the acceptable managing limit of the retail traders,as we are.

(also in term of usability).

"Packet loss" in the Network Communication Literature (and in the Network community) has another meaning.

 

 

As I said before:

 

 

 

If your computer / connection cannot keep up with the real time packet stream, there will be packet loss. Example, try receiving a 10 Mbps real-time stream on a 14.4 kbps dial-up connection. TCP can't perform magic.

Edited February 17, 2010 by paolfili

[BlowFish 10]

 

What about syn-flood like,Initial Sequence Number attack like ,etc in a your custom IP protocol?

Moreover the hardware (router/bridge) and software(kernel driver) are nowadays very tight to standardized protocols.

(Why reinventing the wheel?)

 

SYN is a TCP session parameter. If you are not using TCP you will be immune to a SYN Flood (as you are not using TCP you will not reply to SYN requests). By not using TCP you will be immune to attacks that rely on TCP protocols.

 

To route you need IP and that's it. Nowadays of course routers do all sorts of higher level 'stateful' stuff and look deep into packets to do higher level 'clever' things. (Like traffic shaping and prioritisation for example). None of that information is needed to route.

 

Why re-invent the wheel? I am certainly not advocating that, it would not make sense for a lot of people. However if you do not require all the bells and whistles provided by TCP (a generic protocol after all) but do require some simple transport control and session management then it might make sense.

 

Still this goes way beyond the discussion here.

[SNYP40A1 10]

The ubiquitous *DSL is nowadays the standard.

Huge data flows (10Mbps) are well beyond the acceptable managing limit of the retail traders,as we are.

(also in term of usability).

"Packet loss" in the Network Communication Literature (and in the Network community) has another meaning.

 

It was a general example and my point is obvious. If your connection cannot keep up with the stream, packets will be lost.