Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

taotree

In Pursuit of Accurate Data

Recommended Posts

So... there have been various discussions about data feeds and I'd like to see if I can find a way to ensure I have a quality one. I've heard some recommendations but...

 

Here's why I ask. Supposedly, I would expect that CQG data factory would be of high quality. This is historical data, so there are no questions about connection issues.

 

I downloaded their sample and found this example:

 

F.US.EPM08 20080104 1 0918 144375 B N N 28

F.US.EPM08 20080104 1 0918 144375 B N N 25

F.US.EPM08 20080104 1 0918 144400 A N N 34

F.US.EPM08 20080104 1 0918 144375 T N N 9

F.US.EPM08 20080104 1 0918 144375 T N N 5

F.US.EPM08 20080104 1 0918 144375 T N N 6

F.US.EPM08 20080104 1 0918 144325 B N N 8

F.US.EPM08 20080104 1 0918 144375 A N N 19

F.US.EPM08 20080104 1 0918 144325 B N N 5

 

Bid was at 25, then there were trades of 20 total and then bid went to 8.

 

I asked them if that was normal and they said "what you see is normal and happens quite often". But we have 3 trade events in a row without any bid updates, and when we do get a bid update, it doesn't add up.

 

I guess I might be able to get data from CME directly.

http://www.cmegroup.com/market-data/datamine-historical-data/

 

It would be very interesting to see if they have the same issue of numbers not all adding up.

 

What I'm getting at is... it seems incredibly difficult to find and confirm that data is accurate and I'm starting to wonder if it's possible at all.

Share this post


Link to post
Share on other sites

thats is why ultimately any back test is not necessarily going to give a 100% view of what may happen in reality.

Unfortunately collecting accurate data is costly. I used a group called CSI data for daily EOD data. They came highly recommended etc; etc; even in their data there are discrepancies and issues. We went through and investigated a few of them, did some research as to why, and then adjusted a few things.... is this hindering or helping - I dont know.

But ultimately a back test over a longer period of time a few tick differences should not make a difference. With more high frequency trading I would assume every tick can make the difference.

I guess once you get down to collecting your own data and daily reviewing it is ultimately a final resort..... I think you will unfortunately find this everywhere. It all depends on what you require it for. I have taken the view I just want data that is close enough.... and then I will take any test with a grain of salt.

Share this post


Link to post
Share on other sites
I guess once you get down to collecting your own data and daily reviewing it is ultimately a final resort..... I think you will unfortunately find this everywhere. It all depends on what you require it for. I have taken the view I just want data that is close enough.... and then I will take any test with a grain of salt.

 

I'm a little confused--I would expect historical data to be more accurate than real time data. I was collecting my own data for a while, but I now know that the feed I was using is inaccurate so... collecting data doesn't do me any good unless I know the data I'm getting is good.

 

And as for "close enough"... that's what I'm struggling with. If this type of discrepancy "happens quite often" how do I assess the quality of data? If there are confusing things happening in even the best quality of data available (which I do not know if that's the case), how does one differentiate between the best, the "close enough" and the way off?

Share this post


Link to post
Share on other sites

By the way... It appears that you can get a direct internet feed to CME for $500/month but... does anyone know of a data distributor that just directly passes on CME data for less than $500/month? Then it would be easy to verify that we're getting the best data (can purchase some historical CME data directly and compare it to confirm). Comparing to data feeds that change the format and such is difficult because it requires manipulation and even interpretation.

Share this post


Link to post
Share on other sites

historical data is drawn live data so now you understand why there are issues collecting data then - and why accurate data is expensive.... and all the things that can go wrong with it.....there should be no confusion.

 

There are plenty of discussions already about issues with tick data. with daily data, there should be less issues but there still are....

I go close enough as i dont have the time, resources or inclination to even wonder down the path of data collection.

Share this post


Link to post
Share on other sites

 

What I'm getting at is... it seems incredibly difficult to find and confirm that data is accurate and I'm starting to wonder if it's possible at all.

 

Me too. My hunch is that exchanges aggregate data at times. CME certainly reserve the right to, though I must be honest, the last time I looked for the document that mentioned it I could not find it.

Share this post


Link to post
Share on other sites

Oh, this is fun. So...

 

First of all, let me say that when I looked at the other data it took me less than a couple minutes to find data that didn't line up. So either it does happen all over the place, or I just got really lucky. Sorry... not rigorous check yet since I'm still just exploring.

 

So, I took a look at sample data direct from the CME. I spent a good deal longer finding ones that matched up--maybe 10 minutes? not sure, but then I found this:

 

From: Top of Book

 

2010011509564006628780EES F100300900 201132252A M M100115

2010011509564006628780EES F100300157 201132002B M M100115

2010011509564006628790EES F100300038 201132002 100115

2010011509564006628800EES F100300900 201132252A M M100115

2010011509564006628800EES F100300120 201132002B M M100115

 

What this is showing is:

First: ask = 900, bid = 157

Then a trade of 38

Then: ask = 900, bid = 120

 

Oops, we gained an extra one. So... there is at least one example of the CME's own data not lining up. So apparently the CME doesn't require them to match up.

 

By the way... it's not that a 1 difference in the bid is going to make or break a strategy. It's that these discrepancies are making it difficult for me to assess the accuracy of a data feed.

 

Another one:

 

2010011509564006628980EES F100300870 201132252A M M100115

2010011509564006628980EES F100300098 201132002B M M100115

2010011509564006628990EES F100300001 201132002 100115

2010011509564006629000EES F100300870 201132252A M M100115

2010011509564006629000EES F100300096 201132002B M M100115

 

And another:

 

2010011509564006629250EES F100300083 201132002A M M100115

2010011509564006629250EES F100301185 201131752B M M100115

2010011509564006629260EES F100300001 201132002 100115

2010011509564006629270EES F100300076 201132002A M M100115

2010011509564006629270EES F100301183 201131752B M M100115

 

Found those two quickly after the first... Okay... now time to ask CME directly about this and see if I can dig into what's going on.

 

At this point, I'm going to assume then that probably the only possible way of assessing the accuracy of a data feed is to collect the data, purchase CME data for the same day, and run a check to ensure every tick was received in the same order and context.

Share this post


Link to post
Share on other sites

I posted about this exact issue a month or two ago on here. I spoke with IQ feed a week or two ago and they claimed two important things:

 

1. Timestamps come from the exchange -- the exchange provides the time stamps so that prevents market events from being *logged* out of order (but it's still possible for you to *receive* events out of chronological order...not a major problem, easy to sort out.

 

2. They claim that they never filter data (at least for CME, that's all I asked about). They report the complete datafeed from CME.

 

I might be missing something, but that sounds as good as having a direct line to CME. That's probably why Fulcrum recommends IQ Feed, but then again, he's using a $2k/month feed so he's obviously getting something else too (and I'd love to know what's worth the extra $1800 / month :-)).

 

Can someone please send me just 1 day of ES data (anytime after February) from your feed. I'd be especially interested in a sample from IQ Feed as that's the vendor I am considering, but also interested in comparing against other providers and I promise to document my findings here for the benefit of everyone.

Share this post


Link to post
Share on other sites

I dont collect intraday data so cant help there, but I initially collected a lot of EOD data from CSI Unfair Advantage - they have a good range of products that can be bought so you can get EOD data back from 20 years ago. they were recommended by others.

The data is generally very good, a few hiccups, but as mentioned above that might because a few days get out of sync. The real advantage they offered was a way to back adjust continuous contracts and a fair bit of flexibility to manipulate the data into the manner we required. They also offer stocks - but I did not look at them.

But as stated for me something that is close enough was all I required....so I wont be much help sorry....but would be very interested in any final conclusions of the thread.

Share this post


Link to post
Share on other sites

 

2. They claim that they never filter data (at least for CME, that's all I asked about). They report the complete datafeed from CME.

 

g

 

Filtering and aggregating/coalescing are not quite the same thing. They always used to reserve the right to but in the absence of current evidence I guess we can discount that. :)

 

When they where talking about 'data' was that the full order book?

Share this post


Link to post
Share on other sites
I don't know of any feeds which supply the entire order book. They said that they don't filter their market depth data.

 

I assume that's all he meant by order book--market depth to the same level as the exchange gives. For eMinis that's 10 levels, for other products typically 5 or whatever.

Share this post


Link to post
Share on other sites
I don't know of any feeds which supply the entire order book. They said that they don't filter their market depth data.

 

Yeah that's me being sloppy what I meant was there limit order book to however many levels are reported (or market depth data).

Share this post


Link to post
Share on other sites

I am also in Pursuit of Accurate Data for the purpose of order book analysis and I am seriously considering to go with CQG. taotree, do you have any updates?

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Topics

  • Posts

    • Date: 7th April 2025.   Asian Markets Plunge as US-China Trade War Escalates; Wall Street Futures Signal Further Turmoil.   Global financial markets extended last week’s massive sell-off as tensions between the US and its major trading partners deepened, rattling investors and prompting sharp declines across equities, commodities, and currencies. The fallout from President Trump’s sweeping new tariff measures continued to spread, raising fears of a full-blown trade war and economic recession.   Asian stock markets plunged on Monday, extending a global market rout fueled by rising tensions between the US and China. The latest wave of aggressive tariffs and retaliatory measures has unnerved investors worldwide, triggering sharp sell-offs across the Asia-Pacific region.   Asian equities led the global rout on Monday, with dramatic losses seen across the region. Japan’s Nikkei 225 index tumbled more than 8% shortly after the open, while the broader Topix fell over 6.5%, recovering only slightly from steeper losses. In mainland China, the Shanghai Composite sank 6.7%, and the blue-chip CSI300 dropped 7.5% as markets reopened following a public holiday. Hong Kong’s Hang Seng Index opened more than 9% lower, reflecting deep concerns about escalating trade tensions.           South Korea’s Kospi dropped 4.8%, triggering a circuit breaker designed to curb panic selling. Taiwan’s Taiex index collapsed by nearly 10%, with major tech exporters like TSMC and Foxconn hitting circuit breaker limits after each fell close to 10%. Meanwhile, Australia’s ASX 200 shed as much as 6.3%, and New Zealand’s NZX 50 lost over 3.5%.   Despite the escalation, Beijing has adopted a measured tone. Chinese officials urged investors not to panic and assured markets that the country has the tools to mitigate economic shocks. At the same time, they left the door open for renewed trade talks, though no specific timeline has been set.   US Stock Futures Plunge Ahead of Monday Open   US stock futures pointed to another brutal day on Wall Street. Futures tied to the S&P 500 dropped over 3%, Nasdaq futures sank 4%, and Dow Jones futures lost 2.5%—equivalent to nearly 1,000 points. The Nasdaq Composite officially entered a bear market on Friday, down more than 20% from its recent highs, while the S&P 500 is nearing bear territory. The Dow closed last week in correction. Oil prices followed suit, with WTI crude dropping over 4% to $59.49 per barrel—its lowest since April 2021.   Wall Street closed last week in disarray, erasing more than $5 trillion in value amid fears of an all-out trade war. The Nasdaq Composite officially entered a bear market on Friday, sinking more than 20% from its recent peak. The S&P 500 is approaching bear territory, and the Dow Jones Industrial Average has slipped firmly into correction territory.   German Banks Hit Hard Amid Escalating Trade Tensions   German banking stocks were among the worst hit in Europe. Shares of Commerzbank and Deutsche Bank plunged between 9.5% and 10.3% during early Frankfurt trading, compounding Friday’s steep losses. Fears over a global trade war and looming recession are severely impacting the financial sector, particularly export-driven economies like Germany.   Eurozone Growth at Risk   Eurozone officials are bracing for economic fallout, with Greek central bank governor Yannis Stournaras warning that Trump’s tariff policy could reduce eurozone GDP by up to 1%. The EU is preparing retaliatory tariffs on $28 billion worth of American goods—ranging from steel and aluminium to consumer products like dental floss and luxury jewellery.   Starting Wednesday, the US is expected to impose 25% tariffs on key EU exports, with Brussels ready to respond with its own 20% levies on nearly all remaining American imports.   UK Faces £22 Billion Economic Blow   In the UK, fresh research from KPMG revealed that the British economy could shrink by £21.6 billion by 2027 due to US-imposed tariffs. The analysis points to a 0.8% dip in economic output over the next two years, undermining Chancellor Rachel Reeves’ growth agenda. The report also warned of additional fiscal pressure that may lead to future tax increases and public spending cuts.   Wall Street Braces for Recession   Goldman Sachs revised its US recession probability to 45% within the next year, citing tighter financial conditions and rising policy uncertainty. This marks a sharp jump from the 35% risk estimated just last month—and more than double January’s 20% projection. J.P. Morgan issued a bleaker outlook, now forecasting a 60% chance of recession both in the US and globally.   Global Leaders Respond as Trade Tensions Deepen   The dramatic market sell-off was triggered by China’s sweeping retaliation to a new round of US tariffs, which included a 34% levy on all American imports. Beijing’s state-run People’s Daily released a defiant statement, asserting that China has the tools and resilience to withstand economic pressure from Washington. ‘We’ve built up experience after years of trade conflict and are prepared with a full arsenal of countermeasures,’ it stated.   Around the world, policymakers are responding to the growing threat of a trade-led economic slowdown. Japanese Prime Minister Shigeru Ishiba announced plans to appeal directly to Washington and push for tariff relief, following the US administration’s decision to impose a blanket 24% tariff on Japanese imports. He aims to visit the US soon to present Japan’s case as a fair trade partner.   In Taiwan, President Lai Ching-te said his administration would work closely with Washington to remove trade barriers and increase purchases of American goods in an effort to reduce the bilateral trade deficit. The island's defence ministry has also submitted a new list of US military procurements to highlight its strategic partnership.   Economists and strategists are warning of deeper economic consequences. Ronald Temple, chief market strategist at Lazard, said the scale and speed of these tariffs could result in far more severe damage than previously anticipated. ‘This isn’t just a bilateral conflict anymore — more countries are likely to respond in the coming weeks,’ he noted.   Analysts at Barclays cautioned that smaller Asian economies, such as Singapore and South Korea, may face challenges in negotiating with Washington and are already adjusting their economic growth forecasts downward in response to the unfolding trade crisis.           Oil Prices Sink on Demand Concerns   Crude oil continued its sharp slide on Monday, driven by recession fears and weakened global demand. Brent fell 3.9% to $63.04 a barrel, while WTI plunged over 4% to $59.49—both benchmarks marking weekly losses exceeding 10%. Analysts say inflationary pressures and slowing economic activity may drag demand down, even though energy imports were excluded from the latest round of tariffs.   Vandana Hari of Vanda Insights noted, ‘The market is struggling to find a bottom. Until there’s a clear signal from Trump that calms recession fears, crude prices will remain under pressure.’   OPEC+ Adds Further Pressure with Output Hike   Bearish sentiment intensified after OPEC+ announced it would boost production by 411,000 barrels per day in May, far surpassing the expected 135,000 bpd. The alliance called on overproducing nations to submit compensation plans by April 15. Analysts fear this surprise move could undo years of supply discipline and weigh further on already fragile oil markets.   Global political risks also flared over the weekend. Iran rejected US proposals for direct nuclear negotiations and warned of potential military action. Meanwhile, Russia claimed fresh territorial gains in Ukraine’s Sumy region and ramped up attacks on surrounding areas—further darkening the outlook for markets.   Always trade with strict risk management. Your capital is the single most important aspect of your trading business.   Please note that times displayed based on local time zone and are from time of writing this report.   Click HERE to access the full HFM Economic calendar.   Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding of how markets work. Click HERE to register for FREE!   Click HERE to READ more Market news.   Andria Pichidi HFMarkets   Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past performance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in Leveraged Products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.
    • AMZN Amazon stock watch, good buying (+313%) toi hold onto the 173.32 support area at https://stockconsultant.com/?AMZN
    • META stock watch, local support and resistance areas at 507.48, 557.84 at https://stockconsultant.com/?META
    • TMUS T-Mobile stock, watch for a top of range breakout at https://stockconsultant.com/?TMUS
    • KULR KULR Technology stock watch, pullback to 1.25 triple support area with bullish indicators at https://stockconsultant.com/?KULR
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.