Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

darthtrader

Archiving Tick Data?

Recommended Posts

Does anyone on here archive tick data into a database for future use?

With DTN they give you 8 days but it just seems like a waste to not store this information.

What I would love is a program that can store the info into a database that is open as possible so that it could be used no matter what software I change to in the future.

Anyone into this stuff? Maybe a straight out access database or sql db?

Share this post


Link to post
Share on other sites

Darth, you could easily code that with NT, performance would get slightly worse because of having to use real time indicators. Either use market analyzer or 1 tick charts.

I'm doing a variation of this to export 1min price/volume data for a volume based price/volume histogram. But have been to lazy to code the histogram so far :D.

 

Edit: I complicated things a bit, NT can export tick data just fine, but I think it will only do some if it's already in the DB.

Edited by Sparrow

Share this post


Link to post
Share on other sites

MySQL, or PostgreSQL would be an option for storing such data. Maybe also SQL Server Express or Compact but no idea if they have got space constraints or are really suitable for the job.

I'd go with MySQL because it's more optimized for speed and the stuff you'd like to do is fairly simple.

 

I suppose the tables could grow quite large but with today's hdds, it shouldn't be a problem.

 

If you get the data into some kind of csv format, you'd be able to load the data like this into a MySQL db.

 

I think it would be tedious work if you cannot automate the export/import. Probably there's a way to this but none of the limited amount of trading software I know allows command line export of data.

Edited by Sparrow

Share this post


Link to post
Share on other sites

Hey guys,

I think Ninja can actually do this to some degree if you check "store realtime bar data" in options.

My interest though is to get into something more standard/open incase I'm not even using Ninja 3 years down the line. I guess in the meantime the market scanner in ninja would be the way to go.

I found some interesting post on elite searching for SQL. One thing that was interesting that I didn't know is that I guess relational databases aren't really ideal for time series data. It was mentioned a vector database is the way to go.

Someone posted a link to an open source vector DB and a commercial version

http://kx.com/

http://hdf.ncsa.uiuc.edu/HDF5/index.html

 

Could be interesting. MySQL would be cool but if I set this project up I don't want to hit a brick wall 3 years down the line because the DB is so huge and not using the right tool for the job.

Share this post


Link to post
Share on other sites

Interesting Darth, however it's questionable if storing the data in a database is necessary at all. If you intend to generate statistical studies with secondary tools from the data it should be in a format that is widely used. The only thing you'd need to worry about is bringing it into the right format (5min/monthly/etc.)

In case you want to write your own studies, a database makes sense, otherwise text files are more efficient which you might choose to compress or not.

 

It could also be better to just purchase the data and save yourself all the hassle.

Share this post


Link to post
Share on other sites

Ahh yea, I ran across that idea tonight too. That actually sounds like what the guys who are decent programmers were doing. Just throwing it in a binary/text file and reading in straight away.

Of course buying it would probly be alot better time spent. Nothing though that I really need right now, I would just like to learn this stuff to have a spare computer capturing data for the hell of it I guess.

Share this post


Link to post
Share on other sites

I used tradestation 2000i to store tick data for the longest time. I still maintgain that data in multicharts though it is less manageable. There are several off the shelf packages that have pretty robust data management facilities. (Neoticker instantly springs to mind).

Share this post


Link to post
Share on other sites

I might have to just buy Neoticker at some point. I've read its pretty hard to top as far as its database goes. Still the only issue is see with that is running into something where you want to change software down the road and have this massive database of tick data that can't be converted and would cost far too much to buy.

 

In the meantime I'm just going with flat csv files and will progress as needed. From what i've read about sql is that its just overkill on the overhead and not really ment for time series data.

 

This DB was mentioned on elite, its for storing massive amounts of scientific data and might be interesting, free and open source.

http://www.hdfgroup.org/products/index.html

Share this post


Link to post
Share on other sites

You may want to consider breaking up the data into a days worth of tick data per instrument and use the date as part of the filename. Ensign recently switched to a tree structure and individual files like this. Its easy to maintain, and hard drives are cheap.

 

-mp

Share this post


Link to post
Share on other sites
From what i've read about sql is that its just overkill on the overhead and not really ment for time series data.

http://www.hdfgroup.org/products/index.html

 

SQL comes in many flavors, so just to say that SQL isn't for time series is overshooting the target. SQL can surely handle time data, in fact it handles time data much quicker and better than anything else on the market - but then again... ;) It depends on your needs, if you want fast access agregated data down to 0ms then you need to code it in the businesslogic layer, this has nothing to do with the storage. If you need historical aggregated data you should look for multidimensional datastorage, but still SQL.

 

If you need quick writing? Wonder who does, but lets discuss, then you'll need Oracle, but MSSQL could be a better choice if you can control your writings in the datalayer ;)

 

So... just to say that SQL is a bad choice is a bit ignorant :) What you've read could be very wrong!

Share this post


Link to post
Share on other sites

Has anyone put together an historical tick database?

I am working in MY SQL as was wondering if anyone has any thoughts on constructing such a task?

I intend on being able to read various file formats and converting to approriate format and also integrating levels of granularity ex: convert to X minutes I believe that there might be an intersting study to accomplish relative to the compositon of tick data in comparison to longer term studies.

I believe that the compositon of longer term bars might place trades/fills in different light and that observations of these fills relative to when the trade occured might yield interesting information

Share this post


Link to post
Share on other sites
Has anyone put together an historical tick database?

Have you made any space estimations?

 

For instance, 50.000ticks in a share pr. day, a tick consists of date, time, price and volume.

Date = 8 byte

Time = 8byte

Price = 4byte

Volume = 4byte

 

Date and Time should possible be normalized, which then goes down to 4byte with a pointer to 8 byte.

 

Roughly speaken

1 record of TickData = 4 + 4 + 4+ 4 and the related tables boiles down to 1byte approximately 17 bytes then.

 

50.000 * 17bytes = 850 kbytes pr. day, if you then wants to record 100 shares for a year it would require

850 * 100 * 240 = 20400 mb

 

I don't know if 1000 shares or 50.000 ticks is the reality?

 

Oh btw... I've been working with a database collecting flightinformation, we collected around 2gb of data each day with no problems, but our hardware did had a costprice of $400.000 - anything is possible I guess! The biggest problem we did experience was to get it out again and also if the Internet suddenly went down.

Cheers :)

Share this post


Link to post
Share on other sites

So... just to say that SQL is a bad choice is a bit ignorant :) What you've read could be very wrong!

 

No its actually a function of time, no matter how you cut it.

Its not hard to argue that HDF5 is vastly better than SQL for storing tick data...its a non issue. There is a reason we have the concept of time series and relational databses, outside of chance.

It is plain and simple.

IF the time put into HDF5 that can make it better than SQL though, is the real variable then do it, if not, then don't.

That doesn't change the fact that SQL is not built for high freq time series and there are DB that are.

you have no real point

Share this post


Link to post
Share on other sites

I have set up a sizeable array that can expand to 40Tb, right now we have it running ( hot swapable) with 8 TB. I eventually paln on setitng it up so others cna utilize it via the net, to access what they need if this is possible.

My only interests are in properly back testing trading scenarios.

 

Has anyone worked with OpenTick?owuld be

right now it is shut down, however there might be some people out there that have constructed decent datbases of tick data.

 

It would be nice to collaborate with some for the design.

Is anyone interested?

 

Could anyone possibly offer up a small amount of tick data to with?

 

My effort revolves around creating a platform based on a rules engine where I will create an easy to use rt/historical interface.

 

I know there are many systems out there that are great,

I am not looking to compete with them or to go and sell an application.

 

All I am tring to do is to create an internal module that will serve my needs whihc iis high frequency data anlysis.

 

MYSQL may not be the ideal environment , however until we can afford KX this is what we will use.

 

as mentioned in other threads there are some alternatives, but learning curve is steep.

 

thx

Share this post


Link to post
Share on other sites

one other thought

 

has anyone considered reviewing tick data relative to the bid/ask and their quantites at the partucular point in time ?

 

I notie there are other threads on tape reading and some new fancy tools tlaked about, such as CQG's products.

 

My own interstis to analyze their relationships in historical and then can easily set up a real time monitor forthese situations, so it is not so much trying to watch the tape, as it does move quite fast

any thoughts?

Share this post


Link to post
Share on other sites

I save tick data from eSignal using QCollector. I have it configured to save separate ascii files for each futures contract I follow, e.g. YGZ8_0.csv , and when those files approach 2Gb I rename to YGZ8_yymmdd.csv (where yymmdd is the last day of full data) and then start collecting a new YGZ8_0.csv file. I configured the file content as:

yymmdd,hhmmss,<trade_price>,<trade_volume>,,,,

yymmdd,hhmmss,,,<bid>,<bid_size>,<ask>,<ask_size>

the contents are CSV as either trades or quotes with best bid/ask

When a contract is a back month I compress it with gzip, so

for example ESH8_0.csv becomes ESH8_0.csv.gz .

Gzip seems to achieve about an 8:1 compression on these ascii files.

My applications recognize the .gz file extension and either decompress the files

when needed or else directly read the .gz files (e.g. java GZIPInputStream).

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Topics

  • Posts

    • Date: 11th July 2025.   Demand For Gold Rises As Trump Announces Tariffs!   Gold prices rose significantly throughout the week as investors took advantage of the 2.50% lower entry level. Investors also return to the safe-haven asset as the US trade policy continues to escalate. As a result, investors are taking a more dovish tone. The ‘risk-off’ appetite is also something which can be seen within the stock market. The NASDAQ on Thursday took a 0.90% dive within only 30 minutes.   Trade Tensions Escalate President Trump has been teasing with new tariffs throughout the week. However, the tariffs were confirmed on Thursday. A 35% tariff on Canadian imports starting August 1st, along with 50% tariffs on copper and goods from Brazil. Some experts are advising that Brazil has been specifically targeted due to its association with the BRICS.   However, the President has not directly associated the tariffs with BRICS yet. According to President Trump, Brazil is targeting US technology companies and carrying out a ‘witch hunt’against former Brazilian President Jair Bolsonaro, a close ally who is currently facing prosecution for allegedly attempting to overturn the 2022 Brazilian election.   Although Brazil is one of the largest and fastest-growing economies in the Americas, it is not the main concern for investors. Investors are more concerned about Tariffs on Canada. The White House said it will impose a 35% tariff on Canadian imports, effective August 1st, raised from the earlier 25% rate. This covers most goods, with exceptions under USMCA and exemptions for Canadian companies producing within the US.   It is also vital for investors to note that Canada is among the US;’s top 3 trading partners. The increase was justified by Trump citing issues like the trade deficit, Canada’s handling of fentanyl trafficking, and perceived unfair trade practices.   The President is also threatening new measures against the EU. These moves caused US and European stock futures to fall nearly 1%, while the Dollar rose and commodity prices saw small gains. However, the main benefactor was Silver and Gold, which are the two best-performing metals of the day.   How Will The Fed Impact Gold? The FOMC indicated that the number of members warming up to the idea of interest rate cuts is increasing. If the Fed takes a dovish tone, the price of Gold may further rise. In the meantime, the President pushing for a 3% rate cut sparked talk of a more dovish Fed nominee next year and raised worries about future inflation.   Meanwhile, jobless claims dropped for the fourth straight week, coming in better than expected and supporting the view that the labour market remains strong after last week’s solid payroll report. Markets still expect two rate cuts this year, but rate futures show most investors see no change at the next Fed meeting. Gold is expected to finish the week mostly flat.       Gold 15-Minute Chart     If the price of Gold increases above $3,337.50, buy signals are likely to materialise again. However, the price is currently retracing, meaning traders are likely to wait for regained momentum before entering further buy trades. According to HSBC, they expect an average price of $3,215 in 2025 (up from $3,015) and $3,125 in 2026, with projections showing a volatile range between $3,100 and $3,600   Key Takeaway Points: Gold Rises on Safe-Haven Demand. Gold gained as investors reacted to rising trade tensions and market volatility. Canada Tariffs Spark Concern. A 35% tariff on Canadian imports drew attention due to Canada’s key trade role. Fed Dovish Shift Supports Gold. Growing expectations of rate cuts and Trump’s push for a 3% cut boosted the gold outlook. Gold Eyes Breakout Above $3,337.5. Price is consolidating; a move above $3,337.50 could trigger new buy signals. Always trade with strict risk management. Your capital is the single most important aspect of your trading business.   Please note that times displayed based on local time zone and are from time of writing this report.   Click HERE to access the full HFM Economic calendar.   Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding of how markets work. Click HERE to register for FREE!   Click HERE to READ more Market news.   Michalis Efthymiou HFMarkets   Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past performance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in Leveraged Products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.
    • Back in the early 2000s, Netflix mailed DVDs to subscribers.   It wasn’t sexy—but it was smart. No late fees. No driving to Blockbuster.   People subscribed because they were lazy. Investors bought the stock because they realized everyone else is lazy too.   Those who saw the future in that red envelope? They could’ve caught a 10,000%+ move.   Another story…   Back in the mid-2000s, Amazon launched Prime.   It wasn’t flashy—but it was fast.   Free two-day shipping. No minimums. No hassle.   People subscribed because they were impatient. Investors bought the stock because they realized everyone hates waiting.   Those who saw the future in that speedy little yellow button? They could’ve caught another 10,000%+ move.   Finally…   Back in 2011, Bitcoin was trading under $10.   It wasn’t regulated—but it worked.   No bank. No middleman. Just wallet to wallet.   People used it to send money. Investors bought it because they saw the potential.   Those who saw something glimmering in that strange orange coin? They could’ve caught a 100,000%+ move.   The people who made those calls weren’t fortune tellers. They just noticed something simple before others did.   A better way. A quiet shift. A small edge. An asymmetric bet.   The red envelope fixed late fees. The yellow button fixed waiting. The orange coin gave billions a choice.   Of course, these types of gains are rare. And they happen only once in a blue moon. That’s exactly why it’s important to notice when the conditions start to look familiar.   Not after the move. Not once it's on CNBC. But in the quiet build-up— before the surface breaks.   Enter the Blue Button Please read more here: https://altucherconfidential.com/posts/netflix-amazon-bitcoin-blue  Profits from free accurate cryptos signals: https://www.predictmag.com/ 
    • What These Attacks Look Like There are several ways you could get hacked. And the threats compound by the day.   Here’s a quick rundown:   Phishing: Fake emails from your “bank.” Click the link, give your password—game over.   Ransomware: Malware that locks your files and demands crypto. Pay up, or it’s gone.   DDoS: Overwhelm a website with traffic until it crashes. Like 10,000 bots blocking the door. Often used by nations.   Man-in-the-Middle: Hackers intercept your messages on public WiFi and read or change them.   Social Engineering: Hackers pose as IT or drop infected USB drives labeled “Payroll.”   You don’t need to be “important” to be a target.   You just need to be online.   What You Can Do (Without Buying a Bunker) You don’t have to be tech-savvy.   You just need to stop being low-hanging fruit.   Here’s how:   Use a YubiKey (physical passkey device) or Authenticator app – Ditch text message 2FA. SIM swaps are real. Hackers often have people on the inside at telecom companies.   Use a password manager (with Yubikey) – One unique password per account. Stop using your dog’s name.   Update your devices – Those annoying updates patch real security holes. Use them.   Back up your files – If ransomware hits, you don’t want your important documents held hostage.   Avoid public WiFi for sensitive stuff – Or use a VPN.   Think before you click – Emails that feel “urgent” are often fake. Go to the websites manually for confirmation.   Consider Starlink in case the internet goes down – I think it’s time for me to make the leap. Don’t Panic. Prepare. (Then Invest.)   I spent an hour in that basement bar reading about cyberattacks—and watching real-world systems fall apart like dominos.   The internet going down used to be an inconvenience. Now, it’s a warning.   Cyberwar isn’t coming. It’s here.   And the next time your internet goes out, it might not just be your router.   Don’t panic. Prepare.   And maybe keep a backup plan in your back pocket. Like a local basement bar with good bourbon—and working WiFi.   As usual, we’re on the lookout for more opportunities in cybersecurity. Stay tuned.   Author: Chris Campbell (AltucherConfidential) Profits from free accurate cryptos signals: https://www.predictmag.com/   
    • DUMBSHELL:  re the automation of corruption ---  200,000 "Science Papers" in academic journal database PubMed may have been AI-generated with errors, hallucinations and false sourcing 
    • Does any crypto exchanges get banned in your country? How's about other as Bybit, Kraken, MEXC, OKX?
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.