Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

Szymon

New Open Source Project - MS Sql Server Market Data Colletor.

Recommended Posts

Hi All,

 

I'm starting this thread, it is going to be based on open source licence.

 

The aim of this application project is to produce software that will record hourly price of instruments into a MS Sql server. This server can be sitting locally or somewhere else via the web, you can then use .NET xml technologies to retrieve information.

 

The first part of the project is to find out a cheap or free hourly market data provider that has an API feed.

 

What are the advantages of this kind of application? Well for all you system builders out there, You can then develop other applications that can utilize the information and build systems with your formulas and strategies that your third party applications cannot provide.

 

After we will find the right market data source. We will then get into developing the database structure. I will post SQL files here so you can create your own databases. At this stage I am only interested in 1hour timeframe so that will be the focus for this project but you can change that to suit any other time frame.

 

The last part of the project is to develop using vb.net a small application that will utilize the market data API feed and record new bar information into the database. The information that I want to record is OPEN, CLOSE, HIGH and LOW for multiple instruments, with an update feature in case the comptuer has been switched off for a number of hours.

 

I am willing to provide a MS 2008 database that will have web access available but only to those people that will assist me in this project. I will build a .NET web query xml page where you can utilize this information for yourself and your other projects.

 

By having multiple clients talking to the database server, no matter what machine is switched off the database is constantly being updated from all those other machines that are switched on. By helping you have a choice to utilise this information for your own projects.

 

I hope this makes sense.

 

If you have any questions please do not hesitate to ask me.

 

Regards

 

Simon

Share this post


Link to post
Share on other sites

Hi All,

 

Had some time today to look into this project.

 

After finding out the opentick no longer provides their api services to anyone.

 

I have decided to base this project using openecry API feed as I already have an account with them.

 

The first step is to find out what their API can do by checking out their documentation and sample software.

 

After that I will start construction of the database on a Microsoft 2008 SQL Server.

 

Good trading to you all.

 

Regards

 

Simon

Share this post


Link to post
Share on other sites
how many symbols you will be tracking?

 

I think around 10 for now.

 

But I will put that functionality into the software where you will be able to select how many active instruments to track at any time.

 

Simon

Share this post


Link to post
Share on other sites

The only thing I would say is try not to limit yourself at the design stage. With data storage cheap and processing power fast, don't make compromises you may later regret. In short even though you are only currently interested in collecting hourly bars for a few instruments, in your design conciderations bear in mind that one day you might require tick data from multiple instruments from multiple exchanges. You are likely to get more support from other contributors too!

Share this post


Link to post
Share on other sites

I don't know these languages but well-schooled in Excel -- I was wondering what it would take to set-up something on the internet to have some kind of site where anyone who wants it can pull data on futures contracts --- similar to what Yahoo does for stocks/etfs.

 

Idea is to simply have an efficient site to download data on any contract:

ie,

 

All 30-min data for

 

ES Z7

or

NQ H3

 

just load it by contract once and its done....

 

thanks for any help/ideas

Share this post


Link to post
Share on other sites
I don't know these languages but well-schooled in Excel -- I was wondering what it would take to set-up something on the internet to have some kind of site where anyone who wants it can pull data on futures contracts --- similar to what Yahoo does for stocks/etfs.

 

Idea is to simply have an efficient site to download data on any contract:

ie,

 

All 30-min data for

 

ES Z7

or

NQ H3

 

just load it by contract once and its done....

 

thanks for any help/ideas

 

Hi Frank,

 

Exactly what I am trying to do here. I do not mind sharing a website server and a database to hold that information. And giving access to all users who will contribute to this project.

 

Regards

 

Simon

Edited by Szymon

Share this post


Link to post
Share on other sites
The only thing I would say is try not to limit yourself at the design stage. With data storage cheap and processing power fast, don't make compromises you may later regret. In short even though you are only currently interested in collecting hourly bars for a few instruments, in your design conciderations bear in mind that one day you might require tick data from multiple instruments from multiple exchanges. You are likely to get more support from other contributors too!

 

Hi Blowfish,

 

I will take your comments into consideration. The first stage is to build a test application to see if the concept works then I will consider expanding it.

 

Regards

 

Simon

Share this post


Link to post
Share on other sites

Project Update,

 

OK, everybody who is onboard here is an update.

 

APIFeed for now will be OpenECry.

I have just downloaded the latest version of MS Visual Studio 2009.

 

Will create a simple application that will login to the OpenECry server and see, how does the data look like and what other fields are there to collect.

 

I will post this simple app in exe mode for others to use, you will have to have your own API login to test it out.

 

Now all I have to do is find some time, possibly this weekend.

 

Regards

 

Simon

Share this post


Link to post
Share on other sites

HI All,

 

I have finally got some time here, right now I am installing the latest version of Visual Studio 2008, after that I will start programming a small test application.

 

I also found an excellent charting component that I will incorporate once the first stage is done to utilize the data.

 

Dundas Data Visualization Gallery

 

Simon

Share this post


Link to post
Share on other sites
great idea, I would like to get aboard however I can.

I'm on vacation now tho, be back august 1st.

 

Hi Stanlyd,

 

Welcome aboard.

 

Initially I will need testers to test the application, data, etc...

 

Simon

Share this post


Link to post
Share on other sites
Hi Blowfish,

 

I will take your comments into consideration. The first stage is to build a test application to see if the concept works then I will consider expanding it.

 

Regards

 

Simon

 

I am a great believer in prototyping :) Picked that up from my first boss when I needed to write firmware and software to utilise hardware my fellow researchers had designed. He would insist on writing a test suite first and then a prototype. The code would morph into the app anyway. This is one of the reasons I wasn't particularly bowled over by OOP, I was kind of used to writing re usable code.

 

I do wonder why you picked 1 hour granularity to collect and store? Forgetting tick data for now nearly all applications (and data providers for that matter) provide and store data in chunks of 1 minute. Even if you only want hourly bars it makes sense to collect and store minute data (later perhaps ad tick data) and to aggregate them into hours. The only expense is a little more storage.

 

I am not familiar with OEC's data feed api but it is likely not to be a question of expanding later but of adding code now to filter and aggregate minute data into hour bars and then removing it later. Hope you don't mind me bleating on about this but I think this is a fundememental design concideration that will actually add complexity to diminish utility.

 

The visualisation stuff looks nice! A shame that open source stuff is relatively rudimentary though JFreecharts isn't too bad. I know a couple of people who have done great things with it. IB wasn't one of them!

 

Have fun this weekend with the new software!

Share this post


Link to post
Share on other sites
Hi All,I'm starting this thread, it is going to be based on open source licence.The aim of this application project is to produce software that will record hourly price of instruments into a MS Sql server. This server can be sitting locally or somewhere else via the web, you can then use .NET xml technologies to retrieve information.The first part of the project is to find out a cheap or free hourly market data provider that has an API feed.What are the advantages of this kind of application? Well for all you system builders out there, You can then develop other applications that can utilize the information and build systems with your formulas and strategies that your third party applications cannot provide.After we will find the right market data source. We will then get into developing the database structure. I will post SQL files here so you can create your own databases. At this stage I am only interested in 1hour timeframe so that will be the focus for this project but you can change that to suit any other time frame.The last part of the project is to develop using vb.net a small application that will utilize the market data API feed and record new bar information into the database. The information that I want to record is OPEN, CLOSE, HIGH and LOW for multiple instruments, with an update feature in case the comptuer has been switched off for a number of hours.I am willing to provide a MS 2008 database that will have web access available but only to those people that will assist me in this project. I will build a .NET web query xml page where you can utilize this information for yourself and your other projects.By having multiple clients talking to the database server, no matter what machine is switched off the database is constantly being updated from all those other machines that are switched on. By helping you have a choice to utilise this information for your own projects.I hope this makes sense.

 

If you have any questions please do not hesitate to ask me.

 

Regards

 

Simon

 

Why?

 

Decades of data in every time frame from 1 minute on up is available for every symbol from several different vendors for very little money.

 

Trade Station at $99 per month, free with just a few trades, makes all of this data available and it can be easily exported from TS into either a csv file or via dll to any database you want.

 

Plus with all the great open source db's available why MS db?

 

For Historical data you can get a free trial for TS and download what ever you want.

 

Is the sole reason for this app to save $99 a month? I don't get it.

Share this post


Link to post
Share on other sites

usually best to ignore 'hater' posts like this but I will chime in with why..

 

you are paying Tradestation for their software, not for the data -- which is a commodity and 'should' be free, just like it is for stocks on Yahoo. would you pay $1200 a year to have access to the time of day? of course not, it is a free add-on data feed with your computer.

 

personally, I used Tradestation for years and moved away from it. I find it more appealing to manipulate data in my own way -- not through Tradestation. I have a solid database of data --- but it would be nice to have a place to reference when you do care to expand your own database of data for new markets or different timeframes. the net net answer to why? because in a world of peer to peer networking through the internet, it SHOULD be accessible for free.

Share this post


Link to post
Share on other sites
usually best to ignore 'hater' posts like this but I will chime in with why..

 

you are paying Tradestation for their software, not for the data -- which is a commodity and 'should' be free, just like it is for stocks on Yahoo. would you pay $1200 a year to have access to the time of day? of course not, it is a free add-on data feed with your computer.

 

personally, I used Tradestation for years and moved away from it. I find it more appealing to manipulate data in my own way -- not through Tradestation. I have a solid database of data --- but it would be nice to have a place to reference when you do care to expand your own database of data for new markets or different timeframes. the net net answer to why? because in a world of peer to peer networking through the internet, it SHOULD be accessible for free.

 

Sorry, not a hater post at all.

 

The db you propose is very limited as to time frames and symbol list. Such a db is only suitable for a very limited trading/research operation. For any "REAL" trading or research venture $1200 per year for unlimited data is very inexpensive and if one's trading or research won't support that, then, indeed, such an ersatz approach would be appropriate.

 

Of note is that commercial data feeds like Trade Station are at the bottom of the line. The big boys use direct, near zero latency feeds that cost over $20,000 per month. No matter what you build you can't get that for free. The only reason they pay 20k per month is because they make money by using it.

 

If free one hour data on 10 symbols fulfills your needs then my all means this is the app for you. I run a small private trading and development operation and could neither trade or develop with the data you describe.

Share this post


Link to post
Share on other sites

yes, historical data fulfills my research needs. Whether you pay nothing, $20k per month or $12 million per year (as my old firm paid in annual investment infrastructure), guess what? its the same data.

 

you missed the point by trying to pigeon me into one-hour data and condescending about your 'private trading operation' --- whatever bro, I am not talking about data delivery with zero-latency for trading -- I am talking about historical data. there is a difference. there is no edge in zero-latency raw-compute power. my trading platform with 'infintesimally higher than zero latency' is just fine.

 

in the future, just try to think about the fact that other people might do things differently than you do. there are many different ways to skin a cat and trust me, you don't know all of them. neither do I but the difference is I seem to be aware of this fact.

Share this post


Link to post
Share on other sites

Sorry Frank,

 

My questions regardiing your approach to data collection and managment are about practicality.

 

I and every professional trader I know would find such an approach to be not practical and that to be able to access unlimited data, either real time or historical, for any symbol or instrument at any time to be well worth what we pay.

 

Obviously your approach to data collection and management is adequate for your needs so it is the right one for you.

 

After almost 30 years of professional/management level involvement with the markets, I do, indeed, understand that there are many ways to profit from trading and just didn't understand why anyone would go to the trouble to build such a seemingly inadequate system just to save a few bucks. The answer of course is that it is not inadequate for your needs and thus is perfect for you.

 

I didn't mean to criticize of belittle your approach, I just didn't understand how anyone could go to that effort to get so little just to save a very few bucks. Now I know and wish you the best of luck.

 

cheers

Edited by UrmaBlume

Share this post


Link to post
Share on other sites

Hi Guys,

 

Thanks for your valuable input.

 

At this stage I am happy to take the hourly information, I use this information to calculate future high probability time reversals plus other things and also I wanted to see if this is all possible, to contribute towards this project and share this information like opensource project.

 

Here is a simple OpenECry connector, you can sign up for an API Login from OpenECry and use that to login to this test system. On the next system l will put some basic functionality on the form, just to see what kind of information they are offering.

 

Just extract the folder and double click on the OpenECry_Connector.exe file about 26kb. A form will pop up and then put your login details in, then you will get a test message if all went well.

 

Regards

 

Simon

OpenECry Connector.zip

form.jpg.60317c6f3a169f4b99b7c3eb1b6c7a2d.jpg

Share this post


Link to post
Share on other sites
Why?

 

Decades of data in every time frame from 1 minute on up is available for every symbol from several different vendors for very little money.

 

Trade Station at $99 per month, free with just a few trades, makes all of this data available and it can be easily exported from TS into either a csv file or via dll to any database you want.

 

Plus with all the great open source db's available why MS db?

 

For Historical data you can get a free trial for TS and download what ever you want.

 

Is the sole reason for this app to save $99 a month? I don't get it.

 

I don't know what TS provide history wise but I would be surprised if it really does offer 'all the data you could want'. For example can you can load every tick of the ES since ist inception for example (complete with proper time stamps:)). What about the DAX FTSE CAC, what about more esoteric instruments? Data management is a pain but at least if you do that yourself you know you really do have everything you could possibly want andthat it has been collected and stored accurately.

 

I cant answer for Syzmone but why would you lock yourself in to an archaic propriatory product like TS? :) I presume the point of the project is so that you can connect your database to numerous data providers (including TS) and allow it to be queried by any client application that you can dream of.

 

Don't get me wrong TS is a great product for quick prototyping of ideas but as the cornerstone of a project to collect and manage financial time series data wel :rofl: in fact TS as the cornerstone for any 'big iron' type project .....well no comment but good luck with that.

 

I do agree with your comment on the choice of database. SQL isn't ideal in the first place but despite that is still adequate. Going open would seem to make more sense to me too.

Share this post


Link to post
Share on other sites

Hi All,

 

Can someone tell me why MS Sql 2008 is not good for this, what have you experienced with this product and do you have any suggestions something else that is better than this product.

 

Here is a link to the MS Website for this product.

 

Microsoft SQL Server 2008: Overview

 

 

Simon

Edited by Szymon

Share this post


Link to post
Share on other sites

About ways to store large data sets please notice this extensive discussion (you may have to register first to get access to the yahoo-group):

 

Yahoo! Group TWS-API: Has anybody tried some sort of database?

 

It is a discussion group of people who are writing programs for IB (Interactive Brokers) API.

Anyway the discussion is relevant for everybody who thinks about saving (and accessing) a big amount of data.

Answers your question about SQL also.

 

 

 

Another thing that might deserve your interest:

 

Esper - Complex Event Processing

 

A free, open-source and working event-processing project.

Share this post


Link to post
Share on other sites

One question to ask ...are SQL databases good the best way to store sequential data? Answer is probablly no. Does it matter that much? The answer is again probabbly not. Chances are the yahoo group that was linked will give you a much better idea of pros and cons.

 

Having chose SQl (which is certainly not wrong) then the only choice is which implementation to go with. To be honest I don't really know the current pros and cons, it's years and years since I have had to seriously evaluate DB products. Intuitivly an open source implementation seems to make sense for an open source project. There are many things that could blow that feeling right out of the water. Your preference is one of them :)

Share this post


Link to post
Share on other sites
Why?

 

Decades of data in every time frame from 1 minute on up is available for every symbol from several different vendors for very little money.

 

Trade Station at $99 per month, free with just a few trades, makes all of this data available and it can be easily exported from TS into either a csv file or via dll to any database you want.

 

Plus with all the great open source db's available why MS db?

 

For Historical data you can get a free trial for TS and download what ever you want.

 

Is the sole reason for this app to save $99 a month? I don't get it.

 

There you go again for the fourth time plugging TradeStation sans their ad. :doh:

 

Was it your intent to hijack this thread to talk up TS or was it an accident?

Edited by ZOSO

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Topics

  • Posts

    • Date: 11th July 2025.   Demand For Gold Rises As Trump Announces Tariffs!   Gold prices rose significantly throughout the week as investors took advantage of the 2.50% lower entry level. Investors also return to the safe-haven asset as the US trade policy continues to escalate. As a result, investors are taking a more dovish tone. The ‘risk-off’ appetite is also something which can be seen within the stock market. The NASDAQ on Thursday took a 0.90% dive within only 30 minutes.   Trade Tensions Escalate President Trump has been teasing with new tariffs throughout the week. However, the tariffs were confirmed on Thursday. A 35% tariff on Canadian imports starting August 1st, along with 50% tariffs on copper and goods from Brazil. Some experts are advising that Brazil has been specifically targeted due to its association with the BRICS.   However, the President has not directly associated the tariffs with BRICS yet. According to President Trump, Brazil is targeting US technology companies and carrying out a ‘witch hunt’against former Brazilian President Jair Bolsonaro, a close ally who is currently facing prosecution for allegedly attempting to overturn the 2022 Brazilian election.   Although Brazil is one of the largest and fastest-growing economies in the Americas, it is not the main concern for investors. Investors are more concerned about Tariffs on Canada. The White House said it will impose a 35% tariff on Canadian imports, effective August 1st, raised from the earlier 25% rate. This covers most goods, with exceptions under USMCA and exemptions for Canadian companies producing within the US.   It is also vital for investors to note that Canada is among the US;’s top 3 trading partners. The increase was justified by Trump citing issues like the trade deficit, Canada’s handling of fentanyl trafficking, and perceived unfair trade practices.   The President is also threatening new measures against the EU. These moves caused US and European stock futures to fall nearly 1%, while the Dollar rose and commodity prices saw small gains. However, the main benefactor was Silver and Gold, which are the two best-performing metals of the day.   How Will The Fed Impact Gold? The FOMC indicated that the number of members warming up to the idea of interest rate cuts is increasing. If the Fed takes a dovish tone, the price of Gold may further rise. In the meantime, the President pushing for a 3% rate cut sparked talk of a more dovish Fed nominee next year and raised worries about future inflation.   Meanwhile, jobless claims dropped for the fourth straight week, coming in better than expected and supporting the view that the labour market remains strong after last week’s solid payroll report. Markets still expect two rate cuts this year, but rate futures show most investors see no change at the next Fed meeting. Gold is expected to finish the week mostly flat.       Gold 15-Minute Chart     If the price of Gold increases above $3,337.50, buy signals are likely to materialise again. However, the price is currently retracing, meaning traders are likely to wait for regained momentum before entering further buy trades. According to HSBC, they expect an average price of $3,215 in 2025 (up from $3,015) and $3,125 in 2026, with projections showing a volatile range between $3,100 and $3,600   Key Takeaway Points: Gold Rises on Safe-Haven Demand. Gold gained as investors reacted to rising trade tensions and market volatility. Canada Tariffs Spark Concern. A 35% tariff on Canadian imports drew attention due to Canada’s key trade role. Fed Dovish Shift Supports Gold. Growing expectations of rate cuts and Trump’s push for a 3% cut boosted the gold outlook. Gold Eyes Breakout Above $3,337.5. Price is consolidating; a move above $3,337.50 could trigger new buy signals. Always trade with strict risk management. Your capital is the single most important aspect of your trading business.   Please note that times displayed based on local time zone and are from time of writing this report.   Click HERE to access the full HFM Economic calendar.   Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding of how markets work. Click HERE to register for FREE!   Click HERE to READ more Market news.   Michalis Efthymiou HFMarkets   Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past performance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in Leveraged Products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.
    • Back in the early 2000s, Netflix mailed DVDs to subscribers.   It wasn’t sexy—but it was smart. No late fees. No driving to Blockbuster.   People subscribed because they were lazy. Investors bought the stock because they realized everyone else is lazy too.   Those who saw the future in that red envelope? They could’ve caught a 10,000%+ move.   Another story…   Back in the mid-2000s, Amazon launched Prime.   It wasn’t flashy—but it was fast.   Free two-day shipping. No minimums. No hassle.   People subscribed because they were impatient. Investors bought the stock because they realized everyone hates waiting.   Those who saw the future in that speedy little yellow button? They could’ve caught another 10,000%+ move.   Finally…   Back in 2011, Bitcoin was trading under $10.   It wasn’t regulated—but it worked.   No bank. No middleman. Just wallet to wallet.   People used it to send money. Investors bought it because they saw the potential.   Those who saw something glimmering in that strange orange coin? They could’ve caught a 100,000%+ move.   The people who made those calls weren’t fortune tellers. They just noticed something simple before others did.   A better way. A quiet shift. A small edge. An asymmetric bet.   The red envelope fixed late fees. The yellow button fixed waiting. The orange coin gave billions a choice.   Of course, these types of gains are rare. And they happen only once in a blue moon. That’s exactly why it’s important to notice when the conditions start to look familiar.   Not after the move. Not once it's on CNBC. But in the quiet build-up— before the surface breaks.   Enter the Blue Button Please read more here: https://altucherconfidential.com/posts/netflix-amazon-bitcoin-blue  Profits from free accurate cryptos signals: https://www.predictmag.com/ 
    • What These Attacks Look Like There are several ways you could get hacked. And the threats compound by the day.   Here’s a quick rundown:   Phishing: Fake emails from your “bank.” Click the link, give your password—game over.   Ransomware: Malware that locks your files and demands crypto. Pay up, or it’s gone.   DDoS: Overwhelm a website with traffic until it crashes. Like 10,000 bots blocking the door. Often used by nations.   Man-in-the-Middle: Hackers intercept your messages on public WiFi and read or change them.   Social Engineering: Hackers pose as IT or drop infected USB drives labeled “Payroll.”   You don’t need to be “important” to be a target.   You just need to be online.   What You Can Do (Without Buying a Bunker) You don’t have to be tech-savvy.   You just need to stop being low-hanging fruit.   Here’s how:   Use a YubiKey (physical passkey device) or Authenticator app – Ditch text message 2FA. SIM swaps are real. Hackers often have people on the inside at telecom companies.   Use a password manager (with Yubikey) – One unique password per account. Stop using your dog’s name.   Update your devices – Those annoying updates patch real security holes. Use them.   Back up your files – If ransomware hits, you don’t want your important documents held hostage.   Avoid public WiFi for sensitive stuff – Or use a VPN.   Think before you click – Emails that feel “urgent” are often fake. Go to the websites manually for confirmation.   Consider Starlink in case the internet goes down – I think it’s time for me to make the leap. Don’t Panic. Prepare. (Then Invest.)   I spent an hour in that basement bar reading about cyberattacks—and watching real-world systems fall apart like dominos.   The internet going down used to be an inconvenience. Now, it’s a warning.   Cyberwar isn’t coming. It’s here.   And the next time your internet goes out, it might not just be your router.   Don’t panic. Prepare.   And maybe keep a backup plan in your back pocket. Like a local basement bar with good bourbon—and working WiFi.   As usual, we’re on the lookout for more opportunities in cybersecurity. Stay tuned.   Author: Chris Campbell (AltucherConfidential) Profits from free accurate cryptos signals: https://www.predictmag.com/   
    • DUMBSHELL:  re the automation of corruption ---  200,000 "Science Papers" in academic journal database PubMed may have been AI-generated with errors, hallucinations and false sourcing 
    • Does any crypto exchanges get banned in your country? How's about other as Bybit, Kraken, MEXC, OKX?
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.