Best Strategy Backtesting Software

[Guest Tresor]

I used to be a heavy optimiser and backtester. After having literally killed a number of CPUs I rented a data processing centre for a week to run optimisations of strategies. It wasn't easy to find a centre with Windows servers.

 

I gave up backtesting after some geeks had shown me Matlab's and its toolboxes' capabilities. So, best backtesting software for me is Matlab.

 

The problem is I couldn't operate it. This application was not written with traders in mind.

 

The catch is operating Matlab requires a 100% mathematical / programming brain. This application is so complex (at least for me) that a non-programmer guy who is used to amateur platforms like TS / MC / Ninja / whatever will experience massive problems to switch.

 

The other catch is with all toolboxes I needed the retail price was around $30k at the time I was considering the purchase.

 

Another catch was that at that time only supported data feed in Matlab was Reuters or similarily pricey data feed - in region of $ 1 - 2k per month.

 

Another catch is you need to hire a geek to do some jobs for you from time to time (all rather all the time). At the start the geek will have little salary requirements, but with time his appettite for money will grow.

 

The other catch was...and the other catch was... yet another catch was...

 

So many catches.

 

But still I consider Matlab the best backtesting software, but ... unusable for an average trader.

 

I have recently learned there is a Matlab add-on that utilises graphics cards to speed up computational processes by 25 - 100 times, which has made the applicatiom more powerful.

[BlowFish 10]

@tams it dosen't really matter what you put in, Tradestation is likely to mangle it into garbage. You can get round this to some degree but it is messy too. It just does not have the architecture and internal precision for accurate testing in lower time frames. Using 1 tick bars for testing can help but it is far from ideal.

 

@tresor Matlab is a great product but aimed more at modelling & simulationrather than back testing. Takes you towards a similar place but from a different direction. The big issue with it is that it does not handle large data sets very well (even with all the wizbang addons like the parallel computing toolbox) the main reason being that it needs to load the complete data set it is working on into memory, that can be quite a lot. Some while ago I had the fortune to attend a do that MathWorks hosted for the great & good in the financial industry. A straw pole of the attendees clearly showed that by far and away the predominant use of Matlab was modelling portfolio risk. I did meet a couple of guys that where trying to do interesting stuff at 'higher frequencies' and they where having issues with large vectors. It is possible but it requires a fair bit of 'fudging'.

[LS_Chad 10]

I am going to begin producing a new video series today on taking an idea, turning the idea into a trading system/backtest, testing/refining/optimizing, simulated auto-trading and eventually live auto-trading. I'll share these videos here as I produce them. Hope to have the first one ready today. The idea I plan to work with is fading VWAP bands on the ES.

[LS_Chad 10]

I have completed my first video in the series "Fading the VWAP Bands: From Idea to Automation". In this first video, we formulate our trading idea (Fading the VWAP Bands), develop it into a trading system, add the system to a chart and perform our first backtest. I also discuss why I'm doing this, why this system was chosen, and what I intend to accomplish with this video series.

 

Investor/RT - Fading the VWAP Bands

 

I will also begin a thread on this in the LinnSoft Forum.

[natedredd10 10]

A straw pole of the attendees clearly showed that by far and away the predominant use of Matlab was modelling portfolio risk. I did meet a couple of guys that where trying to do interesting stuff at 'higher frequencies' and they where having issues with large vectors. It is possible but it requires a fair bit of 'fudging'.

 

This may have more to do with the topic of the lecture and who it would attract rather than matlab not being used in trading. If you look on nuclearphy or willmot its pretty ubiquitous.

I doubt anyone here would have to deal with vector sizes that true high frequency guys would be into.

this blog gives some idea what it can do for trading

Trading With MATLAB

If you look at how small the code is to produce a market profile its rather impressive.

My experience with it was through getting my cousin to get me a student license. The syntax of the language just doesn't "jive" with my brain for whatever reason. Just like VBA is sopposed to be much easier than C# but VBA makes no sense to me at all while C# i thought was the easiest language to learn syntax wise.

For backtesting mathworks has this webinar

Algorithmic Trading with MATLAB for Financial Applications - Recorded Webinar

Matlab's strengths vs retail trading software is in data visualization(just look at the 3d sharpe ration plot of the backtest of a simple MA cross system in that webinar), hooking into basically any database you could want already is there as matlab functions and any kind of statistical function you could want can be had for a price if not free.

There is a Dickey–Fuller test for ninja around and just compare the code to matlab or R. A jumbled mess that who even knows if the algorithm is correct in ninja, you have to put your faith in the 1 guy that wrote it. Matlab/R is not going to have wrong statistical algorithms in it exactly because its not specifically for trading. Not to mention the speed you get from piggy backing the work of actual computer scientist, statisticians and mathematicians across fields of interest using the same tool vs some single generous retail trader giving away their algorithm. Not a hard bet that they are not writing the algorithm at its optimal speed, even assuming its output is correct(a HUGE if).

I soppose this all depends on context of the original question. I have a very low opinion of the ability to take OHLC data, data snoop a bunch of windowed "indicators" and do anything beyond simple variance walk forward wise with any meaningful sample of trades. No offense to Investors R/T, which I think is a lovely charting package but algo wise...these are irrational expectations.

[SIUYA 11]

I have not used Matlab, but have heard great things about it.

But the crux of a search for the best backtesting system around that is readily AVAILABLE and AFFORDABLE to a retail person surely needs to focus on how close to reality you can get in the actual backtesting, how much the programmer is aware of the fudging that is required, and then most importantly how good at data handling the system and the programmer is.

Backtesting with fudged data from the start makes things difficult.

The other thing that needs to be taken into consideration of course is time frame and frequency.

A retail person competing on a tick by tick basis may find it hard.On a daily basis assuming you are not trying to get set exactly on opens and closes may be easier.

Most backtesting systems dont take into consideration a portfolio approach - probably not relevant for a retail day trader, so we can ignore this - which makes things easier.

And then ultimately the best backtesting system in the world cannot work unless it can be quickly translated into the real world markets via a system. This is where speed may be important. (I mean does it matter if a test takes 5 mins or 30 secs to run - its a test and you want accuracy not speed)

Data management on ongoing system management can also take up a lot of time and costs depending on the system once it running.

Quite frankly I think a lot of time, effort and energy are spent on backtesting systems without the realisation that they are ultimately a fudge. (and as Nate mentioned elsewhere many people are too lazy to really work on the issues involved and want just a system to do the work)

The time might be better be spent on a system that you can automate and trade in the real world and is easy to tweak (assuming the theory, edge and some fudged backtesting has been done to show it may work)

[Tams 20]

@tams it dosen't really matter what you put in, Tradestation is likely to mangle it into garbage. ....

 

EasyLanguage is a simple procedural language...

 

you do A, then do B...

if C is true, then do D,

else do E...

 

Simplicity has it beauties...

you create the procedure... the program gets you from point A to point B, That's it.

 

Simplicity has its challenges...

most people are conceptual minded, not military procedure oriented.

to convert a concept into a procedure takes work; lots of work.

to convert a complex concept into a step-by-step procedure takes precision work.

to convert a parallel concept into a serial procedure is daunting.

to convert a fuzzy logic concept into a precise procedure is a PhD thesis.

 

I have helped many people with their codes,

the problem is not in TradeStation, or Ninja, or whatever program they are using.

the problem is not in EasyLanguage, or C#, or whatever language they are using...

 

the problem is in their head.

 

ppl don't think through what they want to do... and start coding.

ppl don't draw out a flow chart...

ppl have an IF this then DO that... but left out the entire ELSE !

ppl don't test out their concept with a paintbar, or a plot,

and wonder why the buy/sell triggered at the wrong places.

 

and of course there was this one big shot who hasn't read the manual,

and blame everything on the program.

 

Life is a comedy.

Edited June 19, 2010 by Tams

[AmCan1 10]

I hope that this isn't too far off base as far as the direction this thread is heading. I see there are some programmer math guys chiming in. I'm in the camp that pretty much agrees with WorldTrader. Here's why.

 

Let's say you get a very accurate, automated backtest with a bunch of reliable results. I believe it is a rare bird indeed, that would be able to trade the rules based on those results because the reality will be that the market will throw winning trades and losing trades at you in a random way and the losing trades are sure to cluster up and affect most traders confidence at some point. In other words, I don't think you can 'take ownership' of a trade system, merely by logically talking yourself into it. Automated backtest results give you a good logical reason to trade a system, but pschologically, there is no internal ownership. Most people will fall prey to just being human and will lose their objectivity that was supposed to have been provided by the automated backtest results.

 

So I have to agree with WorldTrader that the tedious, boring, hard work of manually backtesting a system you hope to trade, coupled with some forward testing and practice in real time conditions, is the only way to finally take internal ownership of a system that you can see logically works, but also can trade with confidence, trade after trade, according to the plan. That's been my real life experience too, having spent a lot of time manually backtesting, gathering useful and accurate trade data, analyzing it properly, and then being able to trade the plan without worrying about the random distribution of wins and losses. The edge that I had been able to establish with my manual backtest continues to manifest and the equity curve keeps growing over time as a result. It was worth the effort. A real, hard ass pro could possibly achieve this with automated backtest results only, but, that's a rare bird. Most retail traders could not, imo.

[estate1997 10]

You might want to take a free spin for thirty days on Trading Solutions.

Trading Software, Neural Network, AI, Technical Analysis, Market Timing, Stocks, Futures, FOREX

estate1997

[systo 10]

Hello everyone,

 

quite an interesting thread, specially for a newbie like me. I'm just trying to figure out if these auto-trading strategies can possibly work out for a individual. My first and current futures trading platform is the NanoTrader (aka IQ-Trader) polling tick data from Patsystems.

 

Although I have not much knowledge on other platforms, I'd say IMHO the software is helpful for working on new ideas, but its GUI is sometimes unreasonably slow and is not safe from crashes. Obvious drawback is it uses just one CPU. Testing (specially in walk forward) feels slow, but I lack experience with other platforms to say that for sure.

 

Even though apparently it's not the best, I'd very much like to learn how this platform compares to others to understand if at some point I may dare to auto-trade live with it or should I consider to change in any case?

 

So, if anyone did came across this platform I'd appreciate your evaluation of it, ideally, against any other known platform(s). Thanks!

[tjnoon 10]

Systo, I can't say I know much about the system you are referring to but in general, I remain very skeptical of autotrade systems in general. Call me old school, but I believe a trader needs to be a trader and that the small percentage of art that goes into trading will always haunt autotrade systems.

[cuttshot 11]

Systo, save your time. You won't find an autotraded system that will work over a long period of time. The flip a switch and get rich type of a system is just not out there. You are far better off following tj's advice and taking the time to learn how to trade on your own.

 

As far as backtesting goes, I am a big believe in manual backtesting. I could go on and on about why I believe this. The bottom line is you need to learn your system inside and out as well as your given markets. Spending the time to test hundreds of trades manually will teach you how your system will respong in different market conditions. Most importantly it will help build your confidence in your system and market.

 

I can guarantee if you run an automated test and then go right to live trading you will not be able to make confident decisions. You will run into certain tricky situations as time goes on that you will not know what to do with.

 

Bottom line there are no short cuts in this business.

[AmCan1 10]

Cuttshot has hit the nail on the head. I completely agree. Manual backtesting, while tedious, and a pain in the butt, literally (hours in the chair, lol..) is FAR superior then automation, IMO. I'm sure there are many on TL who would strongly disagree and their entitled to their opinion, and quite frankly, that's what makes a market.

 

As soon as the going gets tough, you'll be gone because you won't have the vision or confidence or even basic understanding to know that the best trades come after hard losing trades. At least with a manual backtest, you can live through that process and begin to absorb it into your psyche. Little by little you build the most important thing of all, confidence, by seeing how the system recovers after losses and how the equity grows, despite the losses. Without that, your instinct for survival will override your trade decisions and you'll be running for your life, the first time you get a 4 or 5 trade losing streak. I'm sure good one's exist, but anybody with a good automated system is NOT going to be making it available. Who would want to kill their golden egg laying goose? Makes no sense.

[LS_Chad 10]

Video 2 ,"Refinement and Optimization", completed in series "Fading the VWAP Bands: Idea Thru Automation": Investor/RT - Fading the VWAP Bands

[UrmaBlume 10]

We hope that this new addition to Trade Station will set a new standard in the kind of back/forward and optimization testing that is available to the public

 

TradeStation Acquires the “Grail” Walk-Forward and Strategy Builder Technology

 

Acquisition Brings TradeStation’s Industry-Leading Strategy Trading Platform to a New Level, Further Raising the Bar against Competitors and Weaker Imitations

 

Plantation FL, June 17, 2010 – TradeStation today announced its acquisition of the Grail System “walk-forward” and “strategy builder” software technology and business from its inventor, Wouter Oosthuizen, and his company, Technovest (Pty) Limited. The products acquired include Grail Walk-Forward Optimizer, Grail Genetic Optimizer and Grail Computer Assisted Strategy Builder, together with all other products, enhancements and new features currently in development. These products will be seamlessly integrated with TradeStation’s award-winning, industry-leading trading strategy design, back-testing and automation platform. Both walk-forward testing and a broad strategy builder offering have been a company priority for TradeStation for some time, as the company has long realized that the potential value of its strategy design, back-testing and automation platform could rise dramatically with the addition of these tools to create and optimize back-tested trading strategies.

 

“The Grail Walk-Forward Optimizer should take TradeStation’s strategy back-testing functionality to a new threshold,” said John Bartleman, TradeStation Securities’ Vice President of Product Management. “Once fully integrated and rolled-out as part of our TradeStation Version 9.0, a major-enhancement release that should be out before year-end, traders will have the ability to test the robustness of their strategies at a whole new level. The Grail’s unique model of ‘Walk Forward’ testing allows the trader to see how the strategy he has tested against a historical time period will perform against various unseen market data metrics and time segments within that back-tested time period. The Grail Walk Forward Optimizer performs this multi-dimensional analysis of the strategy and then concludes with a simple ‘pass’ or ‘fail’ based on the simulated performance results. In other words, if the strategy is going to have a better chance of success in real trading, it must first prove its robustness by passing these rigorous metrics analyses performed by the Grail.

 

“The Grail’s Strategy Builder,” Bartleman added, “will significantly help traders build new strategies from scratch without the need for custom EasyLanguage® programming. Using genetic algorithms, the trader will have the ability to scan through millions of strategy permutations in a relatively short period, identify those strategies he believes show potential, and then submit them to the Grail Walk-Forward Optimizer to test their robustness.”

 

“The Grail products all were developed principally with TradeStation users in mind,” said Mr. Oosthuizen. “TradeStation will be releasing some extremely powerful functionality in its upcoming Version 9.0 release which, in my opinion, not only will dramatically increase the Grail’s speed and value, but will take TradeStation to a rarefied position where the barrier of entry for a competitor seems insurmountable. Teaming up with TradeStation, and having the value of its impressive product development and user support resources and expertise, should take the Grail technology to a whole new level as well.”

 

Mr. Oosthuizen will be joining the company full-time at its Florida headquarters on October 1, under a three-year employment agreement, to help lead the efforts to enhance the technology and seamlessly integrate it with the TradeStation platform. The parties have already started working together to achieve these goals. The purchase was made by TradeStation Technologies, Inc., the trading software development subsidiary of TradeStation Group, Inc. (NASDAQ GS: TRAD) which develops and owns TradeStation’s proprietary software, technology and intellectual property.

 

 

[systo 10]

thanks, for valuable advise tjnoon, cuttshot and AmCan1 !

 

you are right, I may be to optimistic about autotrading. Past few month I spent playing with CFDs and recently with scalping futures, but with mixed results at best. It looks like profits are possible only when terrible abount of time is spent on daily basis. Auto- or half-auto trading would be hopefully less strained. Also, the possibility to backtest is a huge advantage. Manual backtesting would be certaily very useful, but I am not sure if proper adjustments would be easy anough to spot.

 

As for auto-trading, it's obvious that I can't count on developing a system, that will run profitable in long term. I hope, running a set few systems per product one can add to stability and afford scraping system early, as soon as they lose more than in the past (maybe every month or so). If this can work out, I will be more comfortable spending whole time on developing new systems, than trading live on my own.

 

can this possibly work out or am I just going to bump my head against the wall?

[AmCan1 10]

Systo, I think you're over complicating things. You could learn how to trade much easier than trying to do what you suggested. Trading involves about 10% art and you could never program that IMO. It's easier and probably less time consuming to learn a good trade system, backtest it manually, create a simple to follow tradeplan based on the results of your manual backtest, prove you can trade it with a bunch of consecutive mistake free trades in a live sim account, then go live with the minimal amount of risk possible until you've really got it dialed in. That's what I did and it turned my trading around. The process is very cut and dry, step by step but the effect will change everything for you. Why do all that other stuff you mentioned? Doesn't seem like the best approach to me.

[TraderWill 11]

Any system that can be traded and tested manually can be programmed and tested through automation. The problem is that many variables and criteria that we apply when trading manually, never make it into the automated system. Small price adjustments around key price levels, decisions about when to take the first trade, when to stop trading, when to adjust our target and stop expectations, etc. are all things that we do almost insitnctively but never make it into a program specification. Or, if we do try to factor all these variables and criteria into the program, the code can become incredibly complex and subject to coding bugs. Furthermore, the programming language itself may limit what can be accomplished through code.

 

Having said that, automation can be invaluable in refining your system. A basic version of the system, without all the nuances mentioned above, can give you a clear idea whether or not the system even has a chance at becoming profitable. It can also help you identify ranges for each of your variables that provide fairly stable results, and can pinpoint values that are over optimized. If you want to see whether your system works better on a 5 minute or a 2 minute chart, you can get the answer a lot faster with automation than through manual testing.

 

Actually letting the automation place trades for you is a much riskier proposition. Besides the risk of programming bugs that will only become evident when they can hurt the most, you're also subject to the risk of power or internet outages. So I feel comfortable using automation to help refine my strategy, but always place and monitor my trades manually.

[cuttshot 11]

Well said traderwill. Automation can be a nice tool for the situations described. However, you want to make sure you are completely comfortable with the system before you do this. Many traders want to buy a system out of the box and just flip the automation on to do the testing. The problem with this is you never really get to know the ins and outs of the system.

 

You also want to make sure when you go to trade live you can execute the trades in a similar way that automation was showing you. The results can look great on a test but if you can't follow the plan it won't mean a thing. Trust me I have been burned with this over and over again. I also agree that it's very important to manually place the trades.

[AmCan1 10]

Traderwill, you make excellent points. You have succinctly stated the benefits of automation, which mainly are about developing and testing trade ideas vs. manual trading. Even if you happend to program in all the nuances and were successful in debugging, a trader still might need to intervene at times. For example, and I'm sure you'd agree, but if you try to write in rules to adjust around key levels and you consider newly formed swings to be key levels, well, those swings might crop up just outside the parameters of your written rules or code. It's just one example. Trying to navigate through choppy action is another. But the benefits that you describes so well are so true.

 

Manual backtesting on the other hand will give you a chance to relive those trades and to be able to scan down a win loss column and begin the get a deeper understanding of the relationship between the wins and losses vs. the equity curve itself. Hard to do that with any sort of way to take ownership of it, with automated results.

[TraderWill 11]

As Cutshot and AmCan say, manual back testing has the great benefit that it reinforces your understanding of the system rules and will point out areas where you may be unclear on the rules. Maybe even help you spot scenarios that are not even covered by the rules. If you ever buy an automated strategy, make sure that it's not a black box, that the developer actually describes the rules behind the setups and exits. Otherwise the only way you can prove to yourself that it works is by running it live and testing it with a demo account or heavens forbid, real money.

[AmCan1 10]

Yes! TraderWill offers up a very important word of advice. The developer or purveyor of the automated system should provide all the rules as to how and why a system sets up. Black box, while seductive because of the dream it promises to deliver, is just that. A dream! Play with it with a sim account if you want but if you think it will produce REAL profits, as yourself first this question: If you developed a black box trade system that actually worked, consistently and reliably, would you want to share it with the world?

[natedredd10 10]

Any system that can be traded and tested manually can be programmed and tested through automation. .

 

On the other hand its not hard to imagine systems that could not be traded/analyzed manually. Retail algo trading has never really progressed beyond the line of thinking you would use with a 166mhz pentium, 1 gig of hd space and and a data feed hooked up to a single instrument and this is reflected in our retail software. All its really doing with modern cpu cycles is the same type of 1985 analysis faster.

Things surely are not helped that practically all the new automated trading ideas of the last 20 years fall under "quant" if your looking for books and there is a level of sophistication that quickly becomes uneconomic to produce retail software for.

[LS_Chad 10]

Video 3 ,"Refinement and Optimization: Part II", completed in series "Fading the VWAP Bands: Idea Thru Automation"

 

In this third video, we continue to refine and optimize the system. We were able to substantially increase the profit per contract with some changes suggested by users. The band magnitude was optimized and code was added to prevent trading during the first hour and to prevent trading during trend days.

 

Investor/RT - Fading the VWAP Bands

[BlowFish 10]

On the other hand its not hard to imagine systems that could not be traded/analyzed manually. Retail algo trading has never really progressed beyond the line of thinking you would use with a 166mhz pentium, 1 gig of hd space and and a data feed hooked up to a single instrument and this is reflected in our retail software. All its really doing with modern cpu cycles is the same type of 1985 analysis faster.

Things surely are not helped that practically all the new automated trading ideas of the last 20 years fall under "quant" if your looking for books and there is a level of sophistication that quickly becomes uneconomic to produce retail software for.

 

It's always fallen under 'quant'. Just a fancy new word for what people have been doing for aeons, modelling and managing risk across their portfolios. Maybe there is a bit more emphasis on modelling nowadays.

 

As to algo trading the retail trader has never had it so good. You can rent a virtual machine in a managed environment for peanuts and you have things like strategy runner available at brokers. Even if you (are mad enough to) want to run stuff on a PC in your home/office infrastructure has never been so cheap.

 

As to software whilst most has 'features' that frustrates there is remarkable stuff available to retail guys from neural nets, genetic algorithms, to the same algorithmic toolboxes available to the great and the good in the commercial world. There is stuff for under a couple of thousand bucks that will take data from hundred of instruments and run optimisations across a grid of networked computers for example.

 

This thread is all about back testing not deploying run time systems btw. Very different propositions with very different requirements but both quite doable by those with appropriate knowledge. Take a look at Chads videos for something that is pretty straightforward that looks like it is already developing into something quite robust.

 

In short I guess I disagree