Backtesting As a Multi-Step Process

[RichardCox 10]

Technical analysis traders looking to find and implement new trading strategies will often look to backtesting as a means for assessing the general efficacy of a given method (or series of parameters). Relying solely on these results is a mistake, however, as backtesting results have only a limited ability to determine the profitability of any system when used in live markets. To be sure, backtesting can offer traders some valuable information in terms of the potential success or failure present in any trading strategy.

 

But it should be remembered that these results will often be misleading and should be thought of as just one element of the overall evaluation process. Other elements to consider can be found in forward testing and out-of-sample testing, which can help to achieve a better understanding of which markets work best with certain trading strategies. It is important to analyze these results before real money is put at risk in a trade, as this can help to avoid haphazard methods that lack depth.

 

When these three assessment methods (backtesting, forward testing, and out-of-sample testing) are conducted, good correlation is the best indication of a trading system’s viability. Here (and in part 2 to this article), we will look at some of the key elements of that correlation so that backtesting methods can be refined for live market trades.

 

Basics of Backtesting

 

At its core, backtesting is the practice of applying trading systems through historical price fluctuations in order to monitor its level of success or failure over a given time period. Technical analysis strategies can be tested relatively quickly in this fashion without risking real money. Any trading strategy can be tested, from the most basic ideas to the most complex. One example of a simple tested strategy would be to look at moving average crossovers as a basis for trading positions. More complicated tests would involve a larger number of inputs and triggers.

 

Any trading system that can be quantified is suitable for backtesting. Some computer programming ability can be helpful, especially in cases where less common trading platforms are used, or a wider variety of variables will be tested. For example, better knowledge in these areas will enable traders to change the commonly used inputs (the moving average periods themselves). Here, a trader might then be able to test a large number of moving average periods and determine which variables have the best success rates over time.

 

Strategy Optimization

 

Some of the more commonly used trading platforms will enable traders to use “optimization” features during the process. This feature allows traders to input a range of variable (such as a range of moving averages). The computer then looks at the number of successful trades for each input and will determine which period has the highest rate of profitability. Multi-variable optimization can be used for strategies with more than one input variable.

 

In the moving average example, this would mean that the trader would input a range for two moving averages and then the software would be able to identify which combination works best. For example, crossover strategies using a 100-period average and a 55-period moving average might lead to more successful trades than a 100-period moving average and a 50-period moving average.

 

The main benefit of this feature is there there will be some cases where an unprofitable strategy can undergo a few simple tweaks (or “optimizations”), and be transformed into a profitable strategy. It should be remembered, however, that this “profitability” will be in past terms. This does not necessarily mean that this same success rate will be seen in the future in live markets. Curve-fitting uses optimization analytics to identify scenarios with the largest number of successful trades and the best potential for profits.

 

Problems can arise here, however, as methods are “optimized” for the specific data set being analyzed. Since that same price history will not be seen again, many of these systems will become unreliable in the future. So, while backtests and optimization procedures can offer some interesting benefits when looking to develop and assess a trading strategy, the results should not be viewed in isolation. The next part of the process is to test the same strategy against historical price data that was not used in the original tests.

 

Using Out-of-Sample Price Data

 

Once you have a strategy you want to test, it is important to section off historical periods for the testing process. The first set of data is used to test and later optimize the original strategy. This initial data set becomes the “in-sample” data. The other time periods that have been sectioned-off (and not yet tested) now make up the “out-of-sample” data set. It is important to make these differentiations (separating the time periods) because this is the only way to test your original idea on new historical price activity.

 

Initially, your optimization process “custom-designed” its inputs based on a specific set of price action. In order for the strategy to be considered valid, it must be used successfully in new scenarios that were in no way influenced by the optimization process. While this does not remove all of the potential problems in the backtesting process, it does create a greater likelihood that your trading strategy will work in live markets if it is successful in more than one historical time period.

 

The in-sample data is the key component for testing your original strategy, finding its weak points and then optimizing the process to enhance the number of profitable trades. Since this is such a critical component of the original test, traders will usually devote a larger period of time to the in-sample data set. Once the system has been optimized, the system must then be applied to the smaller out-of-sample price data. An added benefit of this approach is that traders can compare the results between the two data sets. When similar performances are seen, there is a better chance for profitability when using the same system with a real trading account.

 

Conclusion

 

Backtesting can allow traders to find, develop, and optimize a technical analysis trading strategy based on the price activity markets have experienced in the past. While this information does have a good deal of value, it is not enough to simply take these backtesting results at face value and expect the success to continue when applied to live markets. For this reason, backtesting is always conducted using a demo account, as additional refinements will be needed before real funds are put at risk. It should also be understood that true backtesting analysis is a multistep procedure that divides price data into different (but necessary segments) In the next sections of this article, we will look at correlation and forward testing as the next key components of the process.

[sergso 10]

The in-sample data is the key component for testing your original strategy, finding its weak points and then optimizing the process to enhance the number of profitable trades. Since this is such a critical component of the original test, traders will usually devote a larger period of time to the in-sample data set. Once the system has been optimized, the system must then be applied to the smaller out-of-sample price data. An added benefit of this approach is that traders can compare the results between the two data sets. When similar performances are seen, there is a better chance for profitability when using the same system with a real trading account.

 

You may want to read the first part of this article. If one backtests 10 systems and finds one that performs well when applied to both in-sample and out-of-sample data chances are the result was due to data-mining.