Smart Optimization And Avoiding Curve-fitting

[Do Or Die 10]

Hi,

 

Optimization refers to the combinatorial search over a range of system input values on price data defined over a fixed number of bars for the cases that produce the best system net profits or some other selected performance variable. It is unavoidable because markets lack stationarity in any form at all and so adaptability is essential for a trading system. By default it is equivalent to curve fitting because you are searching for ‘best fitting’ input parameters. In practice, you can use optimization results to model a more ‘adaptive’ system.

 

Running an optimization can be the easiest way to look for parameters which the market favors in different phases or time intervals. A simple method is to map the changes in optimum input range with a market characteristic such as volatility or trendiness. This in turns helps to build a layer of adaptability or regime analysis which is essential for any trading system.

 

...optimization is only a problem when you fail to take stationarity into account. If you know you are trading within a stable regime, it makes sense to do your best to capture the rules the market is following over that period. My swing trading methodology might best be described as serial optimization: continually hunting for periods of stationary market behavior and trading optimized models derived from those periods.

 

 

Important Points To Avoid Curve-fitting

 

• Simulations should be close to actual execution: Perhaps the most important aspect of backtesting/optimization is to avoid simulations which cannot be executed in practice similarly as the simulation. For example, in intraday systems, optimizing a strategy which takes profits which are less than 5 times of bid-ask spread will likely be meaningless. Similarly, for EOD systems using the optimized performance on 2008 to trade/test during 2010 can be unprofitable because there was a dramatic shift in trading regimes over these periods.
   
  
• Premise based system design: A large number of input parameters or technical indicators lead way to curve-fitting. Using a premise-focused approach helps you keep the parameters minimum and the strategy simple. While modeling a strategy it is easy to get lost on the combination of entry/exit conditions; however, using this approach will help you to focus on the characterstic of market behavior that you intend to exploit in the first place, and which is essential to the ‘edge’ of the system. It is extremely important to code simple “elegant” strategies in order to avoid added complexity which will result in curve fitted solutions.
   
  
• Sufficient sample data for simulation: This is a common adage in statistics; there is hardly anything to elaborate. If the period for which the system is tested is small enough, you may need to test it across a large number of symbols. The (expected) number of trades on which the optimization is performed should also be large enough. I prefer minimum 200 trades (ideally more than 1000) which may be tested on more than 100K bars of data. Small sample data increases the chances of curve fitting to a great extent.
   
  
• Keep your system symmetric: One of the first ideas new traders have when they start system development is to have a separate criteria for Entry/Exit and Short/Long trades (for example using an indicator parameter 20 for long entries but 15 for shorts). This is because separate criteria increase the number of conditions (and degrees of freedom). If a system from profitability to losses depending only on complicated combination of rules it is less likely to be robust. Even though on daily time frame bull markets tend to have totally different characteristics than bear markets, it is better to have a long(short) only strategy than depending on complicated combinations. Also testing the same strategy over different time frames gives information about its stability.
   
  
• Out-of-sample testing periods: A very common practice in system development is go for rolling backtests or walk forward optimization. Certainly the out-sample test doesn’t have to be profitable (as all strategies have profit and draw down periods) but it must at least hold correlation to the draw down depths and profitable periods seen in the past. As a general rule, if an out-sample test shows a MDD (max draw down) more than twice the previous MDD then the strategy may be curve fitted.
   
  
• ‘Sharp’ clusters in profits distribution: It is good to have a look at distribution of profits w.r.t. to the change in input parameters. For example, if input 24 gives profitable results but 23 and 25 are unprofitable then it means that the system is ‘instable’ between the input range 23-25. Such system will be unprofitable in actual trading because it depends on particular optimum criterion. Likewise if the system is profitable with input 40 and its profitability gradually decreases as the range is shifted >50 or <30 than it is much more stable with the parameter 40. From a trader's perspective, finding single sharp peak is useless for trading because that result would be instable (too fragile) and not replicable in real trading.
  

[Do Or Die 10]

Why does the computer optimized strategy performs badly in actual trading?

 

The price data consists of some deterministic signal or pattern (to be used for trading) plus spurious random price movements which are generally referred to as noise.

 

One man's noise is another man's information, so I'll like to elaborate on it. My favorite definition of noise is: what the market does between an entry and exit. For example, you see a pattern formation and buy on breakout at 60 with stop loss at 55 and target 70 respectively. The price action between 55 to 60 is all noise; as well as the price action between 60 to 70. This is a simple example, the noise 'detected' by your system will depend upon the actual strategy inclusive of scaling in/out.

 

The computer optimization of a system’s input parameters on the test data produces performance results that fit the signal AND the noise. The noise process does not repeats in the same way in out-of-sample data or real time (noise is complete randomness!). After an optimization run on the test data, it is difficult to determine whether the input variables chosen from the optimization run have curve fit the noise, which will not be repeated, or have successfully modeled the price signal. The profits on out-of-sample data from choosing the input parameters that generate the best performance results from the optimization fit on the test data will usually disappear in practice, unless the above steps are followed.

Edited November 24, 2011 by Do Or Die

[Do Or Die 10]

The optimization process I typically follow for stocks (while keeping in mind the points in article):

 

1. Divide the stocks in sets, each having minimum 20 and max 100

- Fast moving stocks like FSLR, NOV, WLP, BIDU etc.

- Slow moving stocks like KO, DOW, ABT, MO etc.

- Low priced stocks between 20 to 40

- High prices stocks above 100

- Large caps

- Stocks which have a thin order book

- Set which I think (before optimization) is best suitable for this particular strategy

- Mixed set containing all of the above

 

2. Run optimization on mixed set. Use a stable parameter from this set to back test (out of sample) on rest of the sets.

 

3. After optimization the strategy should show performance improvement in ALL of the sets. If it does not, I try to find a practical reason related to those stocks like too volatile, more susceptible to gaps ets. If I cannot find a valid reason, keep the original parameters and discard optimization results

 

4. Repeat above steps on each premise used for system development to test whether it adds edge or is just redundant. If the strategy does not makes money on a particular set its ok, but it should not lose money after trading costs.

 

I keep separate sets for intraday and EOD trading systems.

[Adrian 10]

Who is the crazy person who is trying to take the topic off thread by asking about copyright? LOL Seriously. This is a topic on system testing and optimisation issues. A topic discussed and written about by hundreds if not thousands of people around the world for more years than I care to remember. There is no copyright on such a topic.

 

This person should be asking themselves why they even broached the irrelevant subject of copyright, and cannot bear to talk about what is really important? We all know, that you know copyright is not relevant here, but it gives an insight into peoples personal hangups and why they fail at trading.

[Balazs227 10]

Hy,

 

You have a very nice and interesting thread going on here. I am very much interested in optimization but sadly so far I could only find one book related to the topic and to be honest it was a bit above my head. So I would be very thankful to you if you could point me to some material in this subject. Thanks!

[Adrian 10]

Try ths book Balazs. Gives a good grounding of all the factors to consider.

 

The Evaluation and Optimization of Trading Strategies by Robert Pardo

 

Trading Sytems That Work by Thomas Stridsman

[Balazs227 10]

Try ths book Balazs. Gives a good grounding of all the factors to consider.

 

The Evaluation and Optimization of Trading Strategies by Robert Pardo

 

Trading Sytems That Work by Thomas Stridsman

 

 

Thank You very much! I will pick them up as soon as I can.

[Do Or Die 10]

You may also like to skim through Evidence Based TA (http://www.traderslaboratory.com/forums/books/3778-evidence-based-technical-analysis.html), delivers some solid concepts on system design.

[Albnd 10]

Hi,

 

Optimization refers to the combinatorial search over a range of system input values on price data defined over a fixed number of bars for the cases that produce the best system net profits or some other selected performance variable. It is unavoidable because markets lack stationarity in any form at all and so adaptability is essential for a trading system. By default it is equivalent to curve fitting because you are searching for ‘best fitting’ input parameters. In practice, you can use optimization results to model a more ‘adaptive’ system.

 

Running an optimization can be the easiest way to look for parameters which the market favors in different phases or time intervals. A simple method is to map the changes in optimum input range with a market characteristic such as volatility or trendiness. This in turns helps to build a layer of adaptability or regime analysis which is essential for any trading system.

 

 

 

 

Important Points To Avoid Curve-fitting

 

• Simulations should be close to actual execution: Perhaps the most important aspect of backtesting/optimization is to avoid simulations which cannot be executed in practice similarly as the simulation. For example, in intraday systems, optimizing a strategy which takes profits which are less than 5 times of bid-ask spread will likely be meaningless. Similarly, for EOD systems using the optimized performance on 2008 to trade/test during 2010 can be unprofitable because there was a dramatic shift in trading regimes over these periods.
   
  
• Premise based system design: A large number of input parameters or technical indicators lead way to curve-fitting. Using a premise-focused approach helps you keep the parameters minimum and the strategy simple. While modeling a strategy it is easy to get lost on the combination of entry/exit conditions; however, using this approach will help you to focus on the characterstic of market behavior that you intend to exploit in the first place, and which is essential to the ‘edge’ of the system. It is extremely important to code simple “elegant” strategies in order to avoid added complexity which will result in curve fitted solutions.
   
  
• Sufficient sample data for simulation: This is a common adage in statistics; there is hardly anything to elaborate. If the period for which the system is tested is small enough, you may need to test it across a large number of symbols. The (expected) number of trades on which the optimization is performed should also be large enough. I prefer minimum 200 trades (ideally more than 1000) which may be tested on more than 100K bars of data. Small sample data increases the chances of curve fitting to a great extent.
   
  
• Keep your system symmetric: One of the first ideas new traders have when they start system development is to have a separate criteria for Entry/Exit and Short/Long trades (for example using an indicator parameter 20 for long entries but 15 for shorts). This is because separate criteria increase the number of conditions (and degrees of freedom). If a system from profitability to losses depending only on complicated combination of rules it is less likely to be robust. Even though on daily time frame bull markets tend to have totally different characteristics than bear markets, it is better to have a long(short) only strategy than depending on complicated combinations. Also testing the same strategy over different time frames gives information about its stability.
   
  
• Out-of-sample testing periods: A very common practice in system development is go for rolling backtests or walk forward optimization. Certainly the out-sample test doesn’t have to be profitable (as all strategies have profit and draw down periods) but it must at least hold correlation to the draw down depths and profitable periods seen in the past. As a general rule, if an out-sample test shows a MDD (max draw down) more than twice the previous MDD then the strategy may be curve fitted.
   
  
• ‘Sharp’ clusters in profits distribution: It is good to have a look at distribution of profits w.r.t. to the change in input parameters. For example, if input 24 gives profitable results but 23 and 25 are unprofitable then it means that the system is ‘instable’ between the input range 23-25. Such system will be unprofitable in actual trading because it depends on particular optimum criterion. Likewise if the system is profitable with input 40 and its profitability gradually decreases as the range is shifted >50 or <30 than it is much more stable with the parameter 40. From a trader's perspective, finding single sharp peak is useless for trading because that result would be instable (too fragile) and not replicable in real trading.
  

 

Great, very nice post. Thank you so much