Avoiding Curve Fitting

[cunparis 10]

I have developed 3 indicators that each test profitably. I've determined the optimal parameters by optimization (periods, thresholds, etc.). I do not expect to get the same results in the future, but I prefer to use the optimized values rather than some arbitrary values.

 

My question is this: I'm now working on combining these 3 into one signal (short, flat, long). I've tried two different approaches to do this:

 

1 - I use the optimal parameters that I determined on each indicator individually

 

2 - I re-optimized all parameters together.

 

#1 seems to be more realistic, with the acknowledgment that the performance will not be the same as the backtests, due to the performance of each system not being the same. This I know. So the final results will probably not be as good.

 

#2 - Seems to be more optimal, with an even stronger acknowledgment that the results will not be as good as the backtest. However there is a greater risk of curve fitting due to the increased rules and degrees of freedom. In defense of the optimization I will say that lots of attempts produced unacceptable results so I believe that if optimization finds something good say PF > 3.0 then it's very likely to be positive in forward testing even though the PF will most likely be less.

 

I'm curious what people think about these two approaches. I am currently forward testing both #1 & #2 but since they trade on daily charts and not very often, it will take a while to have something meaningful.

 

I've developed systems that have held up and systems that have fallen apart. I understand the limitations of backtesting and automation. So I prefer not to debate that but focus on which approach would be more optimal (and not necessarily more realistic).

[swansjr 10]

I have developed 3 indicators that each test profitably. I've determined the optimal parameters by optimization (periods, thresholds, etc.). I do not expect to get the same results in the future, but I prefer to use the optimized values rather than some arbitrary values.

 

My question is this: I'm now working on combining these 3 into one signal (short, flat, long). I've tried two different approaches to do this:

 

1 - I use the optimal parameters that I determined on each indicator individually

 

2 - I re-optimized all parameters together.

 

#1 seems to be more realistic, with the acknowledgment that the performance will not be the same as the backtests, due to the performance of each system not being the same. This I know. So the final results will probably not be as good.

 

#2 - Seems to be more optimal, with an even stronger acknowledgment that the results will not be as good as the backtest. However there is a greater risk of curve fitting due to the increased rules and degrees of freedom. In defense of the optimization I will say that lots of attempts produced unacceptable results so I believe that if optimization finds something good say PF > 3.0 then it's very likely to be positive in forward testing even though the PF will most likely be less.

 

I'm curious what people think about these two approaches. I am currently forward testing both #1 & #2 but since they trade on daily charts and not very often, it will take a while to have something meaningful.

 

I've developed systems that have held up and systems that have fallen apart. I understand the limitations of backtesting and automation. So I prefer not to debate that but focus on which approach would be more optimal (and not necessarily more realistic).

 

I use step 1. Don't optimize together. I always test different "indicators" or rules in isolation then I bring them together one at time. If one rule does not contribute to making the system better I don't re-optimize it - I dump it.

 

Good systems, in my humble opinion, only need a 2-3 basic rules. Your key trading concept shouldd work well without much, if any optimization.

[cunparis 10]

I use step 1. Don't optimize together. I always test different "indicators" or rules in isolation then I bring them together one at time. If one rule does not contribute to making the system better I don't re-optimize it - I dump it.

 

Good systems, in my humble opinion, only need a 2-3 basic rules. Your key trading concept shouldd work well without much, if any optimization.

 

Thanks for the feedback. I did a lot of forward testing this weekend. What I found was that performance going forward was pretty good until the past few years. Then even if I reoptimized it didn't walk forward well. i think it's due to changing from bull to bear and from the increased volatility. At this point I have doubts about the predictive capability. I'm going to give it a few more goes.

 

I'm using a moving average difference for the main signal, so that's 2 rules. Then I added an upper & lower threshold, that's 2 more. I think that's too many. The reason is in some of the optimizations (3-4 years, 100+ trades) I'd have moving averages like 5,6 and other times 7,5. This didn't make sense because having a faster average slower than the slow (inverting them) would effectively inverse all the signals. So I got suspicious.

 

I think I need to find a way to make an indicator without using 2 moving averages. It's too much curve fitting I think.

 

any ideas?

[swansjr 10]

Thanks for the feedback. I did a lot of forward testing this weekend. What I found was that performance going forward was pretty good until the past few years. Then even if I reoptimized it didn't walk forward well. i think it's due to changing from bull to bear and from the increased volatility. At this point I have doubts about the predictive capability. I'm going to give it a few more goes.

 

I'm using a moving average difference for the main signal, so that's 2 rules. Then I added an upper & lower threshold, that's 2 more. I think that's too many. The reason is in some of the optimizations (3-4 years, 100+ trades) I'd have moving averages like 5,6 and other times 7,5. This didn't make sense because having a faster average slower than the slow (inverting them) would effectively inverse all the signals. So I got suspicious.

 

I think I need to find a way to make an indicator without using 2 moving averages. It's too much curve fitting I think.

 

any ideas?

 

I do have a lot of ideas. I wish I had more time to experiment and build systems. But let me say this…

 

In my limited experience attempting to create a trading system with moving averages is very difficult. You can make strategies from basic indicators, but it's hard to do and can result in curve fitting. Try using common indicators in a different way - ways in which most people don't use them. For example, RSI is often used to highlight overbought and oversold conditions. Try using it as a trend indicator. This is just an example.

 

Price patterns are another way to go. Price breaking out from trading ranges or price behavior around opening day gaps are examples of trading without indicators.

 

In short, to make money in automated systems you are either 1) trend following or 2) trend fading. Decide what you want to do and focus on markets and market sessions that are favorable to those conditions. Your trading system does not need to trade all day or even every day. My best system trades about once a month as it fades extreme moves on a 5-minute chart. So be picky.

 

I think it's interesting to note that you stated " This didn't make sense because having a faster average slower than the slow (inverting them) would effectively inverse all the signals. So I got suspicious. "

 

Sounds like fading your original signal is a better idea. In other words, using your moving averages in a method that is "unusual" may produce better results than your original concept.