In Sample and Out of Sample Size

[Xiao si 10]

I see an ongoing debate in some forums about the duration of the data used to design, test, and optimize trading systems.

 

On one side i hear that some like Sunny Harris use 12 months of data only. Other still demand that anything less than a couple of years is just fitting to noise.....

 

If we go back too far we end up seeing a mess of noise related to the top and subsequent rapid decline of 2008. Before that and it was the late stages of the bull market.

 

I also hear some talk that now the markets are more mean reverting than prior to 2008, when they were more trendy.

 

My opinion from all of this so far is that one year of recent data is probably better to build a system that works well as the markets 'currently' behave.

 

This leads to the Out of Sample period reserved for proving the consistency of the systems performace.

 

I'm interested in opinions of the systematic traders here as to what they personally like to use for these data samples and why.

 

Also, what is your preferred roadmap for development of these strategies? example:

-develop the idea or concept

-test the basic entry

-test the entry with basic trade mgt.

-optimize the basic trade management

-back-test in sample

-etc....

 

Thanks in advance to all those systematic traders in the house:thumbs up:

 

Cheers,

 

 

XS

[BlueHorseshoe 10]

It's all very complicated - Thomas Stridman's 'Trading Systems That Work' is probably the best book for reading up on this topic. Especially important are his thoughts one percentage-change scaled backtesting rather than dollar-based tests.

 

I use data prior to the in-sample as my out of sample (there's absolutely know reason why the market one year prior to the in-sample is any more likely to resemble it than the year after the in-sample does).

 

If you're testing a fixed percent stop-loss, say, then you're going to want to test this over a long period to get a result that reflects changes in market volatility. If, however, your stop was a dynamic, volatility-based stop, then there would be less of an argument for this.

 

Also, it depends on how frequently your system trades. If you were testing a long term trend-following system then you'd want at least 10 years in sample, I would say. Whereas an HFT firm may use just the last year or two. Significant sample size is the important factor.

 

I would definitely think about how you can remove variables from your system so that there is very little optimisation to be done.

 

Hope that helps.

[Xiao si 10]

It's all very complicated - Thomas Stridman's 'Trading Systems That Work' is probably the best book for reading up on this topic. Especially important are his thoughts one percentage-change scaled back-testing rather than dollar-based tests.

 

I use data prior to the in-sample as my out of sample (there's absolutely know reason why the market one year prior to the in-sample is any more likely to resemble it than the year after the in-sample does).

 

If you're testing a fixed percent stop-loss, say, then you're going to want to test this over a long period to get a result that reflects changes in market volatility. If, however, your stop was a dynamic, volatility-based stop, then there would be less of an argument for this.

 

Also, it depends on how frequently your system trades. If you were testing a long term trend-following system then you'd want at least 10 years in sample, I would say. Whereas an HFT firm may use just the last year or two. Significant sample size is the important factor.

 

I would definitely think about how you can remove variables from your system so that there is very little optimization to be done.

 

Hope that helps.

 

Thanks BH, i share your view on simplicity.

 

XS

[equtrader 10]

Xiao si, are you talking about EOD or intraday systems?

 

I know that most use 30% of the data for development and the remaining 30% for out of sample testing. Some argue that the in sample should be long enough to include a variety of market conditions. I agree. At least 10 years for EOD and 5 years for intraday is what I use. Other do not agree. This is trading.

 

As far as the roadmap you outlined, in my opinion it is a time consumig and inefficient process. It can be compared to trying to divide decimal numbers by hand instead of using a calculator. More and more traders turn to machine generated strategies. If you like indicators and evolutionary algorithms then Adaptrade is a popular program. If you like price patterns only then Price Action Lab is another popular program. The use of such programs automates development and testing with significant advantages. Good luck to you!

[Xiao si 10]

Xiao si, are you talking about EOD or intraday systems?

 

I know that most use 30% of the data for development and the remaining 30% for out of sample testing. Some argue that the in sample should be long enough to include a variety of market conditions. I agree. At least 10 years for EOD and 5 years for intraday is what I use. Other do not agree. This is trading.

 

As far as the roadmap you outlined, in my opinion it is a time consumig and inefficient process. It can be compared to trying to divide decimal numbers by hand instead of using a calculator. More and more traders turn to machine generated strategies. If you like indicators and evolutionary algorithms then Adaptrade is a popular program. If you like price patterns only then Price Action Lab is another popular program. The use of such programs automates development and testing with significant advantages. Good luck to you!

 

This is great stuff Equ....At the moment my focus is intra-day on the equity index futures in my time zone, so Asian, Australian and Eurpean. I'm actually using Adaptrade's Builder as well as developing my own systems as i learn to code.

 

I've got quite a bit of data now (thanks to a few on here) so i'll be working my way through some of that. I find that the last few years were pretty diverse in terms of market action. If i build a system on 2009-2010 data its more apt to perform on OOS data from 2010 - 2011 than if i use something that includes 2008. Talk about a rough patch.

 

Just wondered how people were using that period, avoiding it or using it?

 

Cheers,

 

 

XS

[Obsidian 10]

...Some argue that the in sample should be long enough to include a variety of market conditions. I agree. At least 10 years for EOD and 5 years for intraday is what I use. Other do not agree....

 

I think it MUST be long enough to include a variety of market conditions.

[Xiao si 10]

I think it MUST be long enough to include a variety of market conditions.

 

But should that variety include a period like we saw from July 2008 to December 2008? Should we expect a system to be able to trade through that huge increase in volatility and still be able to deal with a trending market?

 

I do think that different parts of the trading year are important though. I try and use data that covers all of these seasons evenly.

 

XS

[Obsidian 10]

But should that variety include a period like we saw from July 2008 to December 2008? Should we expect a system to be able to trade through that huge increase in volatility and still be able to deal with a trending market?

 

I do think that different parts of the trading year are important though. I try and use data that covers all of these seasons evenly.

 

XS

 

I think it is crucial to see how your system will perform if a part of the history repeats itself again. If you know your system's weaknesses, you can prepare yourself what to do next. if your system is not performing well under a certain market condition, then you can either choose not to trade or add some filters to the system...just my thoughts...

[Xiao si 10]

I think it is crucial to see how your system will perform if a part of the history repeats itself again. If you know your system's weaknesses, you can prepare yourself what to do next. if your system is not performing well under a certain market condition, then you can either choose not to trade or add some filters to the system...just my thoughts...

 

You know, i usually run it over that nasty 2008 decline just to see what it does, but I've never really looked it that way before...thanks, that's great advice.

 

XS

[karoshiman 10]

But should that variety include a period like we saw from July 2008 to December 2008? Should we expect a system to be able to trade through that huge increase in volatility and still be able to deal with a trending market?

 

I do think that different parts of the trading year are important though. I try and use data that covers all of these seasons evenly.

 

XS

 

Hi XS,

 

Why don't you do out-of-sample testing using the 2008 months you mentioned? I would be curious about how my system would have performed in that high vola period.

 

Regards,

k

 

EDIT: Sorry, posted too late...

[Xiao si 10]

Hi XS,

 

Why don't you do out-of-sample testing using the 2008 months you mentioned? I would be curious about how my system would have performed in that high vola period.

 

Regards,

k

 

EDIT: Sorry, posted too late...

 

Yeah, i think i'm going to start reserving that period for OOS testing and making a point of always testing over it.

 

Cheers,

 

 

XS

[BlueHorseshoe 10]

Yeah, i think i'm going to start reserving that period for OOS testing and making a point of always testing over it.

 

Cheers,

 

XS

 

I would be careful with this approach if I were you. A lot of the strategies that I trade or track performed conspicuously well in that 2008 period (to the extent that the equity curve pretty much brickwalls). This is probably because they are contrarian in nature and therefore doing roughly the opposite of whatever your strategy is trying to do when it struggles in 2008.

 

However, if I developed such a strategy now and then used 2008 for my out of sample period, then I would get a wildly optimistic and hopelessly wrong-headed impression of my strategy's likely performance.

 

. . . A much more interesting period to look at is 2007. This is when the volatility really started to ramp up, as all the higher tranched credit default swaps etc went belly-up behind the scenes - how did your strategy respond to this steadily increasing volatility? Did it adapt?

 

Hope that helps

 

Bluehorseshoe

[Obsidian 10]

You know, i usually run it over that nasty 2008 decline just to see what it does, but I've never really looked it that way before...thanks, that's great advice.

 

XS

 

you are welcome, just shared a piece of my experience

[Xiao si 10]

Great stuff guys thanks heaps...

 

There is a raging debate on this on another forum regarding OOS testing which then leads to the topic of optimization.

 

I sort of think that for my current tie frame, 15 minutes i need only a year of data to construct and 1 year for the first OOS period. Then i use a three to six month period reserved for optimizing...then WF from there. I know i'll get some flack for this...but that seems to work best. I think i'm getting too much variety in the market behavior if i use too much data to develop the system on. More recent data works better.

 

Opinions?

 

Cheers,

 

 

XS

[karoshiman 10]

 

... More recent data works better.

 

 

 

Hi XS,

 

More recent data works only better, if your different walk-forward testing periods prove so. Otherwise you did just curve-fitting to the recent data.

 

Regards,

k

[Xiao si 10]

Hi XS,

 

More recent data works only better, if your different walk-forward testing periods prove so. Otherwise you did just curve-fitting to the recent data.

 

Regards,

k

 

Exactly, and that's where the debate starts going about what is curve fitting and what is over fitting...Many are saying that curve fitting is ok for intraday systems as long as they prove themselves on OOS data. I'm still trying to get the difference between curve fitting and over-fitting.....

 

XS

[M4RC68 10]

Xiao,

 

In my opinion you have to worry about presenting your setup to make the greatest possible amount of operational situations. This not only concerns to OOS, but the combination IS + OOS.

 

Why?

 

1) When you define an IS window period, you need to worry about the degrees of freedom for your model optimization. You need to make room for the model to find the best combinations of parameters to the setup´s rules. Thus, for ranges of 200 MA periods ex., you need to consider that the period will comprise your model extension.

 

2) Once granted an appropriate period, you must remember that the optimizer will (try to) set global maximum to your objective function, specifically for that IS window period. So you're training the model to get the best performance according to that market conditions. You are fitting your model to that market.

 

3) Once you finished the optimization, you will apply the model on the OOS next window period. And see how it behaves. The better will be the result, the better is the predictive capacity of your model. But you must remember the IS period. When you optimized, which was the IS period kind of market? Trend, range, up, down. And now, in the OOS? The market may be similar or completely different. What counts are the combinations: IS in trend and OOS in range, IS in uptrend and OOS in downtrend, OOS in range and IS in up trend, and so on.

 

The final analysis will be done on the cumulative results of all OOS windows. I use WFA precisely because of the various possible IS+OOS testing combinations and thus manage to evaluate the predictive capacity of the model in many different situations. So I can have more confidence when it comes to operational phase.

 

Another important aspect of the OOS window is the number of operations. It is not recommended to have windows with few operations. In my case its worries me more because the strategy (I prefer setups with high trades number) and less because of the statistics. That's because I consider all the operations of all OOS windows to analyze the possible distributions with MC.

 

That's my opinion. Sorry for my poor english and I hope you understood my point of view.

 

Regards

Marcelo

[gianno 10]

Exactly, and that's where the debate starts going about what is curve fitting and what is over fitting...Many are saying that curve fitting is ok for intraday systems as long as they prove themselves on OOS data. I'm still trying to get the difference between curve fitting and over-fitting.....

 

XS

 

I think the point is not fitting or over-fitting. The point is if your parameter adjustment creates a completely different random animal. There is an interesting thread in the Adaptrade forum where a user tries to reverse engineer a system and he ends up getting a different each time he tries. I think this is not only fitting but random fitting:

 

Reverse engineering a known system - Adaptrade Builder | Google Groups

 

I like the classification of curve-fitted systems this guy has come up with:

 

Curve-fitting and Optimization | Price Action Lab Blog

[Xiao si 10]

I think the point is not fitting or over-fitting. The point is if your parameter adjustment creates a completely different random animal. There is an interesting thread in the Adaptrade forum where a user tries to reverse engineer a system and he ends up getting a different each time he tries. I think this is not only fitting but random fitting:

 

Reverse engineering a known system - Adaptrade Builder | Google Groups

 

I like the classification of curve-fitted systems this guy has come up with:

 

Curve-fitting and Optimization | Price Action Lab Blog

 

Yes, i agree. I fact i have been only optimizing MM stops and Exits for sometime now. It makes sense as well if you are using indicators for the basis of a system and you optimize those that the curve fit is much more likely to fail, almost certin IMO. Them being the PAL, using only price patterns of course they are selling that particular POV.

 

I've noticed that the majority of systems that Adaptrade's Builder constructs are actually more Price Action based even if i do not narrow down the range of choices to this, especially the volatility systems.

 

Thanks for the link..curious of the others POV too.

 

Cheers,

 

 

XS

[gianno 10]

Yes, i agree. I fact i have been only optimizing MM stops and Exits for sometime now. It makes sense as well if you are using indicators for the basis of a system and you optimize those that the curve fit is much more likely to fail, almost certin IMO. Them being the PAL, using only price patterns of course they are selling that particular POV.

 

PAL posted another article yesterday on their blog about curve-fitting. I think it points to programs that offer a lot of different stop options to nicely curve-fit results

 

Article link here

[Albnd 10]

I would definitely think about how you can remove variables from your system so that there is very little optimisation to be done.

 

I would possibly comment on this, even if the thread is no longer updated.

 

On another forum I had the occasion of discussing about the usefulness of optimizations and especially of walk forward optimizations with reference to the INS/OOS balance.

 

My point was basically as follows: WFO is probably NOT always a must in comparison with traditional optimization. In particular conditions of market and time, there's a chance that a WFO is NOT necessary and/or useful.

 

Just a specific case, to make my point clear.

 

I’m currently testing a (purely automated/mechanical) strategy with very few parameters, and most of them seem to be unrelated to any kind of periodical change in the market I‘m trading (QM futures contract). I basically started asking myself: is it necessary/useful to include these parameters in the optimization, if they don’t seem to be correlated to any specific change in market behavior? (i.e. they basically seem to casually change from one period to another).

 

One of my mechanical strategies uses a fixed stop loss and evaluates the opening session gap size before taking action. I would usually prefer to set the stop loss just once, by considering the general MAE of the historical backtest. As for the gap size, I usually cannot see any particular change between the sizes of 2002 gaps and the ones related to today’s markets (I can provide data for this). So, does it make sense to include these parameters in the optimization?

 

I agree with your suggestion about reducing the number of the parameters. In fact, my point is that a specific optimization may not be needed for strategies that have very few variables/parameters.

 

By the way, I was replied that in most cases even a strategy with no parameters at all can hide an "implicit" (over-fitting) adaptation to the markets because of the way it's programmed and conceived.