Out-of-Sample

[Tradewinds 12]

Out-of-Sample

An item within the range of a sample that does not conform to the mean of the sample.

 

When testing a strategy, there is something called walk-forward out-of-sample testing. This method of testing is used in an attempt to determine how the strategy will perform under various different situations.

 

It seems that the "Out-of-Sample" data, is data that is a subset of the in-sample data. I originally thought that the "Out-of-Sample" might be some kind of fictitious data generated from the real data.

 

I'm interested in any further clarification or thoughts on the subject.

Edited August 19, 2011 by Tradewinds

[Tradewinds 12]

TradeStation help on the Walk-Forward Optimizer states this:

 

"The Out-of-sample% is that percentage of the initial/moving data window which is allocated to the testing segment, for instance: if the moving data window is 5 years in total, then the first 80% (4 years) will be used as the in-sample segment while the last 20% (1 year) will be used as the out-of-sample (testing) segment.

 

There is no correct ratio of test data to produce good out-of-sample results.

 

Research suggests a ratio of between 3 and 9 to 1, i.e. an Out of-Sample% of between 10%-25% (1/10-1/4). Note that the default value of 20% is the recommended setting and can always be used without change."

 

So is actual historical data used for the "out-of-sample" data? Is the out-of-sample data not fictitious, but simply a subset of the sample data?

[Do Or Die 10]

So is actual historical data used for the "out-of-sample" data?

 

Yes, amibroker does out-of-sample testing same way.

 

Monte Carlo Simulations use 'fictitious' data.