Out-of-sample testing means judging a strategy on data that played no part in designing it. It is one of the simplest and most effective defenses against overfitting.
In-sample vs out-of-sample
- In-sample data is what you looked at while building and tuning the rules.
- Out-of-sample (holdout) data is kept aside and only used to check the final rules.
If performance falls apart out of sample, the in-sample result was probably describing noise.
How the unseen-period check works
In Backte.st the Unseen period check splits each market’s history in time: the first 70% and the last 30%. It scores the last 30% on its own, as if you had never seen it, and reports its return and the number of closed trades since the split date.
Splitting by time, not randomly, matters. Markets have trends and regimes, and prices on neighboring days are related. A random split would leak information from the future into the past.
The catch: a holdout is only unseen once
If you run a test, see that the last 30% did badly, tweak the rules and run again until it looks good, the holdout is now part of your design process. It has become in-sample data. The report says this explicitly in its limitations.
Practical ways to keep it honest:
- Decide your rules first, using only the earlier part of the history if you want to explore.
- Look at the unseen result once and write down what you saw.
- If you must iterate, get fresh data: a different market, a later period, or a different candle size.
Reading the result
- Similar results in and out of sample: encouraging, though one split is still a small sample.
- Much worse out of sample: a classic sign of curve fitting, or of a regime change.
- Too few trades in the unseen period: the check can’t judge. Use a longer history or smaller candles.
Beyond one split
Professionals often use walk-forward analysis, which repeats the split many times as the window rolls forward. A single holdout is a simpler version of the same idea, and it already catches a lot.
Try it on any strategy in the lab and compare the unseen-period line with the full-period return.