Overfitting means your rules describe the noise in one historical sample rather than a pattern that repeats. It is the most common reason a strategy that looks excellent in a backtest disappoints with real money.
How it happens
Every time you try a variation (a different RSI threshold, another moving-average length, one more filter) and keep the one that looked best, you are searching. Search enough and something will look good by chance alone, the same way that among many coin-flippers someone will flip ten heads in a row.
The problem is not any single test. It’s the number of tests you ran and then forgot about. Researchers such as David Bailey and Marcos López de Prado have written extensively about this, including methods like the deflated Sharpe ratio that adjust a result for the number of trials behind it.
Warning signs in a report
- Very few trades. With little data, a few lucky trades can produce almost any curve.
- A result that depends on a few big winners. If removing a handful of trades erases the profit, the edge may not be real.
- Oddly specific parameters. A 37-bar average that works while 30 and 45 do not is a red flag.
- A strong in-sample result and a weak out-of-sample one.
- Many rules or conditions compared to the number of trades.
The best-trades-removed check
Backte.st’s Best trades removed check recomputes the compounded return of closed trades without the five most profitable ones. If the strategy still makes money, its result is spread across many trades. If it flips to a loss, most of the profit came from a few outliers, which are the hardest thing to repeat.
Habits that reduce overfitting
- Write the rule before you test it. Decide what you expect and why, then test once.
- Prefer simple rules. Fewer conditions mean fewer ways to fit noise.
- Keep a holdout and use it once. See out-of-sample testing.
- Check nearby parameters. A real effect rarely lives on a single exact number. See parameter robustness.
- Test across markets. A rule that only works on one ticker is suspicious. Backte.st lets you run up to 10 markets with the same rules.
- Count your attempts. If you tried twenty variations, judge the winner more harshly than if you tried one.
What it looks like in practice
Build a strategy like an RSI reversal in the lab, run it on a few stocks, then tweak the thresholds until one market looks great. Now look at the checks: you will often see the unseen period or nearby parameters turn into weak spots. That is overfitting made visible.