What a backtest really tells you
A backtest is not a promise. It is a question asked to the past. Here is how I read one, and the five things I check before I trust it.
A good-looking equity curve is the easiest thing in trading to produce. Give me enough parameters and enough tries, and I can make almost any idea look brilliant on past data. That is exactly why a backtest needs to be read, not admired.
A backtest does not tell you what will happen. It tells you what would have happened, under assumptions you chose.
1. Is the sample big enough?
Thirty trades tell you almost nothing. A few hundred start to mean something, and thousands are better. The fewer the trades, the more the result is luck dressed up as an edge.
2. Did I fit it, or did it find something?
If I tried a hundred parameter sets and kept the best one, the best one is partly noise. I split the data: build the rules on one part, then run them, untouched, on a part they have never seen. If the edge disappears out of sample, it was never there.
3. Do the neighbours agree?
A real edge does not live on one magic number. If a stop of 20 points works but 18 and 22 fall apart, I do not trust 20. I want a plateau, not a spike.
4. Are the costs real?
Commissions, slippage, the spread on EUR/USD at the hour I trade. Small per trade, huge over thousands of trades. Many strategies that look great are just strategies that ignore costs.
5. Can I live through the worst part?
Every curve has its worst drawdown and its longest losing streak. I look at them first, not last. Then I reshuffle the trades many times (a Monte Carlo test) to see how bad it could have been with a different order. If I would have quit in the middle, the strategy does not work for me, whatever the final number says.
So what is a backtest for?
For killing ideas cheaply. Most of my ideas die in testing, and that is the point: better to lose an idea on data than money in the market. What survives goes live slowly, and the live results are compared against the backtest every week.
If you want to play with these ideas yourself, the Tools page on this site has calculators and experiments built around exactly these questions.
— Alon