Effective Backtesting

Main points:

• backtesting is an attempt to learn the future by looking at the past
• the connection between past and future is chaotic
• if we look at the wealth curve of a backtest, everything we infer about that result could be wrong
• mimicking the backtest with random trades provides a true picture of when the strategy works and when it doesn’t
• it is useful to do the backtest with multiple starting portfolios
• the only truly out-of-sample test is real trading
• herd risk is a problem
• we played the minority game
• optimism bias is a problem

annotated slides (pdf)

Presented 2010 November at the Thalesians.

There is a video of the talk on the Thalesian website (towards the bottom).