Is my backtest overfitted?

Forge optimizes the settings on one part of history, then judges them on years it has never seen. It counts how many settings you tried and marks the result down for it.

Check these first#

You can ask these of any backtest, with or without Oathra.

• How many settings did you try before this one? The more you tried, the more likely the best one is luck.
• Was any part of the price history kept back, and never used to choose the settings?
• Does the result still win after spread and commission?
• Are there enough trades to judge?
• Does it win in most slices of the history, or in one stretch only?

The checks Forge runs#

Each check asks one question of the result. A strategy that fails one is killed there, and the verdict says where.

• Cost check Does it still win after spread and commission?
• Sample check Does the test have enough trades to judge?
• Hold-out Does it hold on price history it was never tuned on?
• K-fold How often it survived across many slices of the same history.
• Walk-forward How often it survived when it was always tested into the future.
• Chance this is luck The odds that the winner you picked was luck.
• Deflated Sharpe The Sharpe ratio, with the credit for trying repeatedly taken back off.

Read more#

How the test works