Optimization

Optimization in backtesting is the process of finding the best parameter values for a trading strategy by testing many combinations against historical data. If your strategy uses a 20-period moving average, optimization asks: would 15 work better? 25? Every value from 5 to 100?

The goal: find parameter values that maximize a performance metric — typically Sharpe ratio, profit factor, or risk-adjusted return — while avoiding overfitting.

Optimization Methods

  • Brute-force (grid search): Test every combination. Simple but slow. 3 parameters with 20 values each = 8,000 backtests.
  • Genetic algorithms: Evolve parameter sets by combining the best performers. Efficient for large parameter spaces.
  • Bayesian optimization: Probabilistic models intelligently guess which combinations to try next. Finds good solutions faster.
  • Random search: Randomly sample combinations. Surprisingly effective and avoids grid artifacts.

Why Optimization Is Both Essential and Dangerous

Done right, optimization finds genuine parameter sweet spots. Done wrong, it produces beautifully overfit garbage.

  • Finding the signal: Markets have structure. Optimization finds the parameter ranges where your logic works, separating useful ranges from noise.
  • Overfitting risk: The more combinations tested, the higher the chance one looks great by luck alone. 10,000 parameter sets on the same data virtually guarantees impressive but meaningless results.
  • Parameter stability: Good optimization doesn’t just find the best single set — it maps the performance landscape to find stable regions where nearby parameters produce similar results.
  • Objective function matters: What you optimize for matters enormously. Raw return optimization overfits more aggressively than Sharpe ratio optimization.

The Optimization Paradox

More optimization often leads to worse live performance:

  • Zero optimization: Strategy may underperform its potential
  • Moderate optimization: Finds genuine sweet spots
  • Heavy optimization: Fits noise, degrades live performance
  • Extreme optimization: Beautiful backtest, catastrophic live trading

Concrete Examples

Grid Search on RSI

Optimizing an RSI mean-reversion strategy on SPY:

RSI PeriodBuy ThresholdSell ThresholdSharpe
1025700.82
1430700.95
1425750.91
1430750.88
2030700.79

The 14/30/70 combination is best, but nearby values (14/25/75, 14/30/75) also perform well. This plateau of good performance suggests a genuine effect — not overfitting.

Optimization Gone Wrong

A trader optimizes 8 parameters across 50 values each: 50^8 = 39 trillion combinations. A genetic algorithm tests 100,000 of them. “Best” result: 85% annual return, Sharpe 1.8. They go live and lose immediately.

The optimizer found a needle in a haystack — parameters that perfectly fit historical noise. With that many degrees of freedom, finding impressive-looking results is guaranteed even on random data.

Optimization Surface

Plotting Sharpe ratio as a heatmap for a dual moving average crossover on daily EUR/USD (fast MA: 5-30, slow MA: 20-100), you see a broad warm zone around fast=10-15, slow=40-60. Encouraging — the edge isn’t a single point but a region.

But there’s an isolated hot spot at fast=27, slow=85 that outperforms everything. That isolated peak is almost certainly noise. Choose the center of the broad plateau, not the isolated spike.

Wrong Objective Function

Optimizing for maximum net profit: the optimizer finds parameters that caught one massive winning trade in 2020. Remove that trade and the strategy is mediocre.

Optimizing for Sharpe on the same data: finds parameters producing consistent, moderate returns across the full period. The Sharpe-optimized version is far more robust.

Best Practices

1. Optimize for Robustness

Use Sharpe ratio, Sortino ratio, or profit factor as your objective — not raw return or net profit.

2. Use Parameter Sensitivity Analysis

After optimization, plot a 3D surface or heatmap across parameter ranges. Look for broad plateaus, not sharp peaks. A peak that disappears with a small parameter change is noise.

3. Limit Degrees of Freedom

Keep strategies to 2-4 optimizable parameters. Each additional parameter exponentially increases overfitting risk.

4. Use Walk-Forward Optimization

Optimize on rolling windows and test on the next out-of-sample period. This is the gold standard. Read more about walk-forward analysis.

5. Reserve Out-of-Sample Data

Hold back 20-30% of data — never optimize on it. After optimization, test on this held-out data exactly once.

6. Coarse Before Fine

Start with large parameter steps (MA periods in steps of 10). Only fine-tune if the coarse grid shows a clear, broad region.

7. Set Boundaries from Logic

Don’t test RSI periods from 1 to 1000. Use domain knowledge: 5-30 makes sense for RSI. Sensible boundaries reduce the search space and overfitting risk.

8. Penalize Complexity

When two parameter sets perform similarly, choose the simpler one — fewer rules, parameters closer to defaults.

Red Flags

  • The optimal parameters sit at the edge of your tested range
  • Small changes cause large performance swings
  • Optimal parameters differ wildly across sub-periods
  • You’ve tested more combinations than you have trades
  • The optimal parameters don’t make economic sense

The Process

1. Define strategy logic with clear hypothesis
2. Set parameter ranges based on domain knowledge
3. Run coarse optimization
4. Identify broad regions of good performance
5. Confirm with walk-forward analysis
6. Validate on out-of-sample data
7. Forward test before deploying capital

Skip any step and you risk deploying an overfit strategy that looked great in the backtest but fails with real money.

Resources