Your Backtest Said 2.1 SQN and Live Says 0.6: Diagnosing Backtest-to-Live Divergence

Ask systematic and algo traders what they actually need from a journal, and the answer that comes up more than any feature request is some version of the same sentence: a way to vet whether a backtested strategy is holding up once it hits live market conditions. That is a narrower question than "is my strategy good," and it deserves a narrower, more mechanical answer than most of what gets written about it.

A backtest and a live run are two different measurements of the same underlying process, taken under different conditions and, almost always, on very different sample sizes. A gap between them is data, not a verdict. This is the order to work through it in.

1. First, Rule Out Variance

Before you touch the strategy, the backtest, or the broker, ask a smaller question: given how much your per-trade R already varies in the backtest, is a 40-trade live sample even capable of showing something close to 2.1 SQN? For a genuinely unchanged edge, the answer is often no, and the reason is arithmetic rather than opinion.

The standard error of a mean shrinks with the square root of the sample size, not with the sample size itself. Doubling your trade count does not halve your uncertainty — it shrinks it by a factor of about 1.4. Compare a 2,000-trade backtest against a 40-trade live sample and the ratio of standard errors is roughly √(2000/40) ≈ 7.1. In plain terms: the confidence interval around your live SQN estimate is about seven times wider than the one around your backtest's. A live result that looks dramatically worse can still sit comfortably inside that wider interval.

The check itself is mechanical. Take your live trades' R-multiples, compute the sample mean and sample standard deviation, then divide the standard deviation by the square root of the trade count to get the standard error of the mean. A rough 95% confidence interval is the mean R plus or minus roughly two standard errors. If the backtest's expected R-per-trade falls inside that interval, you do not yet have evidence of a broken strategy — you have a small sample, and the honest next step is to keep trading it at reduced size while the sample grows, not to abandon or overhaul it.

This cuts both ways. If your live sample has already grown past a hundred or so trades and the confidence intervals still do not overlap, variance has stopped being a plausible explanation on its own, and it is time to move to the causes below — starting with the one that is statistically the most common.

2. The Six Real Causes, Ranked by How Often They Are the Culprit

Once variance is ruled out, the divergence almost always decomposes into some combination of six things. They are listed here in roughly the order they are worth checking — not because the later ones are rare, but because the first two are the easiest to measure and the most frequently the whole story.

3. Measuring Your Actual Slippage

Because slippage and spread are usually the largest single contributor, they are worth measuring precisely rather than estimating. Log two prices for every live trade: the price your signal generated the instant it fired, and the price you were actually filled at. The difference, signed for trade direction and converted into the same units your backtest uses for cost, is your realized slippage for that trade.

Do this across enough trades to see the distribution, not just the average. Slippage is rarely symmetric — it tends to cluster near zero most of the time and then produce a fat tail around news releases, session opens, and thin overnight liquidity. A backtest corrected with the average slippage but not the tail will still diverge from live on exactly the trades that matter most, because the tail is where the cost actually concentrates.

Once you have that distribution, feed it back into the backtest's cost assumptions — either as a revised flat cost set to something closer to your median plus a tail allowance, or, if your backtesting tool supports it, by resampling from the measured distribution on every simulated fill. Re-run the backtest under those corrected costs before drawing any conclusion about the other five causes; a strategy that looked broken at zero-cost assumptions sometimes turns out to simply have never accounted for what it actually costs to trade.

4. The Overfitting Tell

There is one cause worth checking before all six of the above, and it is the least comfortable: if the 2,000-trade backtest was optimized in-sample and never validated on a genuine out-of-sample segment, the live divergence is not a mystery to be solved. It is the out-of-sample test the strategy never took, arriving three months late and at real cost instead of on a spreadsheet.

The tell is usually visible in the build history rather than the trade log: parameters chosen by scanning a grid for the best-performing combination over the full dataset, no reserved holdout period, and an equity curve that looks unusually smooth for the instrument being traded. If that describes your process, run walk-forward analysis retroactively on the same history — chop it into rolling in-sample and out-of-sample windows and check whether the out-of-sample segments were already decaying before you ever went live. It is uncomfortable to find that they were, but far cheaper to find out on historical data than to keep re-discovering it one live quarter at a time. If the strategy's parameters came from a wide sweep in the first place, the mechanics of exactly how that sweep can manufacture an in-sample edge that was never real are covered in the piece on grid search parameter optimization.

5. Regime Change vs. Broken Strategy

If the overfitting check comes back clean and the measured-cost reconciliation still leaves a gap, the remaining question is whether the market changed or the edge did. These produce similar-looking equity curves and call for opposite responses, which is what makes misdiagnosing them expensive in either direction.

The distinguishing tool is a rolling expectancy or rolling SQN — computed over your most recent trailing window of trades — set against the lifetime figure for the same strategy. A strategy in regime change typically shows a lifetime figure that is still intact, a rolling figure that has dropped, and a decline that lines up with something observable in the market itself: volatility compressing or expanding outside its historical range, a correlation that used to hold breaking down, a session's typical range shifting. A strategy that is actually broken shows the same rolling decline with no such correlate, or one that started immediately after a period of re-optimization against the same data now being traded — which is really the overfitting tell from the previous section resurfacing under a new name. The rolling-versus-lifetime comparison is the same diagnostic either way; what differs is what you find alongside it.

The response has to match the diagnosis, and getting it backwards is the costly failure mode. Regime change calls for reducing size and waiting for the environment to normalize, because the edge is still there and re-optimizing against a temporary condition just fits new noise on top of the old kind. A broken strategy calls for stopping, not waiting — there is no environment to wait out, and every trade taken while "giving it more time" is capital spent confirming a conclusion you already have reasonable evidence for.

6. The Reconciliation Loop

Once you have measured your real costs and ruled out the structural causes, the diagnostic converges into a loop rather than a single check. Re-run the backtest with your measured slippage, spread and commission substituted for whatever placeholder it used originally. Restrict both the corrected backtest and the live results to the same instrument, session window and parameter set — comparing a backtest run on the full trading day against a live run you only trade during London hours will manufacture a gap that has nothing to do with cost or edge.

Then compare the corrected backtest curve to the live curve, and judge the residual against a tolerance band rather than expecting an exact match — the confidence interval from step one, recomputed on the corrected numbers, is the right width for that band. If the gap has shrunk into that interval, the reconciliation is done: the strategy is behaving as expected once its real costs are accounted for. If a residual gap remains outside the band, it is telling you something specific — most often that one of the six causes was under-corrected — and the loop runs again: refine the measurement, re-run, re-compare.

This is also where Monte Carlo resampling earns its keep beyond the initial variance check: resampling the corrected backtest's trade sequence gives you a distribution of plausible equity paths rather than one line, and a live curve that sits inside that distribution's typical range is a stronger form of reconciliation than a single side-by-side comparison.

7. What to Log So This Is Answerable at All

Every step above depends on data that most trading setups do not capture by default. None of it is expensive to log going forward, and all of it is unrecoverable in hindsight if you skip it now.

• Expected entry and exit price — the price your signal generated, independent of what you were actually filled at.
• Actual entry and exit price — the realized fill, so slippage is a computed difference rather than a guess.
• Latency — the timestamp gap between signal generation and confirmed fill, per trade.
• Commission per trade — the real figure charged, not an assumed rate.
• A strategy tag — a stable identifier linking every live trade back to the exact backtest version and parameter set it came from, so a comparison months later is still apples to apples.

That last field is the one setups most often skip, and it is the one that turns this entire process from a one-time forensic exercise into something you can re-run in minutes the next time a number looks off. A journal that carries the strategy tag through from backtest to live trade is what lets you overlay any backtest on the live journal for the same strategy and see the reconciliation directly, rather than reconstructing it by hand from two separate exports.