Evaluation Protocols, Backtesting, and Data Leakage

"My equity curve climbed so smoothly that I started shopping for a yacht. Then someone asked which data my scaler had seen, and the yacht sailed away without me."

A Backtest That Looked Far Too Good to Be True

The previous section, Section 2.5, built feature pipelines that respect the arrow of time. Now we ask the harder question: once a model is trained, how do you measure it without fooling yourself? The order matters, because every leakage path from Section 1.6 reappears here at evaluation time, and a flawless training pipeline can still be graded by a dishonest protocol. The thread of this section is a single rule with many consequences: you may only score a model on data that lies strictly in the future of everything it learned from, and "everything it learned from" includes every scaler, statistic, and overlapping label, not just the model weights.

1. Why i.i.d. Cross-Validation Is Wrong for Time Series Beginner

Ordinary k-fold cross-validation rests on one assumption: the samples are independent and identically distributed, so any partition into folds is as good as any other and shuffling is harmless. Time series violate this at the root. Observations are autocorrelated, the distribution drifts (the non-stationarity of Section 1.6), and there is a privileged direction: the past explains the future, never the reverse. When shuffled k-fold scatters samples randomly into folds, it routinely places a sample from time $t+5$ into the training fold used to predict a sample at time $t$. The model is then graded on its ability to interpolate between observations it has effectively already seen, which is a far easier task than the forecasting it will actually face. The score is optimistic, and the optimism grows with autocorrelation and drift.

The fix is conceptual before it is mechanical. Evaluation must mirror deployment, and deployment has a temporal boundary: at decision time $\tau$, the model knows the past $\{t \le \tau\}$ and must predict the future $\{t > \tau\}$. Every honest protocol in this section is just a disciplined way of sweeping that boundary $\tau$ forward through the data while never letting any computation on the right of the wall influence anything on the left.

2. Time-Aware Validation: Hold-Out, Walk-Forward, and Rolling Windows Beginner

The simplest honest protocol is a single temporal hold-out: train on the first portion of the series, test on the tail, never the reverse. It is correct but wasteful, since it grades the model on one window and one regime. Backtesting generalizes it by sweeping the split forward, refitting at each step, and averaging the out-of-sample errors, so the score reflects many regimes rather than one. Two sweep shapes dominate. The expanding window (also called walk-forward) keeps the training set anchored at the start and lets it grow, so the model always uses all available history, which suits stable processes and small data. The rolling window keeps the training set a fixed length and slides both ends forward, so the model always forgets the distant past, which suits non-stationary processes where old data is actively misleading. Figure 2.6.1 contrasts the two.

Scikit-learn's TimeSeriesSplit implements the expanding-window scheme directly, and with its max_train_size argument it becomes a rolling window. Code 2.6.1 shows both and confirms that every test index is larger than every training index, the mechanical signature of a valid temporal split.

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

T = 16
X = np.arange(T).reshape(-1, 1)        # stand-in for an ordered time series

# Expanding window: training set grows, test block is the next chunk.
expanding = TimeSeriesSplit(n_splits=4, test_size=2)
print("EXPANDING")
for k, (tr, te) in enumerate(expanding.split(X), 1):
    assert te.min() > tr.max()          # every test index is in the future of train
    print(f"  fold {k}: train={list(tr)}  test={list(te)}")

# Rolling window: cap the training length so old data is forgotten.
rolling = TimeSeriesSplit(n_splits=4, test_size=2, max_train_size=6)
print("ROLLING (max_train_size=6)")
for k, (tr, te) in enumerate(rolling.split(X), 1):
    print(f"  fold {k}: train={list(tr)}  test={list(te)}")

EXPANDING
  fold 1: train=[0, 1, 2, 3, 4, 5, 6, 7]  test=[8, 9]
  fold 2: train=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]  test=[10, 11]
  fold 3: train=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]  test=[12, 13]
  fold 4: train=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]  test=[14, 15]
ROLLING (max_train_size=6)
  fold 1: train=[2, 3, 4, 5, 6, 7]  test=[8, 9]
  fold 2: train=[4, 5, 6, 7, 8, 9]  test=[10, 11]
  fold 3: train=[6, 7, 8, 9, 10, 11]  test=[12, 13]
  fold 4: train=[8, 9, 10, 11, 12, 13]  test=[14, 15]

3. Purging and Embargo for Overlapping Horizons Intermediate

The split-point rule is necessary but not sufficient when features or labels span time. Suppose a label at time $t$ is the return realized over the next $h$ steps, or a feature is a rolling statistic over a trailing window. Then a training sample whose label window overlaps the test period shares information with the test set even though its timestamp sits before the split. Marcos Lopez de Prado named the two corrections. Purging removes from the training set any sample whose label interval overlaps the test interval, and embargo additionally drops a short gap of training samples immediately after the test block, because serial correlation lets information leak backward across the boundary just past it. Let sample $i$ carry an information interval $[t_i^{\text{start}}, t_i^{\text{end}}]$ (for a horizon-$h$ label, $t_i^{\text{end}} = t_i + h$). With test interval $[\tau, \tau']$, a training sample is purged when its interval overlaps the test interval,

and the embargo extends the forbidden zone past the test block by an embargo length $e$, removing any training sample with $t_i^{\text{start}} \in (\tau', \, \tau' + e]$. Without purging, the model is graded on labels it partly memorized; without embargo, the autocorrelation just after the test window does the leaking instead. Both are invisible to the split-index check of Code 2.6.1, which is exactly why they are dangerous.

4. Forecast Metrics and Their Pitfalls Intermediate

An honest protocol still needs an honest score. The scale-dependent errors are the familiar pair. Mean absolute error, $\mathrm{MAE} = \frac{1}{n}\sum_i |y_i - \hat y_i|$, is robust and interpretable in the unit of the series. Root mean squared error,

penalizes large misses harder and shares the series unit, but it is sensitive to outliers and not comparable across series of different scale. The percentage errors try to fix comparability and introduce new traps. Mean absolute percentage error, $\mathrm{MAPE} = \frac{100}{n}\sum_i |y_i - \hat y_i| / |y_i|$, divides by the truth and therefore explodes when $y_i$ is near zero and is undefined at exactly zero, so it is meaningless for intermittent demand or any series that visits the origin. The symmetric variant sMAPE divides by $(|y_i| + |\hat y_i|)/2$, which bounds it but still misbehaves when both terms are tiny. The clean answer is the mean absolute scaled error, which divides the model's MAE by the in-sample MAE of a naive baseline,

where the denominator is the seasonal-naive error at season length $m$ (use $m=1$ for the random-walk baseline). MASE is scale-free, defined whenever the series is not constant, and carries an instant interpretation: below one means you beat the naive forecaster, above one means you lost to it. For probabilistic forecasts, the pinball (quantile) loss scores a predicted quantile $\hat q_\alpha$ at level $\alpha$ against the realized $y$,

which is minimized in expectation exactly at the true $\alpha$-quantile, so averaging it across levels grades a whole predictive distribution. The full machinery of probabilistic scoring rules, calibration, and conformal intervals is developed in Chapter 19; here the pinball loss is the bridge metric. Table 2.6.1 summarizes when to reach for each, and Code 2.6.2 implements MASE and the pinball loss from scratch.

import numpy as np

def mase(y_true, y_pred, y_train, m=1):
    """Mean Absolute Scaled Error. m is the season length (1 = random walk)."""
    naive = np.abs(np.diff(y_train, n=1) if m == 1
                   else y_train[m:] - y_train[:-m]).mean()   # in-sample naive MAE
    return np.abs(y_true - y_pred).mean() / naive

def pinball_loss(y_true, q_pred, alpha):
    """Quantile (pinball) loss for a predicted alpha-quantile."""
    diff = y_true - q_pred
    return np.mean(np.maximum(alpha * diff, (alpha - 1.0) * diff))

y_train = np.array([10., 12., 11., 13., 12., 14., 13., 15.])
y_true  = np.array([16., 14., 17., 15.])
y_pred  = np.array([15.2, 14.4, 16.1, 15.3])      # a decent point forecast

print(f"MASE (vs random-walk naive): {mase(y_true, y_pred, y_train):.3f}")
# Probabilistic: a low and a high quantile around the same point forecast.
print(f"pinball @0.1: {pinball_loss(y_true, y_pred - 1.5, 0.1):.3f}")
print(f"pinball @0.9: {pinball_loss(y_true, y_pred + 1.5, 0.9):.3f}")

MASE (vs random-walk naive): 0.382
pinball @0.1: 0.175
pinball @0.9: 0.125

5. Backtest Realism: Costs, Drift, Multiple Testing, and Baselines Advanced

A protocol can be leakage-free and still flatter a strategy if it models a frictionless, stationary, single-experiment world that does not exist. Four realism gaps matter most. First, costs and latency: in finance every trade pays a spread, a commission, and slippage, and the signal must be computable and actionable before the bar it trades on closes, so a backtest that fills at the same price it observed is fiction. Second, non-stationarity makes the past an imperfect guide; a strategy validated on one regime can fail the moment the regime turns, which is why the rolling window of Figure 2.6.1 and the drift handling of Chapter 20 exist. Third, multiple testing and backtest overfitting: if you try hundreds of strategy variants and keep the best by its backtest, the winner's superiority is largely the luck of selection, and its out-of-sample edge collapses. Lopez de Prado's deflated Sharpe ratio and the probability of backtest overfitting were designed to quantify exactly this. Fourth, and simplest, always report against a naive baseline. A "good" RMSE is meaningless in isolation; what matters is whether the model beats seasonal-naive or the random walk, which is precisely what MASE encodes. If your elaborate forecaster cannot beat yesterday-repeated, you have learned that, not failed to. Figure 2.6.2 captures the warning sign: a curve too perfect to be honest.

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score

y = prices[5:] / prices[:-5] - 1.0                  # (1) horizon-5 forward return
X = StandardScaler().fit_transform(features)        # (2) scale whole dataset
scores = cross_val_score(Ridge(), X[:-5], y,        # (3) default shuffled KFold
                         cv=5, scoring="r2")
print("mean R2:", scores.mean())