Reproducibility and Temporal Data Pipelines

"Ask me what the price was last Tuesday and I will give you two honest answers: what I believe today, and what I actually knew on Tuesday. Only the second one is allowed to train a model. The first one is how backtests lie."

A Pipeline That Remembers Exactly What It Knew, and When

In the previous section, Section 2.6, we built time-aware splits and evaluation protocols that respect the arrow of time. Now we assemble the whole chapter into a single reproducible pipeline. Reproducibility in temporal machine learning is not the same problem as reproducibility in ordinary tabular work, because two distinct futures can leak into a temporal result: the future relative to the prediction timestamp (look-ahead bias, the cardinal sin named in Section 1.6) and the future relative to the experiment, namely a later revision of a data point that quietly overwrites what was originally recorded. Both must be sealed off. The sections below take the four properties in turn (point-in-time correctness, the leakage-safe transform discipline, offline/online parity, and determinism plus versioning) and then fold all of Chapter 2 into one reference blueprint you can carry into every later project.

1. Point-in-Time Correctness Intermediate

Point-in-time correctness is the single most important property of a temporal pipeline, more important than model choice, feature richness, or tuning. A feature computed for a prediction at time $t$ must reflect only the information that was actually available at $t$, nothing that was learned, observed, or revised afterward. Let $\mathcal{I}_t$ denote the information set available up to and including time $t$, the set of all observations the live system could have read at that instant. A point-in-time feature is then any function of that set and nothing more, and Figure 2.7.2 personifies the discipline as a guardian that hands a model only the smaller past it actually knew,

where $o_s$ is the value of observation $o$ as it was known at time $s$. The notation for information sets and filtrations used throughout the book is fixed in the unified table of Appendix A. The trap is that many real-world quantities are revised after the fact. Quarterly earnings are restated, sensor readings are recalibrated retroactively, a patient's lab value is corrected the next day, and a government statistic is published with a lag and then revised twice more. If your feature store hands back today's best estimate of last Tuesday's value, it has leaked a revision backward in time, and the model trains on knowledge no live system possessed.

Concretely, an as-of feature retrieval scans the history for the most recent record whose transaction time precedes the prediction timestamp. Code 2.7.1 shows the core join on a tiny restated series so the mechanism is unmistakable. The same logic, executed efficiently over millions of rows, is exactly what a production feature store performs internally, the subject of subsection 3.

import pandas as pd

# A metric that was revised after the fact. valid_time is the day the value
# is ABOUT; tx_time is the day we actually learned (or corrected) it.
history = pd.DataFrame({
    "valid_time": pd.to_datetime(["2026-01-05", "2026-01-05", "2026-01-12"]),
    "tx_time":    pd.to_datetime(["2026-01-05", "2026-01-08", "2026-01-12"]),
    "value":      [10.0, 13.0, 20.0],   # the 2026-01-05 fact was restated 10 -> 13
})

def as_of(history, valid_t, known_by):
    """Value for `valid_t` as it was known on or before `known_by`."""
    m = (history["valid_time"] <= valid_t) & (history["tx_time"] <= known_by)
    rows = history[m].sort_values("tx_time")
    return rows["value"].iloc[-1] if len(rows) else float("nan")

asof_jan5 = as_of(history, pd.Timestamp("2026-01-05"), pd.Timestamp("2026-01-05"))
today_jan5 = as_of(history, pd.Timestamp("2026-01-05"), pd.Timestamp("2026-02-01"))
print(f"as known on 2026-01-05 : {asof_jan5}")   # the honest, point-in-time value
print(f"as known on 2026-02-01 : {today_jan5}")  # the leaked, revised value

as known on 2026-01-05 : 10.0
as known on 2026-02-01 : 13.0

2. The Leakage-Safe Pipeline Intermediate

Point-in-time correctness governs the raw data; the leakage-safe transform discipline governs everything you compute from it. Section 1.6 stated the rule and Section 2.6 built the splits; here we make the two airtight by wrapping every stateful transform (scalers, imputers, encoders) and the estimator into a single scikit-learn Pipeline, then handing that object to a time-aware splitter. The pipeline guarantees that fit touches only the training fold and that the fitted statistics flow forward to transform the test fold, never the reverse. There is no loose fit_transform over the whole array to leak the mean and variance of the future into the past, because the split boundary is the pipeline's responsibility rather than yours.

Code 2.7.2 runs the correct end-to-end pattern: build supervised windows from a drifting series (the windowing of Section 2.5), then evaluate an impute-scale-model pipeline with TimeSeriesSplit so that on every fold the test rows lie strictly in the future of the training rows and every transform is fit on the past alone.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import TimeSeriesSplit, cross_val_score

rng = np.random.default_rng(7)
T = 720
t = np.arange(T)
series = 0.03 * t + np.sin(t / 10.0) + rng.normal(0, 0.5, T)   # non-stationary

# Supervised windows: predict series[i] from the previous L lags (Section 2.5).
L = 12
X = np.stack([series[i - L:i] for i in range(L, T)])
y = series[L:]
X[rng.random(X.shape) < 0.02] = np.nan   # a few realistic missing entries

# Every stateful step lives INSIDE the pipeline, so each is fit on train only.
pipe = Pipeline([
    ("impute", SimpleImputer(strategy="mean")),   # mean from the TRAIN fold alone
    ("scale",  StandardScaler()),                 # stats from the TRAIN fold alone
    ("model",  Ridge(alpha=1.0)),
])

# TimeSeriesSplit guarantees every test fold is in the future of its train fold.
tss = TimeSeriesSplit(n_splits=5)
scores = cross_val_score(pipe, X, y, cv=tss,
                         scoring="neg_root_mean_squared_error")
print("per-fold RMSE:", np.round(-scores, 3))
print(f"mean forward-chaining RMSE: {-scores.mean():.3f}")

per-fold RMSE: [0.601 0.557 0.583 0.546 0.572]
mean forward-chaining RMSE: 0.572

3. Feature Stores and Online/Offline Parity Advanced

A model is trained offline on historical features and served online on live ones. Training/serving skew is the failure where the same logical feature is computed differently in the two settings, so the model meets inputs at serving time that it never saw in training. The classic instance is a thirty-day rolling average implemented one way in a batch SQL job for training and another way in a streaming service for serving, differing in window edges, time zone, or null handling, with the result that the offline metric and the online behavior diverge for reasons that have nothing to do with the model. A feature store exists to kill this skew: it computes each feature once from a single definition, materializes the historical values into an offline store for training (served through the point-in-time as-of join of subsection 1) and the latest values into an online store for low-latency serving, so the training feature and the serving feature share one source of truth.

Feature stores are deployment infrastructure, so the full treatment belongs with operations: Chapter 34 develops online/offline materialization, serving latency, and feature monitoring as part of temporal MLOps. For this chapter, the point is narrower and load-bearing: parity is a data-engineering property you design in now, not a deployment patch you bolt on later.

4. Determinism and Versioning Advanced

Point-in-time correctness and parity make a pipeline honest; determinism and versioning make a result reproducible. A run reproduces only if four things are pinned: the random seeds (so stochastic steps repeat), the dependency versions (so the same code path executes), the data version (so the same bytes go in), and the configuration (so the same knobs are set). Of these, data versioning is the one temporal practitioners most often neglect and most need, because the data itself drifts and revises underneath them. Logging the data vintage, the transaction-time cutoff and a content hash of the exact rows, with every model is what lets you answer months later the question "what did this model actually know?" A result without its vintage is a number without provenance.

Code 2.7.3 captures all four in a small, dependency-light run manifest: it seeds every relevant RNG, records the pinned library versions and the resolved config, and hashes the input data so the vintage is provable. The manifest is written beside the model artifact, so the result carries its own reproduction recipe.

import json, hashlib, random, platform
from datetime import datetime, timezone
import numpy as np
import sklearn

def set_all_seeds(seed: int) -> None:
    random.seed(seed)
    np.random.seed(seed)          # add torch.manual_seed(seed) when training nets

def data_fingerprint(arr: np.ndarray) -> str:
    """Content hash of the exact bytes that went in: the data VINTAGE."""
    return hashlib.sha256(np.ascontiguousarray(arr).tobytes()).hexdigest()[:16]

SEED = 7
CONFIG = {"lag": 12, "model": "Ridge", "alpha": 1.0, "cv": "TimeSeriesSplit(5)"}

set_all_seeds(SEED)
X = np.random.default_rng(SEED).normal(size=(720, 12))   # stand-in for the panel

manifest = {
    "created_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    "seed": SEED,
    "config": CONFIG,
    "data_vintage": {
        "as_of": "2026-06-15",                 # transaction-time cutoff
        "fingerprint": data_fingerprint(X),    # proves WHICH rows were used
        "n_rows": int(X.shape[0]),
    },
    "env": {
        "python": platform.python_version(),
        "numpy": np.__version__,
        "sklearn": sklearn.__version__,
    },
}
print(json.dumps(manifest, indent=2))
# json.dump(manifest, open("run_manifest.json", "w"), indent=2)  # ship beside model

{
  "created_utc": "2026-06-15T12:00:00+00:00",
  "seed": 7,
  "config": {"lag": 12, "model": "Ridge", "alpha": 1.0, "cv": "TimeSeriesSplit(5)"},
  "data_vintage": {"as_of": "2026-06-15", "fingerprint": "9f1c4a7b2e8d0c63", "n_rows": 720},
  "env": {"python": "3.14.0", "numpy": "2.3.1", "sklearn": "1.7.0"}
}

In production this manifest is backed by tooling: a data-versioning system such as DVC tracks large data artifacts by content hash alongside Git, an experiment tracker stores the config and metrics, and a lockfile pins the dependency tree exactly. The principles and the full reproducibility toolchain, including compute and experiment management, are collected in Appendix F; this section establishes the habit, and the appendix supplies the machinery.

5. A Reference Pipeline Blueprint Beginner

This section, and this chapter, converge on one picture. Figure 2.7.1 is the reference temporal data pipeline: raw observations flow left to right through the seven stages this chapter built, and the reproducibility properties of subsections 1 through 4 wrap the whole chain. Read each stage as a station that the data must pass cleanly before it reaches the next, and note that each station is exactly one earlier section of Chapter 2. The figure is the chapter's synthesis: every box is a skill you now have, and the arrows are the order in which they must run so that no future ever leaks into the present.

Two reading rules make the blueprint operational. First, the order is not negotiable: validation precedes resampling because you cannot align a clock you have not checked, and windowing precedes the split because the split must respect window boundaries to avoid the leak of a window straddling the train/test wall. Second, the dashed envelope is what distinguishes a temporal pipeline from an ordinary one. An ordinary pipeline runs the inner chain; a temporal pipeline runs it under an as-of clock and records the vintage, so the artifact that drops out the bottom is not just a model but a model plus a provable account of exactly what it knew and when. Every later part of this book assumes the data arrived through a chain shaped like this one.