Reproducibility, Compute, and Experiment Management

"They asked me to reproduce last quarter's number. I had the code. I had the model. I did not have the data as it stood that Tuesday, and so I produced a different number, defended it for a week, and learned the most expensive lesson in temporal machine learning: the result is not the code, it is the code plus the world frozen at one instant."

A Forecaster Who Forgot to Save the Vintage

1. Determinism and Seeds

A reproducible run starts with control over randomness. A temporal model touches several independent random-number generators: Python's own, NumPy's, the deep-learning framework's CPU and GPU generators, and the data loader's shuffling. Seeding one of them is not enough; an unseeded generator anywhere in the path makes the run irreproducible. The first habit is therefore a single function that seeds every generator the process can reach, called once before any data is drawn or any weight is initialized.

import os, random
import numpy as np
import torch

def seed_everything(seed: int = 0) -> None:
    os.environ["PYTHONHASHSEED"] = str(seed)  # stabilize hash-based ordering
    random.seed(seed)                          # Python's stdlib generator
    np.random.seed(seed)                       # NumPy's legacy global generator
    torch.manual_seed(seed)                    # PyTorch CPU generator
    torch.cuda.manual_seed_all(seed)           # all visible GPU generators

seed_everything(0)

Seeding fixes the starting point, but it does not by itself make the arithmetic deterministic. Many GPU kernels accumulate partial sums in a nondeterministic order, so two runs with the same seed can still diverge in the low-order bits, and that divergence compounds across the long unrolled horizons typical of recurrent and attention models. To force bit-for-bit repeatability you ask the framework for deterministic algorithms and disable the autotuner that picks kernels by benchmarking them at runtime.

torch.use_deterministic_algorithms(True)   # error out if a kernel has no deterministic path
torch.backends.cudnn.benchmark = False      # disable the autotuner that varies kernel choice
torch.backends.cudnn.deterministic = True   # pin cuDNN to deterministic convolutions
# Some CUDA reductions also require this environment variable, set BEFORE the process starts:
#   CUBLAS_WORKSPACE_CONFIG=:4096:8

Determinism has a price, and you should pay it knowingly. Deterministic kernels are often slower (a convolution-heavy temporal convolutional network of Chapter 11 can lose a noticeable fraction of throughput), and some operations have no deterministic GPU implementation at all, in which case the call above raises rather than silently varying. The pragmatic policy is to run full determinism for the experiments that back a reported number and for debugging, and to relax it for exploratory sweeps where statistical reproducibility across several seeds matters more than bit-exactness of any single run. Report which regime produced each result.

The remaining source of irreproducibility is the data loader. Parallel workers each need their own seeded generator, or their shuffling and any stochastic augmentation become a function of thread-scheduling timing. Pass a generator to the loader and a deterministic per-worker seed.

def seed_worker(worker_id: int) -> None:
    worker_seed = torch.initial_seed() % 2**32  # derived deterministically from the base seed
    np.random.seed(worker_seed)
    random.seed(worker_seed)

g = torch.Generator(); g.manual_seed(0)
loader = torch.utils.data.DataLoader(
    dataset, batch_size=64, shuffle=True,
    num_workers=4, worker_init_fn=seed_worker, generator=g,
)

2. Environment and Data Versioning

Identical code on two machines can still produce different numbers if the libraries underneath differ, because a minor version of a linear-algebra backend or a deep-learning framework can change kernel selection, default dtypes, or summation order. The environment is therefore part of the result and must be pinned. The minimum is a fully resolved dependency lockfile, not a loose list of top-level packages: a lockfile records every transitive dependency at an exact version and hash, so the environment rebuilds identically.

# Loose, NOT reproducible: top-level packages, versions float
#   torch
#   statsmodels
# Pinned: exact versions for every transitive dependency
pip freeze > requirements.lock     # or: uv pip compile, poetry lock, conda env export
pip install -r requirements.lock   # rebuilds the identical environment

A lockfile pins Python packages but not the system libraries beneath them, the CUDA toolkit, the BLAS implementation, the operating-system C library. For results that must survive a machine change or a hardware refresh, capture the whole stack in a container image and record its digest. The image digest is then a single hash that stands in for the entire software environment.

FROM python:3.11-slim
COPY requirements.lock /tmp/requirements.lock
RUN pip install --no-cache-dir -r /tmp/requirements.lock
# Pin the base CUDA image by digest when GPUs are involved, for example
#   FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04@sha256:...

Code and environment are only two thirds of the input. The data is the third, and in temporal work it is the subtle one. Data versioning tools track large files by content hash and store them outside the code repository while keeping a small pointer file under version control, so a commit pins an exact dataset the way it pins exact source. The pattern is the same whichever tool you use: the repository holds a tiny metadata file, the bytes live in object storage, and a checkout restores the exact data that commit was built against.

dvc add data/vitals_2026Q1.parquet   # hashes the file, writes data/vitals_2026Q1.parquet.dvc
git add data/vitals_2026Q1.parquet.dvc data/.gitignore
git commit -m "Pin Q1 vitals vintage"
# A later checkout + `dvc pull` restores the byte-identical dataset for this commit.

# A point-in-time feature query: use only values KNOWN as of the decision time.
def features_as_of(panel, entity_id, as_of):
    rows = panel[(panel.entity_id == entity_id) & (panel.known_at <= as_of)]
    return rows.sort_values("known_at").groupby("field").last()  # latest KNOWN value, not latest value

3. Experiment Management

With inputs pinned, the next discipline is recording what each run did. An experiment tracker logs, for every run, its configuration, its metrics over time, and the artifacts it produced (the fitted model, plots, predictions), so that months later you can answer "which config produced this number" without rerunning anything. Trackers such as MLflow and Weights & Biases expose nearly the same minimal interface: open a run, log parameters and metrics, log artifacts, close the run.

import mlflow

with mlflow.start_run(run_name="tcn-vitals-h24"):
    mlflow.log_params(cfg)                       # the full resolved configuration
    mlflow.set_tag("git_commit", git_sha)        # bind the run to exact code
    mlflow.set_tag("data_vintage", "2026-04-01") # bind the run to the as-of instant
    for epoch, val_mae in train(cfg):
        mlflow.log_metric("val_mae", val_mae, step=epoch)
    mlflow.log_artifact("model.pt")              # the produced artifact, addressable later

Experiment tracking is what turns a pile of runs into a reproducible record. A tracker logs, for every run, the hyperparameters, the metrics as they evolve, the exact code and version, the seed, and the artifacts produced, so that a result is never an orphan number: it is bound to the inputs that made it. The same record lets you compare runs against each other, surface the configuration that won, and share a live dashboard with collaborators instead of pasting screenshots into a chat. The supported course uses two complementary trackers, and the choice between them is the local-versus-hosted tradeoff in miniature. TensorBoard is local, lightweight, and dependency-free: a SummaryWriter writes event files to a directory you point a viewer at, with no account and no network.

from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter(log_dir="runs/tcn-vitals-h24")
hparams = {"model": "tcn", "horizon": 24, "lr": 1e-3, "seed": 0}
for epoch, val_mae in train(cfg):
    writer.add_scalar("val/mae", val_mae, global_step=epoch)   # the metric curve
writer.add_hparams(hparams, {"hparam/final_val_mae": val_mae})  # config plus its outcome
writer.close()
# View locally with:  tensorboard --logdir runs

Weights & Biases is hosted: each run streams its config, metrics, and artifacts to a server, where the web interface compares runs side by side, groups them by tag, and exposes a shared dashboard the whole team reads. The minimal interface mirrors the tracker pattern of the previous section: initialize a run, record the config, log metrics in the loop, and save artifacts.

import wandb

wandb.init(project="taibook-vitals", name="tcn-vitals-h24",
           config={"model": "tcn", "horizon": 24, "lr": 1e-3, "seed": 0})
wandb.config.update({"git_commit": git_sha, "data_vintage": "2026-04-01"})  # bind code and vintage
for epoch, val_mae in train(cfg):
    wandb.log({"val/mae": val_mae}, step=epoch)   # metrics stream to the hosted dashboard
wandb.save("model.pt")                            # the artifact, versioned on the server
wandb.finish()

The two tools answer different needs. TensorBoard is the right default for a single machine or a quick local experiment: it is fast, private, and adds no dependency beyond the framework you already use. Weights & Biases earns its hosted complexity when runs span machines, when a team needs one dashboard, or when you want server-side run comparison and artifact versioning without building it yourself. They are not exclusive: a common pattern logs to TensorBoard locally and mirrors the same scalars to a hosted tracker for the runs that back a reported number. Whichever you use, the tracker is the mechanism that captures the seed and the resolved config (the levers of the determinism and config discipline above) as part of the run record, so the tracked run is itself a piece of the reproducibility receipt rather than a separate note. One line on sweeps: a hosted tracker can also drive hyperparameter sweeps, launching a grid or Bayesian search across the config space and logging every trial to the same comparable dashboard.

The configuration deserves its own discipline. Burying hyperparameters as literals in the training script makes runs impossible to compare and impossible to sweep cleanly. A config manager keeps the entire experiment definition in a structured, composable file, lets you override any field from the command line, and (the part that matters for reproducibility) saves the fully resolved config of each run. A Hydra-style layout composes a run from reusable groups, so a sweep varies one group while the rest stay fixed and recorded.

from dataclasses import dataclass, asdict
import json

@dataclass
class Config:
    model: str = "tcn"
    horizon: int = 24
    lr: float = 1e-3
    seed: int = 0
    data_vintage: str = "2026-04-01"   # the as-of instant travels WITH the config

cfg = Config(model="tcn", horizon=24)
json.dump(asdict(cfg), open("run_config.json", "w"), indent=2)  # the resolved config IS an artifact

4. Compute Budgeting

Reproducibility tells you whether a result is real; budgeting tells you what it cost and what the next one will cost. The two families of temporal models have very different cost shapes. Classical models (the ARIMA, ETS, and state-space fits of Part II) are CPU-bound and embarrassingly parallel across series, so the right tool is process-level parallelism over a backtest grid, not a GPU. Deep temporal models (Part III) are GPU-bound, and their cost is dominated by the sequence length, the batch size, and the number of epochs.

from joblib import Parallel, delayed

# Classical: fit one model per series across all CPU cores. No GPU needed.
results = Parallel(n_jobs=-1)(
    delayed(fit_and_backtest)(series) for series in panel  # one independent job per series
)

For deep models, estimate before you launch. A rough but useful budget multiplies the per-step cost by the number of steps: the GPU-hours of a run are approximately the number of training tokens or windows seen, divided by the throughput of the hardware. Estimating this on a short pilot (a few hundred steps timed) lets you price a full sweep before committing to it, and lets you choose between local and cloud rationally. The deciding question is utilization: a single long run that saturates one local GPU is cheapest locally; a wide sweep that would queue for days locally is cheaper on rented elastic capacity that runs the configurations in parallel and is released the moment they finish.

def gpu_hours(n_windows, epochs, throughput_windows_per_sec):
    total_windows = n_windows * epochs
    return total_windows / throughput_windows_per_sec / 3600  # measure throughput on a short pilot

print(gpu_hours(n_windows=500_000, epochs=30, throughput_windows_per_sec=1_800))

The last budgeting axis is batch versus online. A batch (offline) forecasting job amortizes setup over a large run and is priced by total compute. An online or streaming system of Chapter 20 is priced by latency and by steady-state cost per prediction, because it runs continuously: a model that is cheap to train once but expensive per inference can dominate the lifetime bill of a deployed temporal service. Budget the regime you will actually operate in, and measure cost per prediction, not just cost per training run, for anything that will serve online.

5. A Reproducibility Checklist for a Temporal ML Result

The four sections above reduce to one operational artifact: a checklist you attach to every reported temporal number. A result is reproducible when a colleague, starting from your repository and storage, can regenerate it to the digit. The six items below are the inputs that, pinned together, make that possible; any one missing is a hole through which a rerun drifts.

A compact way to enforce the checklist is to have every run emit a single manifest that records all six items, written next to its artifacts. The manifest is the receipt: present it with the number, and anyone can reconstruct the run.

import json, subprocess

manifest = {
    "data_vintage": cfg.data_vintage,                                    # item 1
    "seed": cfg.seed,                                                    # item 2
    "config": asdict(cfg),                                               # item 3
    "git_commit": subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip(),  # item 4
    "image_digest": os.environ.get("IMAGE_DIGEST", "unset"),            # item 5
    "eval_protocol": {"scheme": "rolling-origin", "folds": 5, "metric": "MASE"},  # item 6
}
json.dump(manifest, open("run_manifest.json", "w"), indent=2)