Datasets, Benchmarks, and Evaluation Protocols

"Give me a benchmark and a metric, and I will overfit them both by Thursday. Give me a leakage-free split, a proper scoring rule, and a held-out future I have never seen, and I might tell you something true."

A Forecaster Who Has Read the Leaderboard Too Many Times

1. A Per-Task Catalog of Temporal Datasets and Benchmarks

Temporal AI spans many tasks, and each has accreted its own standard datasets over the years. The catalog below is organized by task so you can jump straight to the benchmark that matches your problem. For each entry the table gives the dataset, what it contains and roughly how large it is, the task it is built for, and a working link. Treat every leaderboard with the skepticism the evaluation section recommends: a dataset is a fixed sample of the world, not the world.

1.1 Forecasting

Forecasting benchmarks split into two cultures. The competition lineage (M4, M5) tests breadth across thousands of heterogeneous business series with held-out futures and economic loss functions, while the deep-learning long-horizon lineage (ETT, Electricity, Traffic, Weather) tests a handful of long, high-frequency multivariate series at horizons of hundreds of steps. The Monash archive and the GluonTS and Nixtla dataset loaders unify access to most of them. These series thread through the deep forecasting architectures of Chapter 14 and the foundation models of Chapter 15.

The four long-horizon sets (ETT, Electricity, Traffic, Weather) became the de facto Transformer-forecasting suite and appear together in nearly every long-sequence paper, so reusing their standard input and prediction lengths lets your numbers sit next to published ones. The code below pulls a Monash dataset through the GluonTS loader in a few lines.

from gluonts.dataset.repository import get_dataset, dataset_names

print(dataset_names[:5])                    # discover available datasets
ds = get_dataset("electricity")             # downloads and caches on first call
train, test = ds.train, ds.test             # iterables of dict entries with a target field
freq, horizon = ds.metadata.freq, ds.metadata.prediction_length
print(freq, horizon)                        # the dataset-defined frequency and forecast horizon

1.2 Classification

Time series classification is dominated by two sibling archives maintained at the University of East Anglia. The UCR archive holds univariate series and the UEA archive holds multivariate series, both with fixed train and test splits, a published accuracy table for dozens of classifiers, and a single download portal. They are the standard proving ground for the representation-learning methods of Chapter 16.

Both archives ship through the aeon and sktime loaders, so you rarely parse the raw files yourself. Because the splits are fixed, the honest comparison is against the published per-dataset accuracy table rather than a split you choose; tuning the split is itself a form of leakage.

1.3 Anomaly and Change-Point Detection

Anomaly detection has the most contested benchmark landscape in temporal AI, and this catalog flags that openly. The classic datasets below are widely used, but a pointed critique argues several are too easy or mislabeled, and that the point-adjustment scoring protocol inflates results dramatically. These benchmarks underpin the detection methods of Chapter 8, where the point-adjustment pitfall is dissected in full.

Read the Wu and Keogh paper before you trust any anomaly leaderboard. Its core findings, that triviality, mislabeling, and run-to-failure bias plague these datasets and that point adjustment can make a random scorer look near perfect, are the reason Chapter 8 insists on event-aware metrics and the cleaner UCR Anomaly Archive.

1.4 Reinforcement Learning

Sequential decision making (Part VI, Chapters 22 through 29) evaluates on interactive environments rather than fixed files, plus offline datasets of logged trajectories for the batch setting. Gymnasium is the standard environment API, and D4RL is the standard offline-RL dataset suite. The deep and offline methods of Chapter 25 and the sequence-as-decision view of Chapter 28 both report on these.

The D4RL quality tiers (random, medium, medium-replay, expert) matter because offline-RL skill is largely about handling distribution shift between the logged behavior policy and the learned policy, and the tier fixes how far that shift goes.

1.5 Event Sequences and Temporal Point Processes

When events arrive at irregular, continuous timestamps rather than on a fixed grid, the temporal point process is the right model, and its benchmarks are logs of timestamped, marked events. These datasets drive the event and point-process modeling of Chapter 18.

The Retweet and StackOverflow logs are the standard pair in neural point-process papers, and the curated splits shipped with the Neural Hawkes Process code are what most later work reuses, so starting from that repository keeps you comparable.

1.6 Spatio-Temporal

Spatio-temporal forecasting (the traffic and sensor-network problems of Chapter 32) needs data with both a graph of sensor locations and a time series at each node. Two road-traffic datasets are the field standard.

Both ship with the adjacency matrix that defines sensor proximity, which is what makes them graph problems rather than plain multivariate ones; the DCRNN repository provides the canonical preprocessing.

1.7 Clinical

Clinical time series are irregularly sampled, multivariate, and rich with missingness, the realistic stress test for the filtering of Chapter 7 and the continuous-time models of Chapter 13. The MIMIC databases are the field standard and require a credentialed PhysioNet account plus a short training course, which is itself a lesson in responsible data use.

Access is gated for good reason: these are real patient records, and the credentialing step is the responsible-data practice that Chapter 33 treats as a first-class engineering concern, not a formality.

1.8 Human Activity Recognition

Wearable and smartphone sensor streams labeled with the activity being performed are the standard benchmark for sensor classification, the sensor-domain thread that runs from Chapter 8 through deployment in Chapter 34.

A subtle leakage trap haunts both: if windows from the same subject and session land in both train and test, accuracy is inflated, so the honest protocol is leave-one-subject-out, a point the evaluation section returns to.

2. The Three Running Datasets of This Book and Their Public Analogs

The book rotates three running domains so that the same data reappears as the methods deepen, as laid out in the front matter and used from Chapter 2 onward. Each running dataset is small and self-contained enough to ship with the exercises, but each has a public, full-scale analog you can graduate to. The table maps the book's domain to the public dataset that lets you reproduce and extend every worked example at realistic scale.

3. Evaluation Protocols Done Right

A dataset is necessary but not sufficient. The protocol wrapped around it decides whether a reported number reflects skill or leakage. This section covers the four pillars: time-aware splitting and backtesting, the right metric per task, the anomaly-scoring pitfall, and reinforcement-learning evaluation across seeds, and it ends with a reproducibility checklist.

3.1 Time-Aware Splits and Walk-Forward Backtesting

The first rule, established in Chapter 2.6, is that you never shuffle a temporal split. Random k-fold cross-validation lets the model train on points from after the test point, which is leakage in its purest form: in deployment the future is not available, so an evaluation that grants it overstates skill, sometimes wildly. The correct procedure is a chronological split (train on the past, test on a strictly later block) and, for a robust estimate, walk-forward backtesting that slides the train and test windows forward through time and averages the error over many origins. Two windowing schemes are common: an expanding window that grows the training set at each step, and a rolling window that keeps it fixed; the rolling window better matches a model retrained on recent data, the expanding window better matches one that keeps all history.

import numpy as np

def walk_forward_splits(n, initial_train, horizon, step, expanding=True):
    """Yield (train_idx, test_idx) tuples that never let the future leak backward."""
    start = initial_train
    while start + horizon <= n:
        train_start = 0 if expanding else max(0, start - initial_train)  # expanding vs rolling
        train_idx = np.arange(train_start, start)                        # strictly past
        test_idx = np.arange(start, start + horizon)                     # strictly future
        yield train_idx, test_idx
        start += step                                                    # slide the origin forward

3.2 Forecasting Metrics

The right forecasting metric depends on whether you predict a point or a distribution, and on whether your series are comparable in scale. The four below, developed in Chapter 19, cover the cases you will meet. MASE and sMAPE judge point forecasts; CRPS and the pinball (quantile) loss judge probabilistic ones.

The deepest pitfall is using a plain MAPE or RMSE to rank methods across series of wildly different magnitudes: a single large-scale series then dominates the average and the ranking measures scale, not skill. MASE exists precisely to fix this by dividing out the naive forecast error per series. For probabilistic forecasts, prefer a proper scoring rule (CRPS or averaged pinball) because, unlike interval coverage alone, it cannot be gamed by a forecaster who reports dishonest uncertainty.

import numpy as np

def mase(y_true, y_pred, y_train, m=1):
    naive = np.mean(np.abs(y_train[m:] - y_train[:-m]))   # in-sample seasonal-naive error
    return np.mean(np.abs(y_true - y_pred)) / naive       # scaled, comparable across series

def pinball_loss(y_true, q_pred, q):
    diff = y_true - q_pred
    return np.mean(np.maximum(q * diff, (q - 1) * diff))  # asymmetric quantile penalty

3.3 Classification and Anomaly Metrics, and the Point-Adjustment Pitfall

For balanced classification, accuracy on the fixed UCR/UEA test split is the comparable number. For imbalanced detection, accuracy is meaningless (a detector that never fires scores high when anomalies are rare), so precision, recall, the F1 score, and the area under the precision-recall curve are the honest summaries. The trap unique to temporal anomaly detection, examined at length in Chapter 8, is point adjustment: a scoring convention that, if any single point inside a true anomalous segment is flagged, marks the entire segment as correctly detected. This convention is so generous that a random scorer or a trivial threshold can post a near-perfect adjusted F1, which is exactly the inflation the Wu and Keogh critique (arXiv:2009.13807) documents.

3.4 Reinforcement-Learning Evaluation Across Seeds

Reinforcement-learning results are notoriously high-variance: the same algorithm with the same hyperparameters can win or lose depending on the random seed, because exploration, environment stochasticity, and network initialization all compound. A single training run is therefore not evidence, a point made forcefully in Chapter 25.5. The honest protocol runs each method over many seeds (commonly five to ten, more for noisy environments), reports the distribution rather than the best run, and uses summary statistics that are robust to the long tails RL returns exhibit. The interquartile mean with stratified bootstrap confidence intervals, popularized by the rliable methodology, has become the community standard because the plain mean is dominated by a few lucky or unlucky seeds and the median throws away too much information.

3.5 A Checklist for Leakage-Free, Reproducible Evaluation

The protocols above reduce to a checklist you can run before reporting any temporal result. It folds in the reproducibility discipline detailed in Appendix F.

• Chronological split. Train strictly precedes test; no shuffled k-fold on temporal data. Use walk-forward backtesting over multiple origins for a stable estimate.
• Fit transforms inside the fold. Compute scaling, imputation, and feature statistics on the training window only, never on the full series.
• No future features. Every input available to the model at prediction time must have been observable then; check lagged and rolling features for accidental look-ahead.
• Right metric for the task. MASE or sMAPE for point forecasts, CRPS or pinball for probabilistic, precision-recall and area-under-PR for imbalanced detection, return distributions for RL.
• State the anomaly protocol. Always report whether point adjustment was applied, and prefer event-aware or range-based scores.
• Honest baselines. Include the seasonal-naive forecast (the MASE denominator), the persistence model, and a published strong baseline; a method that does not beat naive has not earned its complexity.
• Multiple seeds. For any stochastic method, especially RL, report a distribution over seeds with confidence intervals, not a single run.
• Fixed, versioned splits and seeds. Record the dataset version, the split indices, every random seed, and the library versions, so the number can be regenerated exactly (Appendix F).
• Separate selection from evaluation. Hyperparameter search, early stopping, and model choice see only validation data drawn from before the test block.
• Report the unglamorous failures too privately. Keep the full experiment registry; only significant, supported results go in the headline, but the diagnostics that did not work stay logged for reproducibility.