"Everyone wants to talk to the model. Nobody wants to talk to me. Yet I am the one who decided, long before any training began, which futures the model would be allowed to see. Treat me carelessly and the smartest architecture in the book will learn tomorrow's answer and call it skill."
A Data Pipeline With Trust Issues
Chapter Overview
There is a quiet, unglamorous layer beneath every forecaster, every recurrent network, and every reinforcement-learning agent in this book, and it decides whether any of them deserve to be believed. That layer is data engineering. Before a single model is fitted, someone has to read raw timestamps that arrived in three different time zones, decide what an hour actually means when observations land irregularly, fill the gaps that sensors and patients leave behind, turn a column of values into features that respect the arrow of time, cut the series into training windows, and split those windows so the test set lives strictly in the future of the training set. Do this work well and a mediocre model looks honest. Do it carelessly and a brilliant model reports a future it could never have known. This chapter is about doing it well.
The thread running through all seven sections is leakage: the silent contamination that occurs whenever information from the future, or from the test set, leaks backward into the inputs a model trains on. Leakage is not an exotic failure. It is the default outcome of ordinary, reasonable-looking code. A scaler fitted on the whole series before splitting leaks. A forward-fill that reaches across the train and test boundary leaks. A rolling feature whose window is centered rather than trailing leaks. A target computed from a horizon that overlaps the features leaks. Each leak inflates validation scores, survives code review, and collapses the moment the system meets live data. The cardinal rule of the chapter, stated once and enforced everywhere, is that a feature available at time $t$ may use only information that was actually known at time $t$. Every section returns to that rule and shows the specific way its own topic tends to break it.
The chapter is also where this book's three running datasets stop being an abstraction and become engineering. The finance returns-and-volatility series exposes the time-zone and event-ordering traps of market data, where a timestamp on the wrong exchange clock quietly reorders cause and effect. The irregularly sampled clinical series, where a missing vital is itself a signal that nobody thought the patient sick enough to measure, drives the missing-data and resampling sections. The industrial sensor-telemetry stream, evenly clocked but enormous, motivates the windowing and pipeline-reproducibility work. The same three datasets thread forward into the classical models of Part II, the deep architectures of Part III, and the deployment chapters at the end, so the engineering decisions made here are decisions the whole book inherits.
Why give a foundations chapter entirely to plumbing? Because the field's most embarrassing results, the backtests that promised yachts and delivered nothing, almost never failed at the modeling step. They failed here, in the data preparation that everyone trusted and no one audited. The last two years have only raised the stakes: feature stores, experiment trackers, and forecasting libraries now automate this layer at scale, which means a single leak written once can propagate into thousands of automated retraining runs before anyone notices. Mastering temporal data engineering is therefore not preparation for the interesting part of temporal AI. For most working practitioners, it is the part that decides whether the interesting part was real.
Prerequisites
This chapter assumes you have read Chapter 1: Introduction to Temporal Intelligence, in particular its distinction between a sequence and a table and its warning that look-ahead leakage is the discipline that most often separates real results from inflated ones. Chapter 2 turns that warning into operational rules. On the tooling side it assumes working fluency in Python and pandas: you should be comfortable with a DatetimeIndex, with groupby and resampling, and with NumPy array manipulation, since every section ships runnable code in that idiom. No prior forecasting, signal-processing, or production-engineering experience is required; each topic is built from first principles and then connected to a production library through the book's "Right Tool" callouts. Readers wanting a mathematics or PyTorch refresher will find the appendices listed in the Table of Contents (Appendix A for notation, Appendix B for probability and statistics, Appendix D for PyTorch).
If you keep one idea from this chapter, keep the cardinal rule: a feature available at time $t$ may use only information that was actually known at time $t$. Everything else in temporal data engineering, the resampling boundaries, the trailing windows, the chronological splits, the purge-and-embargo gaps, is bookkeeping in service of that single constraint. A pipeline that honors it can be slow, ugly, or under-tuned and still be trustworthy. A pipeline that violates it can be elegant, fast, and fast at lying to you. When a result looks too good, do not first suspect the model; first ask what your features knew, and when they knew it.
Chapter Roadmap
- 2.1 Time Stamps, Time Zones, and Event Ordering How a timestamp is just a count of seconds, why naive local times reorder events across time zones and daylight-saving boundaries, and how to establish a single, defensible event order.
- 2.2 Sampling, Resampling, and Aggregation Why every resampled bar is an opinion, how to choose downsampling aggregations and upsampling fills that do not invent the future, and where alignment quietly leaks.
- 2.3 Missing Data and Irregular Observations Treating absence as information rather than nuisance: missingness mechanisms, leakage-safe imputation paired with explicit missing-indicator masks, and the irregular-sampling case.
- 2.4 Feature Engineering for Temporal Data Building lag, rolling, expanding, and calendar features that respect the arrow of time, and the trailing-window discipline that keeps every feature available at the moment it claims to be.
- 2.5 Windowing, Segmentation, and Lag Construction Slicing a long series into supervised $(X, y)$ pairs with sliding and expanding windows, choosing input and horizon lengths, and avoiding overlap between a window's features and its target.
- 2.6 Evaluation Protocols, Backtesting, and Data Leakage Walk-forward and rolling-origin backtesting, purging and embargoing around the split, scale-free metrics such as MASE, and a field guide to the leaks that flatter every backtest.
- 2.7 Reproducibility and Temporal Data Pipelines Composing the previous six stages into one versioned, deterministic pipeline that records exactly what it knew and when, so a backtest can be rerun and trusted months later.
Once you have worked through the seven sections, the Hands-On Lab below assembles them into a single reusable artifact. Each lab step maps to one section, so the lab doubles as a review: by the time the harness prints its score, you have applied the timestamp discipline of Section 2.1, the imputation masks of Section 2.3, the trailing features of Section 2.4, and the walk-forward evaluation of Section 2.6 with your own hands.
Hands-On Lab: Build a Leakage-Free Forecasting Harness
Objective
Assemble the seven stages of this chapter into one reusable module, forecast_harness.py, that takes a raw, irregularly stamped series and returns an honest forecast score. The harness validates and orders timestamps, resamples to a regular grid, imputes gaps while recording an explicit missing-indicator mask, engineers trailing lag, rolling, and calendar features, windows the result into supervised $(X, y)$ pairs, runs a walk-forward split, and scores the mean absolute scaled error (MASE) of a simple model against a seasonal-naive baseline. The finished harness is the instrument you reach for whenever you are handed a new series and asked, before any deep model, "is there real signal here, and is my evaluation honest?" Every stage is wired so that no value available only in the future can reach the model, which is the entire point.
What You'll Practice
- Validating and canonically ordering timestamps from a single time zone, the discipline of Section 2.1 that prevents events from silently reordering.
- Resampling an irregular series onto a regular grid with leakage-safe aggregations and fills, following Section 2.2 and Section 2.3.
- Pairing imputation with an explicit missing-indicator mask so the model can read absence as information, the central idea of Section 2.3.
- Engineering trailing lag, rolling, and calendar features that are available at the timestamp they claim, the trailing-window rule of Section 2.4.
- Windowing into supervised pairs and running a walk-forward split scored with MASE against a seasonal-naive baseline, combining Section 2.5 and Section 2.6.
Setup
You need NumPy, pandas, and scikit-learn for the model and the splitter. A synthetic generator ships inside the harness so it runs out of the box, and the bibliography links real series (the Monash archive for evenly clocked load data, PhysioNet for irregular clinical vitals) for when you want to stress the harness against genuine messiness.
pip install numpy pandas scikit-learnTimeSeriesSplit walk-forward splitter.Steps
Step 1: Validate timestamps and fix a single event order
Parse the incoming timestamps into a timezone-aware DatetimeIndex in one canonical zone (UTC), drop or flag duplicates, and sort so the series has exactly one defensible order. This is the Section 2.1 rule made operational: an ambiguous local time or an out-of-order row is a reordering of cause and effect waiting to happen, so the harness refuses to proceed until time is monotone.
import numpy as np
import pandas as pd
def canonical_order(ts, values, tz="UTC"):
idx = pd.to_datetime(ts, utc=True).tz_convert(tz) # one canonical clock
s = pd.Series(np.asarray(values), index=idx)
s = s[~s.index.duplicated(keep="last")] # one row per instant
return s.sort_index() # monotone time order
Step 2: Resample to a regular grid
Project the cleaned series onto a fixed frequency. Downsampling aggregates with a backward-looking rule so each bar summarizes only the interval it closes; upsampling leaves new slots empty rather than guessing, deferring the decision to Step 3. This is the leakage-aware resampling of Section 2.2: the bar at time $t$ must never borrow a value from after $t$.
def to_grid(s, freq="1H"):
# label='right', closed='right' => each bar covers (t-freq, t], known by t
return s.resample(freq, label="right", closed="right").mean()
Step 3: Impute gaps and record a missing-indicator mask
Fill the empty grid slots with a strictly backward fill, never a forward-looking interpolation, and emit a parallel binary mask marking which values were observed and which were imputed. Following Section 2.3, the mask is itself a feature: in the clinical dataset, the fact that a vital was missing carries clinical meaning, so the model is given the chance to learn from absence rather than having it erased.
def impute_with_mask(g):
observed = g.notna().astype(int) # 1 where real, 0 where gap
filled = g.ffill() # carry last known value only
return filled, observed # value and its provenance
Step 4: Engineer trailing lag, rolling, and calendar features
Build the feature matrix from strictly trailing operations: lagged values, rolling statistics over a backward window, and calendar attributes derived from the index. Every column is shifted so it is available at the timestamp it labels. This is the trailing-window discipline of Section 2.4; a single centered or unshifted rolling mean here would leak the present into its own predictor.
def features(filled, mask, lags=(1, 2, 24)):
df = pd.DataFrame({"y": filled, "observed": mask})
for k in lags:
df[f"lag_{k}"] = filled.shift(k) # past values only
df["roll_mean_24"] = filled.shift(1).rolling(24).mean() # trailing, shifted
df["hour"] = filled.index.hour # calendar, known in advance
df["dow"] = filled.index.dayofweek
return df.dropna()
shift(1) before every rolling statistic is the line that separates an honest feature from a leaking one.Step 5: Window into supervised pairs and run a walk-forward split
Turn the feature frame into $(X, y)$ where $y$ is the next-step target, then split with a walk-forward scheme so every validation fold lies strictly in the future of its training fold. This fuses the windowing of Section 2.5 with the backtesting of Section 2.6; scikit-learn's TimeSeriesSplit enforces the ordering so a random shuffle can never sneak in.
from sklearn.linear_model import Ridge
from sklearn.model_selection import TimeSeriesSplit
def windowed_xy(df):
y = df["y"].shift(-1).dropna() # one-step-ahead target
X = df.loc[y.index].drop(columns="y")
return X, y
def walk_forward(X, y, n_splits=5):
splitter = TimeSeriesSplit(n_splits=n_splits) # past trains, future tests
return [(Ridge().fit(X.iloc[tr], y.iloc[tr]), te) for tr, te in splitter.split(X)]
TimeSeriesSplit hands back expanding-window folds in chronological order, so no fold ever trains on data later than the one it tests.Step 6: Score MASE against a seasonal-naive baseline
Compute the mean absolute scaled error: the model's mean absolute error divided by the in-sample mean absolute error of a seasonal-naive forecast (predict the value one season ago). A MASE below one means the model beats the naive baseline; a MASE at or above one means it does not, and no amount of architecture will save a harness whose honest baseline is already this hard. This is the scale-free verdict of Section 2.6.
def mase(y_true, y_pred, y_train, season=24):
scale = np.mean(np.abs(np.diff(y_train.values, season))) # seasonal-naive error
return float(np.mean(np.abs(y_true.values - y_pred)) / scale)
Expected Output
Running python forecast_harness.py prints one MASE per walk-forward fold and their mean. On the evenly clocked synthetic load series, the ridge model reports a mean MASE comfortably below one: the trailing features carry real signal and the harness is honest about it. On a near-random-walk variant, the MASE hovers around one, the harness correctly refusing to manufacture a victory the data does not support. The instructive experiment is deliberately breaking the harness: remove the shift(1) in Step 4, or replace TimeSeriesSplit with a random split, and watch the MASE plunge to an impossible-looking value. That collapse is leakage, visible and reproducible, and recognizing its signature is the skill this entire chapter exists to teach.
The harness is written out longhand so each leakage guard is visible. In production you would not assemble it by hand. sktime exposes the whole pipeline as a composable ForecastingPipeline with built-in ExpandingWindowSplitter and a one-call MeanAbsoluteScaledError metric that refuse to leak across the split. Darts wraps loading, lag and calendar feature creation, backtesting, and seasonal-naive baselines behind a single historical_forecasts call. Nixtla's statsforecast supplies the seasonal-naive baseline and chronological cross-validation in a few lines. The roughly seventy lines here collapse to under ten library calls, with the libraries handling the window bookkeeping, the split ordering, and the metric scaling. Write the harness once so you can read the leak; use the library forever so you never re-introduce it.
Stretch Goals
- Add purging and embargoing around each walk-forward boundary, dropping training samples whose feature windows overlap the test fold, and confirm the MASE rises slightly, the price of removing the subtle overlap leak that Section 2.6 dissects.
- Wrap the harness in a deterministic, versioned pipeline that hashes its inputs and records the configuration, realizing the reproducibility discipline of Section 2.7, then rerun it and confirm the score is bit-for-bit identical.
- Swap the synthetic generator for a real series from the Monash archive (load data) and from PhysioNet (clinical vitals), both linked in the bibliography, and confirm the missing-indicator mask of Step 3 carries measurable predictive weight on the clinical series.
What's Next?
This chapter built the leakage-free, reproducible inputs that every later model in the book consumes. With honest data in hand, Part II opens the classical forecasting toolkit, and it begins with the statistics that give those models their guarantees. Chapter 3: Statistical Foundations formalizes the concepts this chapter used operationally: stationarity and why a series must be tamed before it can be modeled, autocorrelation and partial autocorrelation as the diagnostic language of temporal dependence, white noise and the random walk as the baselines everything is measured against, and the estimation theory behind the MASE verdict the lab just computed. The three running datasets cleaned here become the worked examples there, so the resampling and feature choices made in this chapter are the choices Chapter 3 inherits and tests.
Bibliography & Further Reading
Key Books
Hyndman, R. J., Athanasopoulos, G. "Forecasting: Principles and Practice," 3rd edition. otexts.com/fpp3
The free modern standard; its evaluation chapter formalizes the seasonal-naive baseline and the MASE metric the lab scores against, and its data-wrangling chapters mirror Sections 2.1 through 2.5.
López de Prado, M. "Advances in Financial Machine Learning." Wiley, 2018. wiley.com
The source of the purging and embargo techniques that remove overlap leakage around a backtest split, the exact discipline Section 2.6 and the lab's first stretch goal implement.
Little, R. J. A., Rubin, D. B. "Statistical Analysis with Missing Data," 3rd edition. Wiley, 2019. wiley.com
The canonical treatment of missingness mechanisms (MCAR, MAR, MNAR) that underpins Section 2.3's argument for pairing imputation with an explicit missing-indicator mask.
Articles & Tutorials
Kaufman, S., Rosset, S., Perlich, C., Stitelman, O. "Leakage in Data Mining: Formulation, Detection, and Avoidance." ACM TKDD 6(4), 2012. dl.acm.org
The paper that named and formalized data leakage; the conceptual backbone of this chapter's cardinal rule and Section 2.6's field guide to the leaks that flatter every backtest.
Bergmeir, C., Benítez, J. M. "On the Use of Cross-Validation for Time Series Predictor Evaluation." Information Sciences 191, 2012. sciencedirect.com
The reference case for why ordinary k-fold cross-validation leaks on temporal data and why the walk-forward and rolling-origin schemes of Section 2.6 are required instead.
pandas time-series and date functionality documentation. pandas.pydata.org
The operational reference for the DatetimeIndex, time-zone handling, and resampling semantics that Sections 2.1 and 2.2 and the lab's first two steps depend on.
Tools & Libraries
scikit-learn TimeSeriesSplit cross-validation documentation. scikit-learn.org
The walk-forward splitter used in the lab's Step 5; it yields expanding-window folds in chronological order so a random shuffle can never contaminate evaluation.
sktime: a unified framework for machine learning with time series. sktime.net
The scikit-learn-style forecasting framework whose ForecastingPipeline, expanding-window splitters, and MASE metric collapse this chapter's hand-wired harness to a few composable lines.
Darts: a Python library for user-friendly forecasting and anomaly detection. unit8co.github.io/darts
Wraps lag and calendar feature creation, seasonal-naive baselines, and leakage-safe historical_forecasts backtesting behind one API, the production form of the lab.
tsfresh: automatic time-series feature extraction. tsfresh.readthedocs.io
Computes hundreds of trailing temporal features with built-in relevance filtering, the automated counterpart to the hand-built lag and rolling features of Section 2.4.
Feast: an open-source feature store for machine learning. feast.dev
Provides point-in-time correct feature retrieval, the production mechanism that enforces Section 2.7's rule that a feature may only use what was known at its timestamp.
DVC: data version control for machine-learning projects. dvc.org
Versions datasets and pipeline stages so a temporal backtest can be reproduced bit-for-bit months later, the tooling behind the reproducibility discipline of Section 2.7.
Datasets & Benchmarks
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., Montero-Manso, P. "Monash Time Series Forecasting Archive." NeurIPS Datasets and Benchmarks, 2021. forecastingdata.org
A large curated collection of forecasting datasets across domains; the source of the evenly clocked load series the lab uses to stress the harness against real data.
Makridakis, S., Spiliotis, E., Assimakopoulos, V. "The M5 Accuracy Competition: Results, Findings, and Conclusions." International Journal of Forecasting 38(4), 2022. sciencedirect.com
The largest public forecasting competition; its hierarchical retail data and evaluation rules motivate the scale-free MASE-style metrics and honest backtesting this chapter adopts.
Goldberger, A. L., et al. "PhysioBank, PhysioToolkit, and PhysioNet." Circulation 101(23), 2000. physionet.org
The standard repository of physiological signals; the source of the irregularly sampled clinical vitals that drive Section 2.3 and the lab's missing-indicator mask.