"They keep asking me to predict the present, and all I ever say is what happened one step ago. Somehow that is usually enough. The past, it turns out, is an excellent and underpaid consultant."
A Lag Feature Living Comfortably in the Past
Feature engineering for temporal data is the craft of turning a one-dimensional stream of observations into a wide supervised table whose every column is computable strictly from the past. Five families of features do almost all of the work: lags that expose the autoregressive structure, rolling and expanding statistics that summarize recent behavior, calendar and cyclical encodings that hand the model the clock and the calendar, domain transforms that stabilize variance and remove trend, and automated feature banks that extract hundreds of descriptors at once. Every one of them lives or dies by a single rule inherited from the previous section: a feature for time $t$ may only see data observed up to $t$. Get that rule right and a humble gradient-boosted tree on engineered features will embarrass many a deep network; get it wrong and you have built the leakage discipline of Section 2.6 into every column.
In the previous section we handled missing and irregular observations, pairing imputation with explicit masks and seeing how a careless fill leaks the future into preprocessing. Now we put that discipline to constructive use. The goal of this section is to convert a raw series into features a standard supervised learner can consume, while never once violating the temporal boundary. We will use the book's running finance dataset, a daily price series, as the worked example throughout, because returns and rolling volatility make the leakage traps concrete and because these exact features recur in the ARIMA and GARCH models of Chapter 5 and Chapter 6 and the deep forecasters of Chapter 14. Each numbered subsection below takes one feature family, states the math where it clarifies, and shows runnable pandas that respects the past-only rule by construction.
A note on framing before we begin. Hand-crafted temporal features are not the only path; Part IV of this book is devoted to learned representations that discover their own features from raw sequences. But engineered features remain the strongest baseline in a great many practical settings, they are interpretable, and they are the clearest place to internalize the leakage discipline that every later method also depends on. Learn to build them by hand here, and the automated and learned alternatives later will read as conveniences rather than mysteries.
1. Lag Features and the Autoregressive View Beginner
The single most important temporal feature is the lag. A lag of order $k$ is simply the value of the series $k$ steps in the past, written $x_{t-k}$. By placing past values alongside the present target in the same row, lags convert a sequence into the familiar supervised shape of a feature matrix and a target vector, which is why they are the bridge between time-series data and ordinary tabular machine learning. The autoregressive view makes this precise: to predict $x_t$ we assemble an embedding vector of recent observations, an honest feature that, as Figure 2.4.1 pictures, only ever trails the present and never reaches ahead of it,
and learn a function $\hat{x}_t = f\!\big(\mathbf{x}_t^{\text{lag}}\big)$. When $f$ is linear this is exactly the autoregressive model AR($p$) of Chapter 5; when $f$ is a tree ensemble or a neural network it is the nonlinear generalization. The vector $\mathbf{x}_t^{\text{lag}}$ is also called a delay embedding, and its dimension $p$ is the order; choosing $p$ is the choice of how far back the model is allowed to look. The notation for series, lags, and the backshift operator is fixed in the unified table of Appendix A. Code 2.4.1 builds a lag table on the finance series.
import numpy as np
import pandas as pd
# Running finance dataset: a daily price series on business days.
rng = np.random.default_rng(7)
idx = pd.bdate_range("2021-01-01", periods=8, name="date")
price = pd.Series(100 + np.cumsum(rng.normal(0, 1.0, len(idx))),
index=idx, name="price")
# Lags turn the series into a supervised table. shift(k) moves values
# DOWN by k rows, so row t holds x_{t-k}: strictly past information.
df = pd.DataFrame({"price": price})
for k in (1, 2, 3):
df[f"lag_{k}"] = df["price"].shift(k) # x_{t-k}
# The supervised target is the next-step price; align it to the same row.
df["target_next"] = df["price"].shift(-1) # x_{t+1}, the label only
print(df.round(2))
Series.shift(k). A positive shift pulls past values forward into the current row so column lag_k holds $x_{t-k}$; the target uses shift(-1) and is never used as an input feature. price lag_1 lag_2 lag_3 target_next
date
2021-01-01 99.55 NaN NaN NaN 99.86
2021-01-04 99.86 99.55 NaN NaN 100.92
2021-01-05 100.92 99.86 99.55 NaN 100.49
2021-01-06 100.49 100.92 99.86 99.55 101.31
2021-01-07 101.31 100.49 100.92 99.86 100.77
2021-01-08 100.77 101.31 100.49 100.92 101.45
2021-01-11 101.45 100.77 101.31 100.49 102.10
2021-01-12 102.10 101.45 100.77 101.31 NaN
NaN because the requested past does not yet exist, and the final target is NaN because the future is unknown. Dropping rows with missing lags leaves a clean supervised table where every feature is observed strictly before its target.The entire reason lags matter is that they reduce forecasting to regression. Once row $t$ contains $\big(x_{t-1},\ldots,x_{t-p}\big)$ as inputs and $x_t$ as the label, any supervised learner, linear model, random forest, gradient boosting, or multilayer perceptron, can be trained without knowing it is touching time at all. The temporal structure is fully encoded in the columns. This is why classical AR models and modern tabular learners on lag features share an identity: they are the same supervised problem viewed through different function classes, an arc this book returns to when linear autoregression reappears in the linear-attention and state-space view of Chapter 13.
2. Rolling and Expanding Statistics Beginner
A single lag is a sharp snapshot; a rolling statistic is a smoothed summary of the recent window. Rolling features replace the raw value at $t$ with an aggregate, the mean, standard deviation, minimum, maximum, or a quantile, computed over the last $w$ observations. The rolling mean over window $w$ is $\frac{1}{w}\sum_{i=0}^{w-1} x_{t-i}$, and rolling standard deviation, range, and quantiles follow the same windowed pattern. An expanding statistic uses everything from the start of the series up to $t$ rather than a fixed window, so its window grows over time. Exponentially weighted features take a softer view still, weighting recent observations more heavily than old ones with a decay factor, which gives a smooth summary that never fully forgets but steadily discounts the past.
The governing rule is non-negotiable and it is the cardinal leakage rule of Section 1.6 and Section 2.6 wearing windowing clothes: a window evaluated at time $t$ may use observations up to and including $t$ and nothing after. Pandas defaults to a trailing, right-aligned window, which is correct. The dangerous option is center=True, which centers the window on $t$ and therefore reaches into the future; on a feature meant for forecasting that is a silent look-ahead leak. Code 2.4.2 builds trailing rolling and expanding features and shows the correct, past-only semantics.
import numpy as np
import pandas as pd
rng = np.random.default_rng(7)
idx = pd.bdate_range("2021-01-01", periods=9, name="date")
price = pd.Series(100 + np.cumsum(rng.normal(0, 1.0, len(idx))),
index=idx, name="price")
df = pd.DataFrame({"price": price})
# Trailing window of length 3: the value at t uses x_{t-2}, x_{t-1}, x_t only.
# Pandas rolling is right-aligned by default (center=False), so it never
# peeks ahead. min_periods=3 keeps partial-window leakage out of early rows.
df["roll_mean_3"] = df["price"].rolling(window=3, min_periods=3).mean()
df["roll_std_3"] = df["price"].rolling(window=3, min_periods=3).std()
df["roll_max_3"] = df["price"].rolling(window=3, min_periods=3).max()
# Expanding statistic: everything seen so far, still strictly up to t.
df["exp_mean"] = df["price"].expanding(min_periods=1).mean()
# Exponentially weighted mean: recent points weighted more, no future use.
df["ewm_mean"] = df["price"].ewm(span=3, adjust=False).mean()
print(df.round(2))
center=True is deliberately avoided because it would let the window straddle the future. price roll_mean_3 roll_std_3 roll_max_3 exp_mean ewm_mean
date
2021-01-01 99.55 NaN NaN NaN 99.55 99.55
2021-01-04 99.86 NaN NaN NaN 99.71 99.71
2021-01-05 100.92 100.11 0.72 100.92 100.11 100.31
2021-01-06 100.49 100.42 0.53 100.92 100.21 100.40
2021-01-07 101.31 100.91 0.41 101.31 100.43 100.86
2021-01-08 100.77 100.86 0.41 101.31 100.48 100.81
2021-01-11 101.45 101.18 0.36 101.45 100.62 101.13
2021-01-12 102.10 101.44 0.67 102.10 100.81 101.62
2021-01-13 101.88 101.81 0.33 102.10 100.93 101.75
NaN because a full window of three is not yet available; the expanding and exponentially weighted columns fill from the first row because they accept whatever past exists. Every value depends only on the present and prior rows.A centered moving average, rolling(window=w, center=True), places the target time in the middle of its window, so the feature for time $t$ is computed partly from $x_{t+1}, x_{t+2}, \ldots$ Centered smoothing is perfectly legitimate for offline visualization and for decomposition you never feed back into a forecaster, but as a model input it is pure look-ahead bias: the model trains on a summary of its own future and reports an accuracy it cannot reproduce live. The same caution applies to any two-sided filter, to shift(-k) used as a feature rather than a label, and to interpolation that fills a gap using values that come after it. When in doubt, ask one question of every feature column: at the instant you must predict, would this number already exist? If not, it is a leak.
3. Calendar and Cyclical Encodings Beginner
A timestamp is itself a rich feature source. Hour of day, day of week, day of month, week of year, month, quarter, and flags for holidays, weekends, month-ends, and promotional periods carry enormous predictive signal in human-driven series: electricity demand, retail sales, web traffic, and call-center volume all swing on the clock and the calendar. Extracting these from a DatetimeIndex is direct and leakage-free, because the timestamp of the prediction instant is always known at prediction time.
The subtlety is in how a periodic field is encoded. Feeding the raw integer hour, $0$ through $23$, to a model implies that hour $23$ and hour $0$ are twenty-three units apart when in truth they are adjacent, one hour across midnight. The fix is a cyclical encoding that maps the period onto a circle, so that the last value and the first value sit next to each other. For a field with period $P$ (for the hour, $P = 24$), encode the integer value $v$ as a pair of sine and cosine coordinates,
$$\big(\sin\tfrac{2\pi v}{P},\ \cos\tfrac{2\pi v}{P}\big),$$which places every value on the unit circle and gives hour $23$ and hour $0$ nearly identical coordinates, exactly as their one-hour separation deserves. The same recipe encodes day of week with $P = 7$, month with $P = 12$, and day of year with $P = 365$. Code 2.4.3 extracts calendar fields and applies the cyclical sine and cosine encoding.
import numpy as np
import pandas as pd
# Hourly timestamps so the hour-of-day cycle is visible across midnight.
idx = pd.date_range("2021-03-01 21:00", periods=6, freq="h", name="ts")
df = pd.DataFrame(index=idx)
# Plain calendar fields, all known at prediction time, so no leakage.
df["hour"] = df.index.hour # 0..23
df["dayofweek"] = df.index.dayofweek # Mon=0 .. Sun=6
df["month"] = df.index.month # 1..12
# Cyclical encoding: map the hour onto the unit circle with period P=24,
# so hour 23 and hour 0 become neighbours instead of far-apart integers.
P = 24
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / P)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / P)
print(df.round(3))
hour dayofweek month hour_sin hour_cos
ts
2021-03-01 21:00:00 21 0 3 -0.707 0.707
2021-03-01 22:00:00 22 0 3 -0.500 0.866
2021-03-01 23:00:00 23 0 3 -0.259 0.966
2021-03-02 00:00:00 0 1 3 0.000 1.000
2021-03-02 01:00:00 1 1 3 0.259 0.966
2021-03-02 02:00:00 2 1 3 0.500 0.866
hour jumps from 23 to 0, but the cyclical coordinates move smoothly (cosine stays near 1, sine passes through 0), so the encoding correctly represents hour 23 and hour 0 as neighbours.Who: A demand-forecasting team at a national grocery chain planning daily store replenishment.
Situation: Their gradient-boosted forecaster used lags, rolling means, and integer day-of-week and month features, and scored well on ordinary weeks.
Problem: Around public holidays the forecasts were badly wrong in both directions: huge stockouts on the days before a holiday and overstock on the holiday itself, and the model could not learn the pattern because nothing in its features told it a holiday was coming.
Dilemma: The team could keep hand-patching predictions around known holidays each year, a brittle manual process, or invest in proper calendar features and retrain, accepting short-term disruption to a model that otherwise behaved.
Decision: They added a holiday calendar and richer calendar encodings rather than perpetuate the manual overrides, judging that a structural blind spot deserved a structural fix.
How: They joined a country holiday calendar (via the holidays package), added binary flags for the holiday itself and for the day before and after, encoded day-of-week and month cyclically with sine and cosine so weekly and yearly seasonality became smooth, and added a month-end and payday flag. All features derived only from the future timestamp being predicted, which is known in advance, so none introduced leakage.
Result: Holiday-period error fell by roughly forty percent and the manual overrides were retired. The biggest single win came from the day-before-holiday flag, which captured the pre-holiday stock-up surge the integer calendar fields had completely hidden.
Lesson: Calendar features are cheap, leakage-free, and often the highest-return engineering on human-driven series. Encode periodic fields cyclically, and never assume the model can infer holidays from raw dates; hand it the calendar explicitly.
Encode day of week as the integers zero through six and a linear model will happily conclude that Sunday (six) is "six times more Monday" than Monday (zero), which is nonsense the model has no way to question. Teams have shipped forecasters that systematically mispriced weekends for exactly this reason, then spent weeks hunting a data bug that was really an encoding bug. The circle does not have this problem: on it, Sunday and Monday are neighbours, as any calendar would insist.
4. Domain Transforms: Differencing, Returns, and Variance Stabilization Intermediate
Raw levels are often the wrong representation. Many series trend or grow multiplicatively, which violates the stationarity that classical models assume and which makes a single scale meaningless across years. Domain transforms reshape the series into something better behaved before any other feature is built. Differencing replaces the level with its change, $\Delta x_t = x_t - x_{t-1}$, which removes a linear trend and is the integrating step of the ARIMA model in Chapter 5. In finance the natural transform is the return. The simple return is $(p_t - p_{t-1})/p_{t-1}$, and the log return, preferred because it is additive across time and symmetric around zero, is
$$r_t = \ln\!\frac{p_t}{p_{t-1}} = \ln p_t - \ln p_{t-1}.$$Log returns turn the multiplicative growth of a price into an additive series whose rolling standard deviation is the volatility that GARCH models forecast in Chapter 6. For strictly positive, right-skewed series a log or Box-Cox transform stabilizes the variance and makes additive models appropriate, and it is inverted at prediction time to return to the original scale. All of these are computable from the past alone and so are leakage-free by construction.
Normalization is the dangerous exception. Subtracting a mean and dividing by a standard deviation is essential for many models, but the statistics must be estimated on the training side of the temporal boundary only, exactly as Section 2.3 demonstrated. A scaler fit on the whole series before splitting leaks the future mean and variance of the test period into every training row. The rule is mechanical: fit the scaler on train, transform train and test with those frozen statistics, and never call fit on data that includes the horizon you intend to forecast.
Of every transform in this section, normalization is the one that most often leaks. StandardScaler().fit_transform(X) over the full dataset computes the mean and standard deviation from rows that include your test horizon, so each training row is silently scaled by knowledge of the future. The same trap hides in a global MinMaxScaler (the maximum may occur in the test period), a global median imputation, and a global Box-Cox lambda. The cure is the one constant of temporal engineering: estimate every statistic on the past, freeze it, and merely apply it forward. Differencing, returns, and log transforms are safe because each value uses only its own row and the immediately preceding one; normalization is unsafe precisely because it pools across rows, and the pool must stop at the training boundary.
5. Automated Feature Extraction and the Learned Alternative Advanced
Hand-building lags, rolling statistics, and calendar fields is clear and controllable, but it does not scale to hundreds of series or to the long tail of descriptors a signal might need. Automated feature extraction answers this by computing large banks of statistics from each series window at once. The tsfresh library calculates hundreds of features per series, autocorrelations, entropy, spectral coefficients, peak counts, distributional moments, and pairs the extraction with a statistical relevance filter that prunes features unrelated to the target. The catch22 set takes the opposite philosophy: it distills the thousands of features of the broader hctsa catalogue down to twenty-two canonical time-series descriptors chosen for collective informativeness and low redundancy, giving a compact and fast fingerprint of a series. Both turn a window into a fixed-length vector that any tabular learner can consume.
The trade-off is between engineered banks and learned representations. A feature bank is interpretable, requires no training to compute, and is a formidable baseline, but it is fixed: it cannot discover a descriptor its authors did not anticipate, and on very large datasets a model that learns its own features can surpass it. That learned alternative is the subject of Part IV of this book, where self-supervised and contrastive methods and the embeddings produced by temporal foundation models extract representations directly from raw sequences. The honest practitioner uses banks as a strong, cheap, interpretable baseline and reaches for learned features when scale and accuracy justify the cost; the comparison is explored in depth in Part IV.
Everything built by hand in this section, and far more, is one call away. With tsfresh, extract_features(df, column_id="id", column_sort="time") computes the full bank, and select_features(X, y) prunes it to the relevant subset, replacing dozens of hand-written lag and rolling lines with a tested primitive that also handles the bookkeeping. The tsfeatures and pycatch22 packages give the compact catch22 fingerprint just as briefly. On the construction side, mlforecast from Nixtla generates lag, rolling, and date features for thousands of series with explicit past-only semantics, and sktime and Darts offer transformer classes that slot lag and window features straight into a leakage-safe pipeline. The reason to learn the mechanics first, as we did above, is that these one-liners only protect you from leakage if you understand what they are computing and where their window edges fall.
The 2024 to 2026 trajectory is a steady shift from hand-crafted and automated feature banks toward learned, transferable representations. Temporal foundation models, TimesFM, Moirai, Chronos, Lag-Llama, and MOMENT, are pretrained on enormous and diverse corpora and expose frozen embeddings that serve as off-the-shelf features for downstream forecasting, classification, and anomaly detection, often matching or beating bespoke tsfresh pipelines with no per-dataset engineering. At the same time, the classical banks have not vanished: catch22 and its successors remain the interpretable benchmark that every new learned representation is measured against, and recent work explicitly compares foundation-model embeddings to catch22 features on standard archives to quantify what the learned features actually buy. The emerging consensus is pragmatic. Engineered features still win when data is scarce, interpretability is required, or compute is tight; learned embeddings pull ahead at scale and transfer across domains. The two are converging into hybrid pipelines that feed both a feature bank and a foundation-model embedding into the same downstream model, a theme developed across Chapter 15.
Explain in your own words why encoding the hour of day as the single integer $0$ through $23$ misleads a linear model, and why the sine and cosine pair fixes it. Then argue why a single sine term alone is insufficient (hint: consider which two distinct hours share the same sine value) and why the cosine companion resolves the ambiguity. Finally, give the period $P$ you would use to cyclically encode day of week, month of year, and minute of hour.
Starting from a daily price series, write a function make_features(price) that returns a feature table containing: lags 1, 2, and 5; a trailing rolling mean and rolling standard deviation over window 5; the log return $r_t = \ln(p_t / p_{t-1})$; and cyclical sine and cosine encodings of the day of week. The target is the next-day log return. Verify by inspection that no column at row $t$ uses any value dated after $t$, then deliberately set center=True on the rolling calls, observe how the rolling columns change, and write one sentence explaining why that version would inflate a backtest.
Take any univariate classification archive (for example a UCR dataset). Extract features two ways: the compact catch22 set via pycatch22, and the full tsfresh bank followed by select_features. Train the same classifier on each and compare accuracy, feature count, and extraction time. In a short paragraph, state under what conditions, dataset size, interpretability needs, and compute budget, you would prefer each, and explain where a learned foundation-model embedding from Part IV would plausibly overtake both.