Sampling, Resampling, and Aggregation

"People keep asking me to make their data tidy. I oblige, but understand what you are buying: every bar I return is an opinion. The mean is my favorite opinion, the last value is my laziest, and the one you forgot to choose is the one that will hurt you."

A Resampler With a Strong Opinion About the Mean

The previous section established the temporal index as the spine of every series and showed how to attach a clean, timezone-aware timestamp to each observation (see Section 2.1). With a trustworthy index in hand, the natural next question is the spacing of the observations along it. That spacing, the sampling interval, is rarely fixed by nature; it is fixed by the instrument, the logging policy, or a downstream convenience, and changing it is the single most common transformation in temporal data engineering. We resample to align channels that arrived on different clocks, to coarsen a torrent of ticks into manageable bars, to fill a grid for a model that demands regular spacing, and to match the cadence at which a decision is actually taken. Each of those is legitimate; each also has a way to go wrong that no error message will warn you about.

The plan of this section follows the lifecycle of a grid. Subsection one fixes intuition for what a sampling rate buys and forbids. Subsections two and three handle the two directions of change, coarser and finer, with their characteristic dangers of aliasing and invention. Subsection four aligns several series onto a shared clock, the operation that makes multivariate temporal data possible at all. Subsection five returns to the leakage theme of Section 1.6 and shows how an innocent bin-labeling choice can smuggle the future into the present.

1. Sampling Rate and What It Determines Beginner

A uniformly sampled series observes an underlying continuous process at instants spaced by a constant interval $\Delta t$, the sampling interval, whose reciprocal $f_s = 1/\Delta t$ is the sampling rate. This single number governs what is knowable. The classical sampling theorem states that a signal can be reconstructed from its samples only if it contains no frequency higher than half the sampling rate, a bound called the Nyquist frequency,

The practical reading for time series is blunt: if a pattern repeats faster than once every two samples, your data physically cannot represent it, and any model you train on that data is blind to it no matter how clever the architecture. A vibration that cycles every second is invisible in data logged once per second; you need at least two samples per cycle, and in practice several, to see it honestly. The sampling interval is therefore not a storage detail but a hard ceiling on the fastest structure available to every downstream stage. When you choose $\Delta t$ you are choosing the bandwidth of your own perception, and you cannot buy it back later by interpolation. This frequency-domain way of thinking, where a series is understood through the rates it contains, is developed in full when we reach spectral analysis; the forward pointer is Chapter 4, and the notation for sampling rate and frequency is fixed in the unified table of Appendix A.

2. Downsampling and Aggregation Intermediate

Downsampling moves a series onto a coarser grid, replacing the many samples that fall inside each new, wider bin with a single summary. The summary is the aggregator, and the choice of aggregator is a modeling decision disguised as a default. Formally, writing the coarse bin starting at time $\tau$ as the half-open window $[\tau, \tau + \Delta\tau)$, an aggregated value is a functional of the samples it contains; the mean, the workhorse, is the windowed average

Different questions want different functionals over $B_\tau$. A flow quantity such as rainfall or sales wants the sum; a level quantity such as temperature wants the mean; a state you want sampled at the bin edge wants the last (or first); and a financial bar conventionally wants all four of open, high, low, and close, the OHLC summary, because a single number throws away the path the price took inside the bar. Code 2.2.1 downsamples a minute-level price series to five-minute bars two ways, a plain mean and a full OHLC, so the difference in what each retains is concrete.

import numpy as np
import pandas as pd

# One trading hour of synthetic minute-level prices on a clean uniform index.
rng = np.random.default_rng(7)
idx = pd.date_range("2026-01-02 09:30", periods=60, freq="1min")
price = 100 + np.cumsum(rng.normal(0, 0.05, size=60))   # a random walk in price
s = pd.Series(price, index=idx, name="price")

# Downsample to 5-minute bars. The aggregator is the decision, not the freq.
mean_5min = s.resample("5min").mean()                   # level summary: average price
ohlc_5min = s.resample("5min").ohlc()                   # open/high/low/close per bar

print(mean_5min.head(3).round(3))
print()
print(ohlc_5min.head(3).round(3))

2026-01-02 09:30:00    100.041
2026-01-02 09:35:00    100.123
2026-01-02 09:40:00    100.205
Freq: 5min, Name: price, dtype: float64

                       open     high      low    close
2026-01-02 09:30:00  100.012  100.121   99.978  100.069
2026-01-02 09:35:00  100.083  100.196  100.061  100.151
2026-01-02 09:40:00  100.214  100.279  100.160  100.198

So far the aggregator looks like a free choice, but there is a trap underneath it that the mean does not advertise. Downsampling is decimation, and decimating a signal that still contains energy above the new Nyquist frequency causes aliasing: fast oscillations that the coarse grid cannot represent do not vanish, they fold down and masquerade as slow oscillations that were never really there. A vibration at fifty-five hertz sampled at sixty hertz reappears as a phantom five-hertz trend, and a mean over the bin does not save you because the mean is itself a weak filter that leaks high frequencies. The discipline borrowed from signal processing is to low-pass filter first and decimate second, so that the energy above the new Nyquist limit is removed before it can fold. The warning below states the failure mode in operational terms, and a real team that learned it the hard way appears in the practical example.

3. Upsampling and Interpolation Intermediate

Upsampling moves a series onto a finer grid, which immediately raises an awkward question: where do the new in-between values come from? They were never measured, so something must invent them, and the method of invention is a statement about what you believe happened between observations. Forward-fill (holding the last observed value until the next one arrives) encodes a step function and is the honest choice for a quantity that is genuinely piecewise-constant, such as a thermostat set-point or a configuration flag. Linear interpolation encodes a constant rate of change between samples and suits a slowly varying physical level. Spline interpolation encodes smoothness and is appropriate only when you have real reason to believe the underlying signal is smooth. None of these adds information; every one of them fabricates plausible values, and a model that then treats those fabricated points as if they were measurements will be overconfident about a resolution it does not have. Code 2.2.2 contrasts the three on a deliberately sparse series so the differences in what they assert are visible.

import numpy as np
import pandas as pd

# A sparse, irregularly observed level signal: four real measurements.
obs = pd.Series(
    [10.0, 14.0, 9.0, 12.0],
    index=pd.to_datetime(["2026-01-02 09:00", "2026-01-02 09:05",
                          "2026-01-02 09:20", "2026-01-02 09:35"]),
    name="level",
)

grid = obs.resample("5min").asfreq()          # target finer grid, gaps as NaN
ffill  = grid.ffill()                          # step: hold last observation
linear = grid.interpolate(method="time")       # constant rate between points
spline = grid.interpolate(method="spline", order=2)  # smooth curve through points

print(pd.concat(
    {"ffill": ffill, "linear": linear.round(2), "spline": spline.round(2)}, axis=1
))

                         ffill  linear  spline
2026-01-02 09:00:00     10.0   10.00   10.00
2026-01-02 09:05:00     14.0   14.00   14.00
2026-01-02 09:10:00     14.0   12.33   11.06
2026-01-02 09:15:00     14.0   10.67    9.20
2026-01-02 09:20:00      9.0    9.00    9.00
2026-01-02 09:25:00      9.0   10.00   10.13
2026-01-02 09:30:00      9.0   11.00   11.27
2026-01-02 09:35:00     12.0   12.00   12.00

When is upsampling legitimate rather than misleading? The honest use is alignment: bringing a slow channel onto the same grid as a fast one so they can be modeled jointly, where forward-fill correctly says "the last known value still holds" and introduces no claim beyond persistence. The misleading use is manufacturing apparent resolution, presenting an interpolated minute-level curve as if the underlying quantity were actually observed every minute, which inflates sample counts, deflates estimated variance, and tempts a model to learn the interpolation rule instead of the signal. A crucial caveat ties back to the leakage discipline of Section 1.6: linear and spline interpolation are two-sided, using the next observation to fill the gap, so an interpolated value at time $t$ depends on data from after $t$ and must never feed a feature used to predict at $t$. Only the strictly backward-looking forward-fill is leakage-safe for real-time features. The honest treatment of genuinely irregular and missing observations, including masking and models that consume irregular timestamps directly without resampling at all, is the subject of the next section, Section 2.3.

4. Aligning Multiple Series Intermediate

Multivariate temporal modeling presumes that at each timestamp you can read every channel, yet real channels arrive on their own clocks: a price ticks irregularly, a sentiment feed updates on news, a sensor reports on a fixed cadence, and no single instant has all three freshly observed. Aligning them onto a common clock is therefore a precondition for joint modeling, and the tool of choice for asynchronous channels is the as-of join, which for each row of one series finds the most recent row of another whose timestamp is at or before it. This is exactly persistence done correctly: each channel carries its last known value forward to the query time and no value ever comes from the future. The pandas primitive is merge_asof, and a tolerance window caps how stale a carried-forward value may be, after which the match is dropped rather than silently trusted. Code 2.2.3 joins an irregular trade series to an irregular quote series exactly as a market-data pipeline would.

import pandas as pd

trades = pd.DataFrame({
    "time":  pd.to_datetime(["2026-01-02 09:00:01", "2026-01-02 09:00:04",
                             "2026-01-02 09:00:09"]),
    "trade_px": [100.1, 100.3, 100.2],
})
quotes = pd.DataFrame({
    "time": pd.to_datetime(["2026-01-02 09:00:00", "2026-01-02 09:00:03",
                            "2026-01-02 09:00:07", "2026-01-02 09:00:30"]),
    "bid":  [100.0, 100.2, 100.1, 100.4],
})

# For each trade, attach the most recent quote at or before it (never after),
# but only if that quote is no more than 5 seconds stale.
aligned = pd.merge_asof(
    trades, quotes, on="time",
    direction="backward",                       # look back in time only
    tolerance=pd.Timedelta("5s"),               # drop matches staler than 5s
)
print(aligned)

                 time  trade_px    bid
0 2026-01-02 09:00:01     100.1  100.0
1 2026-01-02 09:00:04     100.3  100.2
2 2026-01-02 09:00:09     100.2    NaN

The backward direction is not a cosmetic default; it is the leakage guard. A forward or nearest as-of join would let a trade borrow a quote that had not yet happened, which is precisely the look-ahead bias of Section 1.6 wearing a join's clothing. For state-space and filtering models that consume aligned multivariate streams, this asynchronous-channel alignment is the data-engineering step that feeds them; the modeling side picks up in Chapter 7, and the book's running healthcare dataset, an irregularly sampled multivariate clinical series, is exactly the kind of asynchronous record that merge_asof with a tolerance was built for.

5. Aggregation Semantics and Leakage Advanced

The most consequential resampling decisions are the quietest: which edge of a bin owns its boundary timestamp, and which edge the resulting bar is labeled with. Pandas exposes these as the closed and label arguments, and on a daily or hourly grid they look like trivia. They are not. A bar that bundles observations from $[\tau, \tau + \Delta\tau)$ but is labeled with the left edge $\tau$ is reporting, at a timestamp $\tau$, a summary that includes information from all the way up to $\tau + \Delta\tau$. If a downstream feature reads that bar as "the value known at $\tau$" it is reading the future, because the bar's close was not knowable until the bin ended. This is leakage by labeling, and it is invisible because the timestamps all look causally innocent. Code 2.2.4 makes the off-by-one-bin concrete on a tiny series.

import pandas as pd

idx = pd.date_range("2026-01-02 09:00", periods=6, freq="1h")
s = pd.Series([1, 2, 3, 4, 5, 6], index=idx, name="x")

# Two-hour bars. The only difference is which edge owns and labels the bin.
left  = s.resample("2h", closed="left",  label="left").sum()
right = s.resample("2h", closed="right", label="right").sum()

print("closed/label = left :"); print(left)
print("\nclosed/label = right:"); print(right)

closed/label = left :
2026-01-02 09:00:00     3
2026-01-02 11:00:00     7
2026-01-02 13:00:00    11
Freq: 2h, Name: x, dtype: int64

closed/label = right:
2026-01-02 10:00:00     3
2026-01-02 12:00:00     7
2026-01-02 14:00:00    11
Freq: 2h, Name: x, dtype: int64

The rule that prevents the leak is to label aggregated bars with the timestamp at which the bar became fully observable, the right edge of a closed-right bin, whenever the bar will be used as a feature for prediction. A bar must never be available to a model before its own window has closed. This connects directly to the decision cadence: in trading you must act before the bar closes, so a model that quietly conditions on the close of the very bar it is trading is using information it will not have live, and the backtest that results is the cardinal sin of Section 1.6 reincarnated as a labeling default. Because these conventions are so easy to get wrong and so costly when they are, they belong to evaluation as much as to engineering; the leakage-proof pipelines and time-aware evaluation protocols that police bin labeling end to end are built in Section 2.6.