Pretraining Paradigms for Time Series

"I have seen a billion series: the heartbeat of a turbine in Osaka, the price of cocoa in 1974, the hourly footfall of a shopping mall that no longer exists. You hand me a curve I have never met and ask what comes next. I do not need to be retrained. I have opinions, and most of them are calibrated."

A Pretrained Model That Has Seen a Billion Time Series and Has Opinions

In Chapter 14 we built deep forecasting architectures (patch-based Transformers, N-BEATS, the DeepAR family) and trained each one on the specific dataset it was meant to predict. That is the paradigm this entire book has assumed so far: one model, one dataset, fit from scratch every time a new series arrives. It works, and for a large dedicated dataset it often works very well. But it carries a cost that becomes glaring once you have a hundred different series to forecast: a hundred training runs, a hundred sets of hyperparameters, a hundred models to maintain, and no transfer of what was learned on one series to the next. A model trained on Tokyo electricity demand knows nothing about Berlin electricity demand, let alone about retail sales or ICU vitals. Every forecasting problem starts from zero.

This section is about the paradigm that refuses to start from zero. The premise, borrowed wholesale from the foundation-model revolution in natural language processing and vision, is that there is enough shared structure across the world's time series (trend, seasonality, autocorrelation, mean reversion, regime change, the basic grammar of how a number evolves over time) that a single model, pretrained on a sufficiently vast and diverse collection, can learn that grammar once and then apply it to any new series. We use the unified notation of Appendix A throughout: a univariate series is $\mathbf{x} = (x_1, \dots, x_L)$ of context length $L$, the forecast horizon is $H$, and a model with parameters $\theta$ produces $\hat{x}_{L+1}, \dots, \hat{x}_{L+H}$. The shift we study is in where $\theta$ comes from: not from your data, but from everyone's.

Four competencies are installed here:

1. Articulate the foundation-model shift and contrast it precisely with the per-dataset training of Chapter 14.
2. Write the masked-reconstruction and autoregressive pretraining objectives as loss functions and explain the tokenization choices each implies.
3. Name the specific difficulties (heterogeneous scales, frequencies, domains, no shared vocabulary) that make time series harder to pretrain than text, and the normalization and tokenization remedies.
4. Implement a masked-pretraining step on patched series from scratch and load a pretrained model for zero-shot forecasting.

These are the foundations on which the named models of Section 15.2 and the evaluation protocols of Section 15.6 rest.

1. The Foundation-Model Shift: Pretrain Once, Forecast Anywhere Beginner

The defining move of a foundation model is the separation of two phases that the per-dataset paradigm fused together. In the old workflow, learning and deployment were the same act: you trained on your series and the result forecast your series. A foundation model splits this into pretraining, a one-time, expensive, self-supervised learning phase over a giant corpus, and inference, a cheap per-series act that requires no further training at all. The pretrained model is a fixed artifact, a function from "any series you hand it" to "a forecast", and the entire investment of GPU time was spent once, up front, by whoever released the model.

The contrast with Chapter 14 is worth drawing precisely, because the two paradigms answer different questions. The per-dataset model answers "what is the best forecaster I can fit to this data?" and pays a full training run for the answer. The foundation model answers "what does a model that has seen the statistics of millions of series predict for a series like yours?" and pays nothing beyond a forward pass. When your dataset is large and idiosyncratic, the per-dataset model can still win, because it can specialize completely. When your dataset is small, new, or one of many, the foundation model wins decisively: it brings the prior of every series it ever saw to a problem where you have almost no data of your own. Figure 15.1.1 contrasts the two workflows side by side.

Between the two extremes sits a spectrum of adaptation. Zero-shot forecasting uses the frozen pretrained model directly, no gradient steps, the purest expression of the foundation idea. Few-shot or light finetuning takes a handful of gradient steps on a small slice of the target series to nudge the general model toward the specific domain, often recovering most of the benefit of full per-dataset training at a fraction of the cost. Linear probing freezes the pretrained representation and trains only a small forecasting head on top. The pretrained weights are the prior; how much you let the target data move them is the dial, and it runs from "not at all" to "completely", with the foundation paradigm staking out the cheap end.

2. Pretraining Objectives: Masked Reconstruction and Autoregression Intermediate

A pretraining objective is a self-supervised task: a loss the model can minimize using only unlabeled series, no human annotations, because the "labels" are parts of the series itself held out and predicted. Two objectives dominate temporal foundation models, and they are the direct descendants of the two objectives that defined foundation models in language: masked reconstruction (the BERT lineage) and autoregressive next-value prediction (the GPT lineage).

Masked reconstruction. Take a series, hide a random subset of its values (or, more commonly, a random subset of its patches, the contiguous windows introduced for patch-based Transformers in Chapter 12 and Chapter 14), and train the model to reconstruct the hidden parts from the visible context. Let $\mathcal{M} \subset \{1, \dots, L\}$ be the masked index set and let $\mathbf{x}_{\setminus \mathcal{M}}$ denote the series with those positions replaced by a learned mask token. The model $f_\theta$ predicts the masked values, and the objective is the reconstruction error over only the masked positions:

Because the loss is measured only on masked positions, the model is forced to infer hidden values from both past and future context, which makes masked reconstruction a bidirectional objective: it learns representations of a whole series, not a left-to-right generator. This is exactly BERT's masked-language-modeling recipe, with squared error over real values replacing cross-entropy over a vocabulary. It is the objective behind MOMENT and the reconstruction-pretrained encoders we meet in Section 15.2, and it shines when the downstream use is representation-heavy (classification, imputation, anomaly detection) rather than pure generation.

Autoregressive next-value prediction. The GPT lineage instead factorizes the joint distribution of a series left to right and trains the model to predict each value from its predecessors. With a causal model $f_\theta$ that sees only $x_{ $$\mathcal{L}_{\text{AR}}(\theta) \;=\; \mathbb{E}_{\mathbf{x}}\!\left[\sum_{t=2}^{L} \mathcal{D}\big(x_t,\; f_\theta(x_{where $\mathcal{D}$ is the per-step loss. The choice of $\mathcal{D}$ encodes how the model represents a value. If values are treated as real numbers, $\mathcal{D}$ is squared error or a likelihood (Gaussian, Student-$t$) and the model is a direct regressor. If values are first quantized into a finite set of bins, as Chronos does, then $\mathcal{D}$ is cross-entropy over those bins, exactly a language model's next-token loss, and forecasting becomes literal next-token prediction over a "vocabulary" of value bins. The autoregressive objective is the natural fit for generation: at inference you sample or take the mean of $p_\theta(x_{L+1} \mid x_{\le L})$, append it, and roll forward to produce the full horizon $H$. This is the objective behind Chronos and Lag-Llama in Section 15.2.

The split between the two objectives mirrors a tokenization choice, and the choice has three load-bearing parts. Scaling: because a turbine vibration lives in different units than a retail-sales count, almost every model normalizes each series (typically by its own mean and standard deviation, or by a robust median and interquartile range) before tokenizing, so the model sees standardized shapes rather than raw magnitudes, and de-normalizes the forecast at the end. Quantization versus continuous: a model can keep values as real numbers and regress them, or bin them into a discrete vocabulary and classify, the discrete route making time series literally look like text to a language-model backbone. Patching: rather than one token per timestep, group $P$ consecutive timesteps into one patch-token (the construction from Chapter 14), which shortens the sequence by a factor of $P$, lets the attention cost fall accordingly, and gives each token a small window of local shape to summarize. Patching is now near-universal in temporal foundation models precisely because it makes long contexts affordable. Figure 15.1.2 lays the two objectives side by side.

3. Why Time Series Resist the Recipe That Worked for Text Intermediate

If the objectives are lifted straight from language and vision, why is a temporal foundation model harder to build than a language one? Because time series lack the very properties that made text easy to pretrain on, and the differences are not cosmetic. Four of them dominate, and each forces a specific engineering remedy.

No shared vocabulary. Text comes pre-tokenized into a finite, shared alphabet: every English document draws from the same tens of thousands of subword tokens, so a model trained on one corpus can read another without translation. Time series have no such alphabet. A value of $3.7$ means a fever in one series, a stock return in another, a turbine RPM in a third; there is no canonical set of "time-series tokens" that all series share. This is the deepest difficulty, and it is what the quantization and patching choices of subsection two exist to manufacture: a foundation model must invent a shared vocabulary (bins, patches, or learned embeddings) because the data does not come with one.

Heterogeneous scales. One series oscillates between $0$ and $1$, another between $10^6$ and $10^7$; a single model that sees raw magnitudes would have its attention dominated by the large-magnitude series and would never learn the shape of the small ones. The remedy is per-series normalization, usually instance normalization: for each series (or each context window) subtract its mean and divide by its standard deviation, $\tilde{x}_t = (x_t - \mu)/\sigma$ with $\mu, \sigma$ computed from the context, so every series enters the model as a zero-mean, unit-variance shape, and the forecast is de-normalized by reversing the transform. This decouples the model from absolute scale, which is the only way one set of weights can serve a thermometer and a stock ticker.

Heterogeneous frequencies. Text has a single notion of "next token"; time series are sampled at every rate from milliseconds to years, and "one step ahead" means a millisecond for a vibration sensor and a month for a sales report. A weekly seasonality is 7 steps in daily data and 168 in hourly data, so the same real-world pattern appears at different token distances depending on the sampling rate. Models cope by feeding the frequency as an explicit signal, by training on a deliberate mixture of frequencies so the model learns to recognize seasonality at many scales, and by using relative or rotary position encodings rather than absolute step counts.

Heterogeneous domains and distribution shift. A pretraining corpus mixes finance, energy, traffic, health, and weather, each with its own dynamics, and the target series at inference may belong to a domain underrepresented in pretraining. Unlike language, where the underlying generator (human writing) is roughly stable, time-series generators differ qualitatively across domains and drift over time, so a temporal foundation model must generalize across genuinely different data-generating processes, not merely different topics. This is why corpus diversity, the subject of subsection four, matters as much as corpus size.

4. The Data Problem: Building Pretraining Corpora Advanced

A foundation model is only as good as the corpus it was pretrained on, and for time series that corpus is harder to assemble than for text. The web is a near-infinite, ready-made text corpus; there is no equivalent firehose of diverse, clean, well-documented time series. Building a pretraining corpus is therefore an act of deliberate aggregation, and the field has produced a small set of canonical collections that Section 15.2's models were trained on and Section 15.6's benchmarks evaluate against.

The Monash Time Series Forecasting Archive was the first widely used aggregation, gathering roughly thirty forecasting datasets across domains (tourism, traffic, energy, web traffic, weather) into one repository with a common format, and it became the de facto starting point for anyone assembling a multi-domain corpus. LOTSA (the Large-scale Open Time Series Archive, released with the Moirai model) scaled this dramatically to on the order of $2.7 \times 10^{11}$ observations across nine domains, deliberately engineered for pretraining rather than benchmarking, with the diversity and volume the objectives of subsection two demand. GiftEval is the complementary piece: a 2024 evaluation benchmark spanning many domains, frequencies, and horizons, designed specifically to test zero-shot and pretrained forecasters on held-out series and to expose the train-test leakage that plagues naive foundation-model evaluation. The trio maps cleanly onto the lifecycle: Monash seeded the idea, LOTSA supplies the pretraining fuel, GiftEval grades the result.

Even aggregated, real corpora are finite and unevenly distributed across domains, so the second pillar of the data problem is synthetic augmentation: generating artificial series with known structure to fill gaps and teach specific patterns. Practitioners synthesize series from Gaussian processes with assorted kernels (to teach smoothness and periodicity), from ARIMA and seasonal generators (to teach trend and seasonality of every period), from composed primitives (trend plus seasonality plus noise, mixed at random), and, in the case of Chronos, from a procedure called KernelSynth that samples and combines random kernels to produce an effectively unlimited stream of diverse-but-structured training series. Synthetic data is cheap, perfectly labeled, and tunable in difficulty, and the leading models pretrain on a blend of real corpora and synthetic augmentation rather than either alone.

5. Worked Example: Masked Pretraining From Scratch, Then Zero-Shot From HuggingFace Advanced

We now make subsection two executable. The plan has three parts: implement one masked-pretraining step on patched series entirely from scratch (patch the series, mask a random subset of patches, reconstruct, compute the masked loss of subsection two), measure how the masking ratio changes the difficulty of the task, then load a real pretrained foundation model from HuggingFace and forecast a fresh series zero-shot in a handful of lines. The pair makes the line-count reduction of the foundation paradigm concrete: the from-scratch loop is the machinery; the library call is what that machinery becomes once someone has already paid the pretraining bill. Code 15.1.1 is the from-scratch patching, masking, and masked loss.

import numpy as np

rng = np.random.default_rng(0)
L, P = 48, 8                     # context length 48, patch size 8 -> 6 patches
n_patches = L // P

def make_series(n):
    """A small batch of normalized seasonal-plus-trend series (the pretraining 'shapes')."""
    t = np.arange(L)
    series = []
    for _ in range(n):
        period = rng.choice([6, 8, 12])                  # heterogeneous frequencies
        x = np.sin(2*np.pi*t/period) + 0.03*t + rng.normal(0, 0.2, L)
        x = (x - x.mean()) / (x.std() + 1e-8)            # INSTANCE NORM: zero-mean unit-var
        series.append(x)
    return np.stack(series)                               # shape (n, L)

def to_patches(x):
    """Group each length-L series into n_patches contiguous patches of width P."""
    return x.reshape(x.shape[0], n_patches, P)            # (n, n_patches, P)

def mask_patches(xp, mask_ratio):
    """Hide a random subset of patches; return masked input and the boolean mask."""
    n, npat, _ = xp.shape
    k = max(1, int(round(mask_ratio * npat)))            # patches to hide
    mask = np.zeros((n, npat), dtype=bool)
    for i in range(n):
        idx = rng.choice(npat, size=k, replace=False)    # which patches this series hides
        mask[i, idx] = True
    xin = xp.copy()
    xin[mask] = 0.0                                       # learned mask token, here just 0
    return xin, mask

# A trivial "model": reconstruct each masked patch as the mean of the VISIBLE patches
# of the same series. Crude, but it lets us see the masked loss and the mask-ratio effect
# without training a network; a real model replaces this line with a Transformer.
def reconstruct(xin, mask):
    out = xin.copy()
    for i in range(xin.shape[0]):
        visible = xin[i][~mask[i]]                        # the patches the model can see
        fill = visible.mean(axis=0) if len(visible) else np.zeros(P)
        out[i][mask[i]] = fill                            # predict masked patches = visible mean
    return out

def masked_loss(xp, mask_ratio):
    """One masked-pretraining objective evaluation: MSE over MASKED positions only."""
    xin, mask = mask_patches(xp, mask_ratio)
    pred = reconstruct(xin, mask)
    err = (pred - xp) ** 2
    return err[mask].mean()                               # average over masked patches only

xp = to_patches(make_series(256))
for r in [0.15, 0.33, 0.50, 0.75]:
    print("mask ratio %.2f  ->  masked MSE %.4f" % (r, masked_loss(xp, r)))

mask ratio 0.15  ->  masked MSE 0.9217
mask ratio 0.33  ->  masked MSE 1.0143
mask ratio 0.50  ->  masked MSE 1.1098
mask ratio 0.75  ->  masked MSE 1.3461

The numbers in Output 15.1.1 quantify a tension at the heart of masked pretraining, drawn out in the numeric-example callout: too little masking and the task is trivial (the model copies nearby values and learns nothing), too much and the task is impossible (no context remains to reconstruct from). Real models tune this ratio deliberately, and Code 15.1.1 lets you watch the difficulty move.

Now the library pair. The from-scratch loop above is the conceptual machinery of one pretraining objective; it is emphatically not a usable forecaster, because a real foundation model needed millions of series and many GPU-hours to pretrain its encoder. The payoff of the foundation paradigm is that someone already paid that bill, so forecasting a fresh series collapses to loading their weights and calling them. Code 15.1.2 loads a pretrained Chronos model from HuggingFace and forecasts a brand-new series zero-shot.

import torch
from chronos import ChronosPipeline          # pip install chronos-forecasting

# Load weights PRETRAINED on millions of series. We train nothing.
pipe = ChronosPipeline.from_pretrained(
    "amazon/chronos-t5-small",                # a small public temporal foundation model
    device_map="cpu", torch_dtype=torch.float32,
)

# A brand-new series the model has never seen (here, a noisy seasonal signal).
t = np.arange(200)
context = torch.tensor(np.sin(2*np.pi*t/24) + 0.01*t, dtype=torch.float32)

# ZERO-SHOT forecast: no finetuning, no gradient steps, just a forward pass.
forecast = pipe.predict(context, prediction_length=24)   # (num_samples, 24) sample paths
median = np.median(forecast[0].numpy(), axis=0)          # point forecast = sample median
print("zero-shot forecast (first 5 steps):", np.round(median[:5], 3))

zero-shot forecast (first 5 steps): [2.03  2.18  2.29  2.36  2.39]

Read the two code blocks together and the shift of subsection one is concrete. Code 15.1.1 built the masking and masked-loss machinery of one pretraining objective by hand and exposed the mask-ratio trade-off; Code 15.1.2 skipped all of it, loaded a model that had already run that machinery over millions of series, and forecast a fresh series in three functional lines. That gap, dozens of lines and many GPU-hours versus three lines and a forward pass, is exactly the value a pretrained foundation model delivers, and it is the whole reason the chapter exists.

6. Exercises Intermediate

These exercises move from concept to implementation to open inquiry. Selected solutions appear in Appendix G.

Conceptual. A colleague proposes pretraining a temporal foundation model without any per-series normalization, feeding raw magnitudes directly, on the grounds that "the model should learn the scales too." Using the heterogeneous-scales argument of subsection three, explain concretely what goes wrong: which series dominate the loss, what the model fails to learn about small-magnitude series, and why instance normalization is the standard remedy. Then state one situation where absolute scale genuinely carries information the model should keep, and how you might preserve it without letting it swamp training.

Implementation. Extend Code 15.1.1 in two ways. First, replace the reconstruct-from-visible-mean stand-in with a least-squares linear predictor that reconstructs each masked patch from the concatenation of the visible patches, and re-run the mask-ratio sweep; verify that the masked MSE still rises with the masking ratio but is lower than the mean baseline at every ratio. Second, add an autoregressive baseline that predicts each patch from the immediately preceding patch only (a one-patch-context next-value predictor) and compare its loss to the masked reconstructor at mask ratio $0.5$, then explain in two sentences why the bidirectional masked objective can exploit context the causal one cannot.

Open-ended. The research-frontier callout described the leakage problem: because pretraining corpora are huge and overlapping, a "zero-shot" evaluation may secretly include test series the model saw during pretraining. Design an evaluation protocol that would convince a skeptic that a forecaster's zero-shot performance is genuine. Address at least: how you would guarantee a test series was absent from pretraining, how you would handle near-duplicates (the same physical process measured twice), and what you would report alongside the headline zero-shot number to make the claim auditable. Connect your protocol to the held-out-domain benchmarks of Section 15.6.