Unified Sequence Models Across Modalities

"They asked me to forecast, so I read the past and wrote the future. They asked me to generate, so I wrote the future without being asked. They asked me to decide, so I wrote the future I preferred. It was the same loop every time, and I never once stopped to ask which job it was. A sequence came in; a sequence went out; the rest was their interpretation."

An Architecture That Forecast, Generated, and Decided, and Never Asked Which Job It Was

Every earlier chapter solved a specific problem with a specific tool, and that was the honest way to learn the field. But a reader who has reached Part IX has by now noticed something that the chapter boundaries kept politely hidden: the tools keep turning out to be the same tool. The recurrence $\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$ that Section 7.6 identified as the shared skeleton of the Kalman filter, the HMM forward pass, and the RNN is the same recurrence that Chapter 13 parallelized into a state-space model, that Chapter 23 turned into an agent's belief state, and that Chapter 28 conditioned on a return to obtain a policy. This section names that pattern out loud and treats it as the thesis of the book: temporal intelligence is sequence modeling, and a general temporal intelligence is a sequence model general enough to wear every modality's clothes. We use the unified notation of Appendix A: $\mathbf{x}_{1:T}$ an input sequence, $\mathbf{z}_{1:T}$ its token representation, $\hat{\mathbf{y}}$ the model output, $p_\theta$ a single trained model.

Why open the final chapter this way rather than with a new technique? Because the most useful thing a textbook can do at its close is reorganize what the reader already knows into a smaller, more powerful idea. The reader does not need a thirty-seventh architecture; the reader needs to see that the thirty-six chapters were variations on one. That reorganization is not merely aesthetic. It is the practical engine behind the temporal foundation models of Chapter 15, the multimodal systems of the next section, and the agentic systems of Part VII: all of them are bets that one architecture, pretrained once, adapts to many temporal tasks more cheaply than many architectures each trained from scratch. This section is the conceptual statement of that bet, and the rest of the chapter cashes it out.

The competencies this section installs are four. To articulate the unifying claim precisely, naming the modalities and the architecture family that subsumes them. To tokenize an arbitrary temporal signal into the shared sequence-of-vectors format that every sequence operator consumes, recalling the Chronos tokenization of Section 15.2 and the patching of Section 14.5. To explain how forecasting, generation, representation, and decision making become one pretrain-then-adapt recipe. And to recognize the limits of unification, the no-free-lunch caveat that keeps a tuned simple baseline competitive, so that generality is chosen with eyes open rather than as fashion.

1. The Unifying Claim, Now Earned Beginner

State the claim plainly. A signal indexed by time, whatever its surface form, is a function from an ordered index set to a value space: prices indexed by trading day, an audio waveform indexed by sample, sensor channels indexed by reading, events indexed by their occurrence, video indexed by frame, an agent's experience indexed by step. Each is a sequence $\mathbf{x}_{1:T} = (\mathbf{x}_1, \dots, \mathbf{x}_T)$. The differences between modalities, continuous versus discrete values, regular versus irregular spacing, scalar versus high-dimensional elements, are differences in the value space and the index, not in the fact of sequentiality. And every architecture this book built to model sequences, the recurrent cell of Chapter 10, the temporal convolution of Chapter 11, the Transformer of Chapter 12, the structured state-space model of Chapter 13, and the diffusion sampler of Chapter 17, is a map from a sequence of vectors to a sequence of vectors. It does not know, and does not need to know, which modality produced the vectors.

This is not a slogan invented for the final chapter; it is the through-line the book has been quietly laying. The clearest evidence is the temporal thread, the deliberate set of callbacks in which one classical idea returns, chapter after chapter, in progressively more learned and more general form. The single most developed strand of that thread is recursive state estimation: a hidden state carried forward through time, updated by each new observation. It appears first as the Kalman filter, a hand-specified linear-Gaussian recurrence; it returns as the RNN, the same recurrence with the transition learned by the backpropagation through time of Section 9.3 rather than derived from a model; it returns again as the structured state-space model, the same recurrence made linear and parallelizable by an associative scan; it becomes an agent's memory, the same carried state now summarizing a history of observations and actions; and it becomes a policy, the same state read out as a decision. Table 36.1.1 lays this strand out as the spine of the book, with the section where each incarnation lives.

The table is the book in one image. The first row is Part II's hand-built filter; the last is Part VI's learned policy; every row in between is one chapter trading a little hand-specification for a little more learning and a little more generality, while the loop itself never changes. Once recursive state estimation, recursive generation, and recursive decision making are seen as the same loop with different read-outs, the claim that text, audio, sensors, events, video, and trajectories are one problem stops being an aspiration and becomes a description of what the preceding chapters already did.

2. Everything Is Tokens: One Format for Six Modalities Intermediate

If one architecture is to consume six modalities, the modalities must first be made to look alike. The operation that does this is tokenization: turning a raw temporal signal into a sequence of vectors, $\mathbf{x}_{1:T} \mapsto \mathbf{z}_{1:L}$, drawn from a shared representation space that the sequence operator understands. Tokenization is the universal adapter. Everything modality-specific lives in it, and once it is done, the downstream model is modality-agnostic. This is why the slogan "everything is a sequence" is, in engineering practice, the slogan "everything is tokens": the architecture sees only tokens, and the tokenizer is where a waveform, a price, and a word are reconciled into the same currency.

The book has already built two of the most important tokenizers, and recalling them shows the range. The first is patching, introduced for forecasting in Section 14.5: a long numeric series is cut into contiguous windows (patches), each window flattened and linearly projected to a token, so a series of thousands of scalars becomes a sequence of dozens of tokens. The second is value tokenization, the Chronos recipe of Section 15.2: each scalar observation is scaled and quantized into one of a fixed number of bins, and the bin index becomes a discrete token in exactly the vocabulary a language model expects, so a temperature series becomes a sentence in a numeric language. Between these two ideas, continuous patches and discrete value bins, lies a tokenizer for almost any modality, summarized in Table 36.1.2.

The payoff of forcing everything into one format is that a single model can be trained on, and transferred across, modalities that share no surface features. A model pretrained on the discrete value-tokens of millions of numeric series (the Chronos idea) can be fine-tuned on event tokens or, with a swapped tokenizer, on text, because the body of the model only ever manipulated abstract tokens. The cost, which the next sections take seriously, is that the tokenizer becomes a load-bearing design choice: a poor quantization throws away resolution a numeric task needed, and a patch length chosen badly hides the very dependency the model was meant to learn, the same window-versus-dependency tension that Section 9.3 raised for truncation. Tokenization is where modality knowledge is spent, and spending it well is most of the craft.

3. Convergence: Forecasting, Generation, Representation, Decision Intermediate

Tokenization unifies the inputs. A second, deeper convergence unifies the tasks. Forecasting, generation, representation learning, and decision making look like four different jobs, and the book taught them in four different parts, yet on a sequence of tokens they are all the same operation: predict tokens from context. Forecasting predicts the next value-tokens of a series. Generation, in Chapter 17, predicts the next tokens of a sample (autoregressively, or by denoising a sequence of tokens). Representation learning, in Chapter 16, predicts masked or future tokens as a pretext, then keeps the hidden states as features. Decision making, in Chapter 28, predicts the next action-token conditioned on a desired return. Four parts of this book, one objective: model $p_\theta(\mathbf{z}_t \mid \mathbf{z}_{

Because the objective is shared, the recipe is shared, and it is the recipe that organizes modern temporal AI: pretrain one large sequence model on vast unlabeled token streams with the next-token (or masked-token) objective, then adapt it cheaply to each downstream task by fine-tuning, by prompting with in-context examples, or by attaching a small head. This is the temporal foundation-model paradigm of Chapter 15 stated as a general principle: one expensive pretraining amortized over many inexpensive adaptations. Forecasting becomes "prompt the model with the history and read its continuation"; representation becomes "take the pretrained hidden state"; decision making becomes "condition on a return and read the action". The four tasks are four ways of querying one pretrained next-token model.

The figure compresses Parts III through VI into one diagram, and the compression is the point of the chapter. The four task heads on the right are the four parts the book devoted to forecasting deep architectures, representation learning, generative models, and sequential decision making; the single body in the center is the one sequence model they all turned out to share; the tokenizers on the left are Section 14.5 and Section 15.2 generalized. A general temporal intelligence is precisely this diagram with the body trained well enough that the four heads are thin.

4. What Unification Buys, and What It Costs Advanced

A book that spent thirty-five chapters building specialized tools would be dishonest to end by declaring all of them obsolete, and it does not. Unification is a trade, and stating both sides is the book's recurring humility. What unification buys is real: one codebase and one architecture instead of many; transfer, where a model pretrained on abundant data in one modality bootstraps a task with scarce data in another; emergent in-context adaptation, where a pretrained model forecasts a never-seen series from a prompt with no gradient step at all; and the sheer engineering leverage of amortizing one expensive pretraining over many cheap downstream uses. These are the advantages that made foundation models the dominant paradigm of Chapter 15.

What unification costs is equally real and easy to forget in the excitement. Generality trades against specialization: a universal tokenizer that quantizes values into bins discards resolution that a bespoke continuous model would keep, and a one-size body cannot exploit structure (a known seasonality, a physical constraint, a linear cointegration) that a purpose-built model encodes for free. The compute and data appetites are enormous, concentrating capability where those resources sit. And the failure modes are harder to anticipate, because a model trained on everything has no clean notion of "in distribution". Against all of this stands the most stubborn fact in forecasting, the one this book has repeated from Chapter 5 onward: a well-tuned simple baseline is shockingly hard to beat. Seasonal naive, exponential smoothing, and a tuned linear model remain the bar that every grand unified model must clear, and on many real series they win.

5. Worked Example: One Model, Three Modalities, Unchanged Advanced

The cleanest possible demonstration of the unifying claim is to write one small sequence model and apply it, without changing a line of the model, to three different temporal modalities: a numeric series, an event sequence, and a token sequence. Only the tokenizer (the modality-aware front) and the output head (the task-aware back) differ; the body is identical bytes in all three cases. Code 36.1.1 builds the from-scratch core: a tiny causal sequence model and three trivial tokenizers that all emit the same tensor shape.

import numpy as np
import torch
import torch.nn as nn

torch.manual_seed(0)
D = 16          # shared token dimension every modality maps into
L = 24          # sequence length (tokens) the body always sees

# --- The MODALITY-AGNOSTIC body: identical for all three modalities. ---
class SeqCore(nn.Module):
    """A tiny causal sequence model: one GRU over tokens of dim D, then a linear read-out.
    It never learns which modality produced the tokens; it only sees (batch, L, D)."""
    def __init__(self, d=D, hidden=32, n_out=D):
        super().__init__()
        self.rnn = nn.GRU(d, hidden, batch_first=True)   # the carried-state loop of Ch 10
        self.head = nn.Linear(hidden, n_out)             # swappable task-aware back

    def forward(self, z):                # z: (batch, L, D) tokens, ANY modality
        h, _ = self.rnn(z)               # h: (batch, L, hidden), one state per step
        return self.head(h)              # (batch, L, n_out): next-token prediction

# --- THREE modality-aware tokenizers, all emitting the SAME (batch, L, D) tensor. ---
def tokenize_numeric(series, proj):
    """Numeric series -> tokens by value-binning (the Chronos idea, Section 15.2)."""
    s = (series - series.mean()) / (series.std() + 1e-8)   # scale, as Chronos does
    bins = np.clip(((s + 3) / 6 * D).astype(int), 0, D - 1) # quantize into D value bins
    z = np.eye(D)[bins]                                      # one-hot bin -> a D-vector
    return torch.tensor(z[None], dtype=torch.float32)       # (1, L, D)

def tokenize_events(types):
    """Event sequence (integer type per step) -> tokens by type embedding (Ch 18)."""
    z = np.eye(D)[np.asarray(types) % D]                    # event type id -> a D-vector
    return torch.tensor(z[None], dtype=torch.float32)       # (1, L, D)

def tokenize_text(token_ids):
    """Token sequence (integer ids) -> tokens by id embedding (Ch 12)."""
    z = np.eye(D)[np.asarray(token_ids) % D]                # token id -> a D-vector
    return torch.tensor(z[None], dtype=torch.float32)       # (1, L, D)

core = SeqCore()                                             # ONE model instance

# Build one example per modality; all become (1, L, D).
num_series = np.sin(np.linspace(0, 8, L)) + 0.1 * np.random.randn(L)  # numeric
event_ids  = np.random.randint(0, D, size=L)                          # events
text_ids   = np.random.randint(0, D, size=L)                          # tokens

z_num = tokenize_numeric(num_series, None)
z_evt = tokenize_events(event_ids)
z_txt = tokenize_text(text_ids)

for name, z in [("numeric", z_num), ("events", z_evt), ("text", z_txt)]:
    out = core(z)                                           # SAME core, SAME call
    print(f"{name:8s}: input {tuple(z.shape)} -> output {tuple(out.shape)}")

numeric : input (1, 24, 16) -> output (1, 24, 16)
events  : input (1, 24, 16) -> output (1, 24, 16)
text    : input (1, 24, 16) -> output (1, 24, 16)

The output is the section's thesis in three lines: the same model, the same call, the same shapes, for a numeric series, an event sequence, and a token sequence. The only difference upstream was the tokenizer, exactly as subsection two argued. Now the library pair. In practice you do not hand-roll a tokenizer-plus-body for each modality; a foundation model ships the pretrained body and a tokenizer, and applying it across modalities is a few lines. Code 36.1.2 shows the HuggingFace-style unification with the Chronos forecaster, where one pretrained checkpoint forecasts an arbitrary numeric series with no training.

# pip install chronos-forecasting  (wraps a pretrained value-tokenized foundation model)
import torch
from chronos import ChronosPipeline

# One pretrained checkpoint: tokenizer + body + head, all inside.
pipe = ChronosPipeline.from_pretrained("amazon/chronos-t5-small",
                                       device_map="cpu", torch_dtype=torch.float32)

context = torch.tensor(num_series, dtype=torch.float32)   # ANY numeric series
forecast = pipe.predict(context, prediction_length=12)    # (num_samples, 12): zero-shot
print("zero-shot forecast shape:", tuple(forecast.shape))
print("median next step:", float(forecast.median(dim=0).values[0, 0]))

zero-shot forecast shape: (20, 12)
median next step: 0.731

6. Second-Generation Universal TSFMs: Scaling to General Temporal Intelligence Advanced

The temporal foundation models of Chapter 15 were introduced as the beginning of a paradigm shift, a single pretrained checkpoint applied zero-shot to arbitrary numeric series. The landscape from 2023 to 2026 has already passed through two recognizable generations and is converging on a third, and the progression mirrors the NLP arc from BERT and GPT-2 to ChatGPT to GPT-4 closely enough that the analogy is instructive rather than merely decorative.

The first generation (2023-2024) established the paradigm: Chronos (Ansari et al., 2024), TimesFM 1.0 (Das et al., 2024), and Moirai 1.0 (Woo et al., 2024) each pretrained on large and diverse corpora of univariate numeric series, then generalized zero-shot to new series in any domain. The unifying idea was straightforward: tokenize values (bin-and-embed for Chronos, patch for TimesFM, universal patch with any-variate masking for Moirai), pretrain next-token (or masked-token) prediction, and at inference simply prompt the model with the observed history. First-generation models were strong specialists: they handled univariate series well, they required no domain labels, and they outperformed classical models on broad benchmarks. What they did not handle was multivariate context, exogenous covariates, or tasks other than probabilistic forecasting.

The second generation (2025) addressed those gaps directly. Chronos-2, TimesFM 2.5, and Moirai 2.0 each introduced three structural advances. First, multivariate support: rather than treating each channel independently, second-generation models attend across channels within a patch, letting the model discover cross-series correlations that the first generation discarded. Second, covariate conditioning: the models accept observed and known-future covariates (calendar features, promotions, macroeconomic indicators) as side-channel tokens alongside the target series, which is the time-series equivalent of conditioning a language model on a system prompt. Third, architectural convergence toward decoder-only causal Transformers: the first generation split across encoder-only (Moirai's masked approach), encoder-decoder (Chronos's T5 backbone), and prefix-LM designs; the second generation largely abandoned encoder-decoder and masked designs in favor of decoder-only autoregressive generation. An important efficiency gain accompanied the shift: second-generation models achieve equivalent or better performance than their predecessors with roughly 30 times fewer parameters, because causal generation over patches is more information-efficient than bidirectional masking over raw values, and because the decoder-only architecture eliminates the encoder's redundant computation. The standard evaluation harness for second-generation models is GIFT-Eval (Aksu et al., 2025), a benchmark of 144 time series datasets spanning 7 domains, 4 frequencies, and both univariate and multivariate settings.

The third-generation trajectory (2026 and beyond) points toward a more radical unification: a single TSFM checkpoint that handles not only probabilistic forecasting but also anomaly detection, imputation, classification, and synthetic generation, all within one forward pass, distinguished only by the task token placed in the context. This is the GPT-3 to ChatGPT to GPT-4 arc applied to time series: GPT-3 was a strong next-token predictor; ChatGPT showed that instruction-tuning unlocks multi-task behavior from the same weights; GPT-4 extended the capability and context. For TSFMs, the analogous steps are: first-generation pretrained forecasters play the role of GPT-3; second-generation models with covariate conditioning and multivariate support play the role of GPT-3.5; the emerging multi-task TSFMs, trained to forecast, impute, detect, and classify within one prompt-driven interface, are the GPT-4 moment. The difference that makes this plausible for time series where it was hard in NLP is that all four tasks share the same underlying operation: model $p(\mathbf{x}_t \mid \mathbf{x}_{

The book's recurring "temporal thread" closes here. Every classical idea returned in learned form: the Kalman filter became the RNN, which became the structured SSM, which became agent memory, which became a decision policy (Table 36.1.1). TSFMs are the final turn of that thread applied to the filter itself: a second-generation TSFM with 1 billion parameters is, in a precise sense, a Kalman filter learned from the entirety of human-measured time. The Kalman filter of Section 7.6 maintained a Gaussian belief over a low-dimensional latent state, with the transition and observation matrices specified by a domain expert. A second-generation TSFM maintains a high-dimensional latent state across all the series it has ever been trained on, with the transition and observation mappings learned from billions of observed sequences, and it applies that learned state-tracking to any new series placed in its context. The loop $\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$ never changed; only $f$ grew from two hand-specified matrices to a billion learned parameters. Understanding this equivalence is not a historical curiosity: it predicts exactly which classical desiderata (observability, identifiability, structural sparsity, domain constraints) remain useful when building the next generation of TSFMs, because the Kalman filter's well-understood failure modes are the same modes that will surface as billion-parameter TSFMs scale into noisier, sparser, or more non-stationary data regimes. The transition from classical to learned is not a replacement but a scaling, and the classical literature is still the right place to look when the scaled version fails.

7. Exercises

These exercises ask you to test the unifying claim rather than take it on faith, in the three registers the book uses: conceptual, implementation, and open-ended.