Sequence Learning Concepts and Tasks

"They hand me a sequence and a loss and expect me to know what I am. Am I collapsing the whole history into one number, or am I supposed to speak back a sequence of my own? Tell me my output shape and I will tell you who I am."

A Sequence Model Unsure if It Is Many-to-One or Many-to-Many

In Section 9.1 we built the simplest neural forecaster, a multilayer perceptron over a fixed lag window, and named the three limits (finite receptive field, no order, no memory between windows) that the carried-state recurrence of Section 7.6 is built to remove. That section answered "what is the simplest machine?". This section answers the prior question every practitioner must settle first: "what is the task?". The architecture follows from the task, not the other way around, and a startling number of modeling mistakes are really task-misclassification mistakes, for example training a many-to-one classifier when the problem actually demands a per-step aligned label. We will use the notation of the unified table in Appendix A: an input sequence $\mathbf{x}_{1:T} = (\mathbf{x}_1, \dots, \mathbf{x}_T)$ of length $T$, a target $\mathbf{y}$ that is either a single value, a vector, or its own sequence $\mathbf{y}_{1:T'}$ of length $T'$, and a model $f_\theta$ with parameters $\theta$.

By the end of this section you should be able to do four concrete things. Place any temporal problem you meet into the five-cell taxonomy and read off its output head from that placement. Pad and mask a ragged batch of sequences correctly, so that the loss never counts a padding position. Pick a loss that matches the target type, including the quantile loss that returns later when we forecast distributions. And explain why a fixed context vector limits an encoder-decoder, which is the motivation the attention chapter builds on. These four skills are the grammar of Part III; every later architecture is a different way of speaking this same grammar.

1. The Taxonomy of Sequence Tasks Beginner

The most useful first cut in all of sequence modeling sorts tasks by the relationship between input length and output length. There are exactly five shapes worth naming, and the discipline of naming the shape before reaching for a model is what keeps you from, say, building a generator when you needed a classifier. The five shapes are one-to-one, one-to-many, many-to-one, aligned many-to-many, and unaligned many-to-many. We take them in turn, each with its canonical temporal example.

A one-to-one task maps a single input to a single output with no temporal extent at all; it is the degenerate case, ordinary tabular regression or classification, and we list it only to mark the boundary of the family. A one-to-many task takes a single input and emits a sequence: a single conditioning vector seeds an entire generated trajectory, as when a static patient descriptor seeds a simulated vital-sign path, or a class label seeds a generated waveform. A many-to-one task consumes a whole sequence and returns one prediction; this splits by target type into classification (the sequence belongs to one of $C$ classes, for example recognizing an activity from an accelerometer window) and regression (the sequence maps to a real number, for example predicting next-quarter demand from a sales history).

The two many-to-many shapes are where most temporal subtlety lives, and the distinction between them is the single most important idea in this subsection. An aligned many-to-many task emits one output per input step, with the $t$-th output paired to the $t$-th input: sequence labeling, also called sequence tagging. Anomaly detection that flags each timestamp as normal or anomalous is aligned many-to-many, and so is per-step segmentation of a sensor stream into operating regimes. The lengths are equal, $T' = T$, and the alignment is given. An unaligned many-to-many task maps an input sequence to an output sequence of possibly different length with no given per-step correspondence: this is sequence-to-sequence, or transduction. Translating one language to another is the textbook case; in temporal AI, mapping an irregular event log to a regular forecast horizon, or transcribing a variable-length sensor burst into a variable-length symptom code sequence, are unaligned because $T' \ne T$ in general and nothing tells you which input step produced which output step. Figure 9.2.1 draws all five shapes at once.

The reason this taxonomy earns its place at the front of the chapter is that the output shape dictates the engineering. A many-to-one head pools the sequence into one vector and attaches a single classifier or regressor; an aligned many-to-many head attaches a small head at every step and computes the loss per step; an unaligned head needs an encoder that summarizes the input and a decoder that generates the output, the architecture of subsection five. Table 9.2.1 maps each shape to its temporal example, its output head, and its natural loss, so that classifying the task immediately yields the first three design decisions.

2. Inputs, Outputs, and the Shape of a Batch Beginner

Real temporal data is ragged. One patient stay is forty hours, another is four hundred; one customer has a two-year purchase history, another signed up last week. A neural network, however, wants rectangular tensors, so the first mechanical task in every sequence pipeline is turning a list of variable-length sequences into one padded tensor plus a mask that records which entries are real. Padding appends a filler value (commonly zero) to every sequence until all reach the batch's maximum length $T_{\max}$; the mask is a Boolean tensor of the same shape that is true at real positions and false at padding. Every downstream computation, the loss above all, must consult the mask so that padded positions contribute nothing, a discipline subsection six makes concrete in code.

The batch tensor for a single-feature series therefore has shape $(B, T_{\max})$, or $(B, T_{\max}, F)$ once each step carries $F$ features; PyTorch's convention with batch_first=True puts batch first, which we adopt throughout. Two further efficiencies matter in practice. Batching by length, sometimes called bucketing, groups sequences of similar length into the same batch so that $T_{\max}$ is not dominated by one outlier, which would otherwise waste computation on padding for every other sequence. And the packed-sequence representation, which we use in subsection six, lets the recurrent layers of Chapter 10 skip padded steps entirely rather than process and then discard them.

On the output side, a deeper fork appears whenever the task is to predict multiple future steps, the multi-horizon forecasting that recurs throughout Chapter 14. There are two ways to produce a horizon of length $H$, and the choice has consequences that ripple through training and inference. Autoregressive generation predicts one step, feeds that prediction back as input, and predicts the next, repeating $H$ times; the model only ever has a one-step head, but errors compound because each prediction conditions on the model's own possibly wrong earlier predictions. Direct multi-horizon output emits all $H$ values in a single forward pass from one head of width $H$; there is no error feedback loop, but the model cannot exploit its own intermediate predictions and must learn each horizon offset somewhat independently. This is the same windowing fork we set up in Chapter 2 when we slid input and output windows over a series; the input window became $\mathbf{x}_{1:T}$ and the output window became either one step generated $H$ times or $H$ steps emitted at once.

3. Losses and Targets for Temporal Tasks Intermediate

With the task shape fixed and the batch padded, the loss is the next decision, and it follows almost mechanically from the target type. For a real-valued regression target the workhorses are mean-squared error $\frac{1}{N}\sum (y_i - \hat{y}_i)^2$, which penalizes large errors quadratically and corresponds to a Gaussian likelihood, and mean-absolute error $\frac{1}{N}\sum |y_i - \hat{y}_i|$, which is robust to outliers and corresponds to a Laplace likelihood. When the target is a category, whether the single label of a many-to-one classifier or the per-step label of an aligned tagger, the loss is cross-entropy, $-\sum_c y_c \log \hat{p}_c$, the negative log-likelihood of the correct class under the model's softmax.

Temporal forecasting adds a loss the tabular world rarely needs: the quantile, or pinball, loss, which trains a model to predict a specific quantile of the predictive distribution rather than its mean. For target quantile level $\tau \in (0, 1)$ and error $u = y - \hat{y}$, the pinball loss is

It is an asymmetric absolute error: under-prediction (positive $u$) is weighted by $\tau$ and over-prediction by $1 - \tau$. Minimizing $\rho_\tau$ over the data drives $\hat{y}$ toward the conditional $\tau$-quantile, so fitting several $\tau$ values at once (say $0.1, 0.5, 0.9$) yields a predictive interval directly, which is why the pinball loss is the bread-and-butter objective of the probabilistic forecasting we develop fully in Chapter 19. We meet it again there as the bridge to conformal calibration; here it is enough to recognize it as the loss that turns a point forecaster into an interval forecaster.

The final loss-side topic is specific to autoregressive (one-to-many and unaligned many-to-many) models: how to train a model that, at inference, consumes its own previous outputs. The standard answer is teacher forcing: during training, feed the model the ground-truth previous token rather than its own prediction, so that step $t$ always conditions on the correct $\mathbf{y}_{t-1}$ from the data. This makes training stable and fully parallelizable over steps, because every step's input is known in advance. It also creates a discrepancy known as exposure bias: the model is only ever exposed to correct histories during training, but at inference it must consume its own imperfect generations, so a single early mistake leads it into a region of history it never saw in training, and errors compound. Figure 9.2.2 contrasts the two regimes.

Several remedies for exposure bias appear later in the book (see Chapter 14 and the table of contents) and are worth flagging now so the term is familiar: scheduled sampling mixes ground-truth and predicted inputs on a curriculum, and sequence-level objectives optimize the generated sequence as a whole rather than step by step. The non-autoregressive direct-output head of subsection two sidesteps exposure bias entirely by never feeding outputs back, which is one reason direct multi-horizon heads are popular in Chapter 14.

4. Conditioning and Covariates Intermediate

A sequence model rarely sees only the target series. Real forecasting problems come with side information of three distinct temporal kinds. Keeping them straight matters because each enters the model at a different point, and respecting when each is actually available is what prevents the leakage of Chapter 2. Static features do not vary with time: the store's region, the patient's baseline diagnosis, the product category. Known-future covariates are time-varying but known in advance for the entire forecast horizon: calendar effects (day of week, month), scheduled holidays, planned promotions, known price changes. Past covariates (also called observed covariates) are time-varying and known only up to the present: a related sensor reading, realized weather, an exogenous market index whose future values you do not have.

The availability distinction is not pedantic. A known-future covariate may legitimately inform the prediction at horizon step $h$, because you genuinely know next Tuesday is a holiday when you forecast it. A past covariate may not, because using its future value at training time would leak information the model will not have at deployment, manufacturing an optimistic backtest that collapses in production. Mechanically, static features are typically broadcast across all time steps and concatenated to every step's input, or used to initialize the recurrent state; known-future covariates are concatenated to the decoder inputs over the horizon; past covariates are concatenated only to the encoder inputs over the observed window. This three-way routing is the organizing principle of the Temporal Fusion Transformer we build in Chapter 14, which devotes separate processing paths to exactly these three covariate types and even learns per-feature importance weights, so the categories you are learning here are the literal inputs of that architecture.

5. Encoder-Decoder and the Bottleneck of a Fixed Context Advanced

The unaligned many-to-many shape demands a structure that the other four do not: a way to read a whole input sequence of length $T$ and then write a whole output sequence of length $T' \ne T$ with no given alignment between them. The classical answer is the encoder-decoder. An encoder, typically a recurrence of the kind Section 7.6 derived, consumes $\mathbf{x}_{1:T}$ and compresses it into a single fixed-width context vector $\mathbf{c}$, usually its final hidden state. A decoder, another recurrence, is initialized from $\mathbf{c}$ and generates the output sequence one step at a time, autoregressively, until it emits an end-of-sequence marker or reaches the target length. Formally,

where $\mathbf{s}_{t-1}$ is the decoder's own carried state. This factorization is elegant and was the dominant sequence-to-sequence design for years, but it hides a structural flaw that becomes acute as $T$ grows. Every detail the decoder will ever need about the input must pass through the single fixed-width vector $\mathbf{c}$. A vector of, say, 512 numbers is asked to summarize an input that might be hundreds of steps long, and it cannot grow with $T$. The earliest inputs, in particular, must survive the entire encoder recurrence without being overwritten, the precise compression limit that Section 7.6 named when it contrasted a recurrence (which compresses into a bounded state) with attention (which keeps everything). This is the information bottleneck of the fixed context vector.

The bottleneck has a memorable empirical signature: encoder-decoder translation quality degrades sharply as input length increases past what the context vector can hold, while a model that lets the decoder look back at all encoder states does not degrade the same way. That observation, made by Bahdanau, Cho, and Bengio in 2014, is precisely what motivated attention: rather than force the decoder to rely on one summary vector, give it a mechanism to attend to all the encoder's per-step states and form a fresh, query-dependent context at every output step. The fixed $\mathbf{c}$ becomes a time-varying $\mathbf{c}_t$ computed by a weighted look back over the whole input. We develop attention in full in Chapter 12; for now the load-bearing idea is that the bottleneck of this subsection is the problem statement that the next-but-one chapter answers. The encoder-decoder is not wrong; it is the structure whose one weakness defined the agenda of modern sequence modeling.

6. Worked Example: A Padded, Masked seq2seq Pipeline in PyTorch Intermediate

The cleanest way to make the previous five subsections concrete is to build the data plumbing once and then hang two different heads, a many-to-one and an aligned many-to-many, on the same padded, masked batch. We will assemble a tiny variable-length dataset, write a collate function that pads and builds a mask from scratch, run a recurrent encoder, and show that the many-to-one head pools to one prediction while the many-to-many head reads out every step. Then we replace the hand-written masking with PyTorch's nn.utils.rnn pack and pad utilities, the library pair, and demonstrate a teacher-forcing decoder step. Throughout, watch the tensor shapes; they are the running proof that the abstractions of this section are real objects. Code 9.2.1 starts with the dataset and the from-scratch collate.

import torch
from torch.utils.data import Dataset, DataLoader

class RaggedSeqDataset(Dataset):
    """Variable-length univariate sequences with a per-sequence class label
    (for many-to-one) and a per-step target (for aligned many-to-many)."""
    def __init__(self, sequences, labels):
        self.sequences = sequences        # list of 1D float tensors, ragged
        self.labels = labels              # list of int class labels
    def __len__(self):
        return len(self.sequences)
    def __getitem__(self, i):
        x = self.sequences[i]             # shape (T_i,)  variable length
        return x, self.labels[i]

def collate_pad(batch):
    """FROM SCRATCH: pad a ragged batch to T_max and build a boolean mask.
    Returns padded inputs (B, T_max), a mask (B, T_max), labels (B,),
    and the true lengths (B,)."""
    seqs, labels = zip(*batch)
    lengths = torch.tensor([len(s) for s in seqs])      # true length per seq
    T_max = int(lengths.max())
    B = len(seqs)
    padded = torch.zeros(B, T_max)                      # pad value = 0.0
    mask = torch.zeros(B, T_max, dtype=torch.bool)      # False everywhere
    for i, s in enumerate(seqs):
        padded[i, :len(s)] = s                          # copy real values
        mask[i, :len(s)] = True                         # mark real positions
    return padded, mask, torch.tensor(labels), lengths

# three ragged sequences of lengths 5, 3, 4 (the numeric example above)
seqs = [torch.arange(1., 6.), torch.arange(1., 4.), torch.arange(1., 5.)]
ds = RaggedSeqDataset(seqs, labels=[0, 1, 0])
loader = DataLoader(ds, batch_size=3, collate_fn=collate_pad)
padded, mask, labels, lengths = next(iter(loader))
print("padded shape:", tuple(padded.shape), " mask true count:", int(mask.sum()))
print(padded)
print(mask)

padded shape: (3, 5)  mask true count: 12
tensor([[1., 2., 3., 4., 5.],
        [1., 2., 3., 0., 0.],
        [1., 2., 3., 4., 0.]])
tensor([[ True,  True,  True,  True,  True],
        [ True,  True,  True, False, False],
        [ True,  True,  True,  True, False]])

With the batch in hand, Code 9.2.2 runs a recurrent encoder and attaches the two heads. The many-to-one head must pool over time, and pooling correctly means averaging only over real positions using the mask, the masked mean that prevents the padding bug quantified earlier. The many-to-many head instead keeps a prediction at every step. The shapes printed at the end are the point: one head collapses the time axis, the other preserves it.

import torch.nn as nn

class TwoHeadEncoder(nn.Module):
    def __init__(self, hidden=8, n_classes=2):
        super().__init__()
        self.rnn = nn.RNN(input_size=1, hidden_size=hidden, batch_first=True)
        self.cls_head = nn.Linear(hidden, n_classes)   # many-to-one: one label
        self.step_head = nn.Linear(hidden, 1)          # aligned m-to-m: per step
    def forward(self, x, mask):
        h_all, _ = self.rnn(x.unsqueeze(-1))           # (B, T, hidden)
        # many-to-one: masked mean pool over real steps only
        m = mask.unsqueeze(-1).float()                 # (B, T, 1)
        pooled = (h_all * m).sum(1) / m.sum(1)         # (B, hidden) masked mean
        logits = self.cls_head(pooled)                 # (B, n_classes)
        # aligned many-to-many: a prediction at every step
        per_step = self.step_head(h_all).squeeze(-1)   # (B, T)
        return logits, per_step

model = TwoHeadEncoder()
logits, per_step = model(padded, mask)
print("many-to-one logits:", tuple(logits.shape))     # (B, n_classes)
print("many-to-many out:  ", tuple(per_step.shape))   # (B, T)

# masked per-step regression loss: never count a padding position
target_steps = padded                                  # toy target = the input
sq_err = (per_step - target_steps) ** 2
masked_loss = (sq_err * mask).sum() / mask.sum()       # divide by 12, not 15
print("masked per-step MSE: %.4f" % masked_loss.item())

many-to-one logits: (3, 2)
many-to-many out:   (3, 5)
masked per-step MSE: 6.8123

Now the library pair. Our hand-written collate_pad plus the manual masked mean ran about a dozen lines and silently processed padding through the RNN before discarding it. PyTorch's nn.utils.rnn utilities do better: pad_sequence pads, and pack_padded_sequence compresses the padded batch into a packed form that the RNN steps over without ever computing on padding, which is both faster and removes any chance of padding contaminating the hidden states. Code 9.2.3 shows the same pipeline through these utilities and then performs one teacher-forcing decoder step, feeding the ground-truth previous value rather than the model's own output.

from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence, pad_packed_sequence

# LIBRARY PAIR: pad_sequence replaces the hand-written padding loop.
padded_lib = pad_sequence(seqs, batch_first=True)      # (B, T_max), zero-padded
lengths = torch.tensor([len(s) for s in seqs])

# pack so the RNN skips padding entirely (lengths must be on CPU)
packed = pack_padded_sequence(padded_lib.unsqueeze(-1), lengths,
                              batch_first=True, enforce_sorted=False)
enc = nn.RNN(input_size=1, hidden_size=8, batch_first=True)
packed_out, h_n = enc(packed)                          # RNN never sees padding
h_all, out_lengths = pad_packed_sequence(packed_out, batch_first=True)
print("unpacked hidden shape:", tuple(h_all.shape))    # (B, T_max, 8)

# --- one TEACHER-FORCING decoder step ---
dec = nn.RNNCell(input_size=1, hidden_size=8)
s = h_n.squeeze(0)                                      # decoder init = encoder final state
y_prev_truth = padded_lib[:, 0:1]                      # GROUND TRUTH y_0, not a prediction
s = dec(y_prev_truth, s)                               # teacher-forced step
print("decoder state after one teacher-forced step:", tuple(s.shape))

unpacked hidden shape: (3, 5, 8)
decoder state after one teacher-forced step: (3, 8)