Representation Learning Fundamentals

"For years they cut me from a subsequence by hand: a mean here, a spectral peak there, a count of zero crossings, each feature a guess about what mattered. Now they ask me to discover for myself what a window of the world is about, and to say it in sixty-four numbers that every downstream task can read. I am a compression of recent history into a point, and I am still searching for which directions of myself carry the meaning."

An Embedding Vector Searching for the Meaning of a Subsequence

Part III ended with temporal foundation models in Chapter 15, models pretrained once on enormous and diverse time-series corpora and then applied, often with little or no fine-tuning, to tasks they never saw during training. That capability rests on an idea we have used implicitly but never isolated: that a network can learn a representation, an internal encoding of an input that is useful far beyond the objective it was trained on. Part IV makes representation learning the explicit subject. This first section lays the conceptual groundwork: what a representation is, what properties make one good, how the classical linear methods of Part II already learned representations, and how we measure the quality of a learned encoding. The sections that follow build on it. Section 16.2 designs the self-supervised pretext tasks that train encoders without labels; the remaining sections of the chapter develop contrastive learning and the temporal-specific objectives that this groundwork motivates.

The reason to elevate representations to a first-class object is economic and scientific at once. Economically, labeled time-series data is scarce and expensive (a cardiologist annotating arrhythmias, an engineer labeling machine faults) while raw unlabeled series are abundant, so an encoder trained on the cheap abundant data that then makes every labeled task easy is enormous leverage. Scientifically, a good representation is a statement about what is invariant and what is informative in a domain: if shifting a window in time does not change its label, the representation should be invariant to that shift, and forcing that invariance is how we inject prior knowledge about temporal structure into a learned system. The whole of Part IV is an elaboration of these two motivations, and they recur in every section.

The four competencies this section installs are these: to define an encoder $f_\theta: \mathcal{X} \to \mathbb{R}^d$ and a downstream linear probe, and to articulate why a vector-valued representation serves many tasks where a single end-to-end model serves one; to name the properties of a good representation and to specialize them to time series through the phase, scale, and warping invariances; to see PCA and dynamic factor analysis as linear representation learning and the nonlinear encoder as their generalization (a thread we follow from Section 6.4 forward); and to evaluate a representation by linear-probe accuracy, transfer, and clustering, the train-once-reuse-everywhere paradigm that Chapter 15 pushed to its limit. These are the load-bearing skills for the rest of Part IV.

1. What a Representation Is and Why We Learn One Beginner

A representation is a re-description of data in coordinates chosen to make subsequent work easy. For time series the data is a window: a contiguous slice $\mathbf{x} = (x_{t-L+1}, \dots, x_t)$ of $L$ steps (possibly multivariate, so each $x$ is itself a vector of channels). The encoder is a function $f_\theta$ with learnable parameters $\theta$ that maps that window to a vector,

where $\mathcal{X}$ is the space of input windows and $d$ is the embedding dimension, typically far smaller than the raw window length times its channel count. The vector $\mathbf{z}$ is the representation, also called the embedding, the latent, or the feature vector. The defining claim of representation learning is that a single such $\mathbf{z}$ exposes the structure needed for many different downstream tasks at once: a forecaster reads $\mathbf{z}$ to predict the next value, a classifier reads the same $\mathbf{z}$ to assign a label, an anomaly detector flags windows whose $\mathbf{z}$ is far from the bulk, and a retrieval system finds similar windows by nearest neighbors in $\mathbb{R}^d$. One encoding, many readouts.

The sharpest way to see what learning buys is to contrast it with the hand-crafted features of Section 2.4, where we built a window descriptor by computing a fixed catalogue of statistics: mean, variance, skew, dominant spectral frequency, autocorrelation at a few lags, count of zero crossings, and so on. That catalogue is a representation too, a map from window to vector, but it is fixed in advance by a human guess about what matters. Hand-crafted features have real virtues: they are interpretable, cheap, and need no training data. Their limitation is that any structure the designer did not anticipate is simply absent from the vector, and no downstream model can recover information the representation threw away. A learned encoder instead discovers, from data, which directions of the window carry task-relevant signal, and can represent structure no one thought to hand-code. The trade is the usual one: hand-crafted features cost human insight and risk omission, learned features cost data and compute and risk learning the wrong invariances.

This separation into a shared encoder and per-task heads is the structural commitment of Part IV, and it changes how we think about a model's value. An end-to-end forecaster trained on raw windows is worth exactly one task; its internal activations may encode reusable structure, but we discard them. An encoder, by contrast, is an asset: trained once, it amortizes across every task that reads its output, and the cost of adding a new task drops to fitting a small head. That amortization is precisely what makes the foundation-model paradigm of Chapter 15 economically sensible, and it is why "what makes an encoder good, independent of the head?" is the question this chapter exists to answer.

2. Properties of a Good Representation Intermediate

If an encoder can be anything, "good" needs a definition. Four properties recur across the representation-learning literature, and each has a precise temporal specialization. We state them generally, then ground each in what it means for a window of a time series.

Invariance versus equivariance. A representation is invariant to a transformation $g$ if $f_\theta(g \cdot \mathbf{x}) = f_\theta(\mathbf{x})$: applying $g$ to the input leaves the embedding unchanged. It is equivariant if the transformation passes through to a structured change in the embedding, $f_\theta(g \cdot \mathbf{x}) = \rho(g)\, f_\theta(\mathbf{x})$ for some known action $\rho$. Which you want depends on the task. A classifier of heartbeat type wants invariance to where in the window the beat falls (phase) and to recording gain (scale); a forecaster wants equivariance to a time shift, because the prediction must move when the window moves. The art of representation design is choosing, per nuisance factor, whether the encoder should ignore it (invariance) or track it (equivariance).

Smoothness. A representation is smooth if windows that are close in a task-relevant sense map to embeddings that are close in $\mathbb{R}^d$: small, meaningless perturbations of the input should not jump the embedding across the space. Smoothness is what makes nearest-neighbor retrieval and clustering in embedding space sensible, and what lets a linear head generalize from a finite training set to nearby unseen windows.

Disentanglement. A representation is disentangled if distinct generative factors of the data occupy distinct, ideally independent, coordinates of $\mathbf{z}$: one set of dimensions captures trend, another seasonality, another noise level, so that varying one factor in the world moves one part of the embedding. Disentanglement aids interpretability and makes downstream heads simpler, though perfect disentanglement is rarely achievable and not always necessary.

Linear separability. A representation is linearly separable for a task if a linear function of $\mathbf{z}$ suffices to solve it: classes are separated by a hyperplane, or the target is an affine readout. This is the property we most directly test, because it is the cleanest operational definition of "the structure is exposed". If a linear probe on $\mathbf{z}$ solves the task, the encoder has done the hard nonlinear work and left only a trivial readout. Formally, with a linear probe $\mathbf{W}_p \in \mathbb{R}^{c \times d}$ and bias $\mathbf{b}_p$, the prediction is

and the encoder is judged by how well this affine map performs with $f_\theta$ frozen. The probe is deliberately weak so that any success is credited to the representation, not to the head.

For time series these four properties acquire a specific list of invariances that a general-purpose image or text encoder would not name. Three matter most. Phase invariance: shifting a window by a few steps usually should not change its class (an ECG beat is the same beat whether it sits at the start or middle of the window), so $f_\theta$ should be approximately invariant to small time translations for classification, even while a forecaster wants the opposite. Scale invariance: multiplying a window by a constant (a sensor with a different gain, an asset quoted in a different unit) often should not change its meaning, so the representation should be invariant to amplitude scaling, which is why instance normalization of each window is a near-universal preprocessing step. Warping invariance: stretching or compressing a window along the time axis (a gait performed slowly versus quickly, a heartbeat at a different rate) frequently preserves identity, so robustness to mild temporal warping is the time-series analogue of the deformation invariances that vision encoders cultivate. These three, phase, scale, and warping, are the invariances the self-supervised objectives of Section 16.2 are specifically engineered to instill, by constructing training signals that hold identity fixed while varying exactly these nuisances.

3. The Classical-to-Learned Bridge: PCA and Dynamic Factor Models as Linear Encoders Intermediate

Representation learning can feel like a wholly new idea introduced by deep learning, but its linear case is decades old and we have already used it. Principal component analysis takes a data matrix and finds the orthogonal directions of maximal variance; projecting a window onto the top $d$ of those directions is precisely an encoder $f_\theta(\mathbf{x}) = \mathbf{P}^\top(\mathbf{x} - \boldsymbol{\mu})$, where $\mathbf{P} \in \mathbb{R}^{L \times d}$ holds the leading eigenvectors of the window covariance and $\boldsymbol{\mu}$ is the mean window. This is a learned representation by every criterion of subsection one: parameters $\mathbf{P}, \boldsymbol{\mu}$ fit from data, a fixed-length output $\mathbf{z} \in \mathbb{R}^d$, reused across tasks. It is simply constrained to be linear and to maximize reconstruction variance rather than any downstream objective.

The dynamic factor models of Section 6.4 are the temporal generalization of this same move. There we modeled a high-dimensional multivariate series as driven by a small number of latent factors evolving over time, $\mathbf{x}_t = \boldsymbol{\Lambda}\mathbf{f}_t + \boldsymbol{\epsilon}_t$ with $\mathbf{f}_t$ following its own low-dimensional dynamics. The factor vector $\mathbf{f}_t$ is exactly a learned representation of the observation $\mathbf{x}_t$: a low-dimensional code, fit from data, that captures the comovement structure many downstream analyses (forecasting the panel, detecting regime shifts) can read. The loading matrix $\boldsymbol{\Lambda}$ plays the role of $\mathbf{P}$, and inferring $\mathbf{f}_t$ from $\mathbf{x}_t$ is the encoder. Classical factor analysis is linear representation learning with a temporal prior on how the code evolves.

Why does removing the linearity constraint matter? Because the structure that distinguishes time-series classes is often not a linear subspace of the raw window. Two heartbeats of the same type, one slightly warped in time, live on a curved manifold that no single projection matrix can flatten; phase and warping invariance are intrinsically nonlinear requirements. A linear encoder can be invariant to amplitude scaling (project onto normalized directions) but cannot be invariant to time warping, because warping acts nonlinearly on the window. The nonlinear encoder is exactly the tool that can learn to collapse a curved manifold of equivalent windows to a single region of embedding space, which is why deep encoders outperform PCA on tasks dominated by these nonlinear nuisances, and underperform their own potential when trained with an objective (like pure reconstruction) that does not ask for the right invariances. That last caveat is the entire motivation for the pretext-task design of Section 16.2: the objective, not just the architecture, decides which invariances the encoder learns.

4. Evaluating a Representation: Linear Probe, Transfer, and Clustering Intermediate

A representation is a means, not an end, so it is evaluated by what it enables, not by its training loss. Three protocols dominate, and together they form the standard evaluation suite for any temporal encoder.

Linear-probe accuracy. Freeze the encoder, fit only a linear head (logistic regression for classification, ridge regression for forecasting) on the embeddings, and measure downstream performance. The probe is deliberately weak so that its success measures the encoder, not the head: if a single hyperplane separates the classes in embedding space, the encoder has exposed the structure linearly, which is the property of subsection two. Linear-probe accuracy is the single most reported number for a self-supervised encoder, precisely because it isolates representation quality from head capacity.

Transfer. Train the encoder on one corpus or task and evaluate it on a different one, with the encoder either frozen (feature transfer) or lightly fine-tuned. Strong transfer is the operational meaning of "general-purpose representation" and is the defining capability of the foundation models of Chapter 15: an encoder pretrained on broad data should make a held-out downstream task easy with few or no labels. Transfer measures whether the encoder learned domain structure or merely memorized its training distribution.

Clustering and retrieval. Without any labels, measure whether the embedding space organizes the data sensibly: do windows of the same (held-out) class form tight clusters, do nearest neighbors in $\mathbb{R}^d$ share semantics? Metrics like cluster purity, normalized mutual information, and nearest-neighbor accuracy (purity: the fraction of each cluster sharing one true label; NMI: label-cluster agreement normalized to $[0,1]$; nearest-neighbor accuracy: how often a window's closest neighbor shares its label) quantify the smoothness and separability of the space directly. This protocol matters most when labels are entirely absent at evaluation time, the common case for anomaly detection and exploratory analysis.

Across all three protocols the governing paradigm is the same: train the encoder once, reuse it everywhere. The encoder is fit by an expensive, often label-free procedure on abundant data; then a sequence of cheap downstream evaluations reads its frozen output. This is the paradigm Chapter 15 pushed to its extreme, pretraining one encoder on a corpus spanning domains and applying it zero-shot, and the evaluation protocols here are exactly how that chapter measures whether the pretraining paid off. A subtle but critical discipline accompanies the suite: the data split must respect time. Because windows overlap and nearby windows are correlated, a naive random split leaks future information into the training probe and inflates every metric, the temporal leakage hazard of Section 2.4. Probes must be fit and evaluated on time-disjoint segments, or the numbers are fiction.

5. Worked Example: Train an Encoder by a Pretext Task, Read It With a Linear Probe Advanced

We now make the whole pipeline concrete and runnable. The plan demonstrates every idea above on a small synthetic dataset where we control the ground truth. We generate windows belonging to two classes, sine waves at two different base frequencies, each instance randomized in phase and amplitude (the phase and scale nuisances of subsection two) so that the class is defined by frequency alone. We train an encoder with a label-free forecasting pretext: predict the next few steps of the window from its prefix, an objective that forces the encoder to capture the window's frequency content because frequency is what determines continuation. Then we freeze the encoder, fit a linear probe to classify frequency from the embedding, and measure accuracy, the linear-separability test of subsection two. Code 16.1.1 builds the data and the from-scratch encoder plus forecasting pretext.

import torch, torch.nn as nn

torch.manual_seed(0)
L, H, d, N = 48, 12, 16, 1200      # prefix length, horizon, embed dim, dataset size

def make_dataset(n):
    """Two classes by base frequency; each instance random in PHASE and AMPLITUDE."""
    t = torch.linspace(0, 1, L + H)
    freqs = torch.where(torch.rand(n) < 0.5,
                        torch.tensor(3.0), torch.tensor(6.0))   # class 0: 3 Hz, class 1: 6 Hz
    phase = torch.rand(n) * 6.283                                # nuisance: random phase
    amp   = 0.5 + torch.rand(n) * 1.5                            # nuisance: random amplitude
    series = amp[:, None] * torch.sin(2 * 3.14159 * freqs[:, None] * t + phase[:, None])
    series = series + 0.05 * torch.randn(n, L + H)              # observation noise
    x_prefix = series[:, :L]                                     # encoder input (the window)
    y_future = series[:, L:]                                     # pretext target (continuation)
    label = (freqs > 4.5).long()                                # downstream class, UNUSED in pretext
    return x_prefix, y_future, label

xp, yf, lab = make_dataset(N)
xp = (xp - xp.mean(1, keepdim=True)) / (xp.std(1, keepdim=True) + 1e-6)  # per-window scale invariance

class Encoder(nn.Module):
    """f_theta: window in R^L -> embedding in R^d, a small MLP."""
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(L, 64), nn.ReLU(), nn.Linear(64, d))
    def forward(self, x):
        return self.net(x)                                      # z = f_theta(x)

enc  = Encoder()
head = nn.Linear(d, H)                                          # pretext head: z -> next H steps
opt  = torch.optim.Adam(list(enc.parameters()) + list(head.parameters()), lr=1e-2)

for epoch in range(300):                                         # SELF-SUPERVISED: labels never used
    opt.zero_grad()
    z = enc(xp)                                                 # embed the window
    pred = head(z)                                              # forecast the continuation
    loss = ((pred - yf) ** 2).mean()                           # forecasting pretext loss
    loss.backward(); opt.step()
print("final pretext (forecasting) loss = %.4f" % loss.item())

final pretext (forecasting) loss = 0.0417

The encoder is now trained without a single label. The test of subsection two is whether the frequency class is linearly readable from the frozen embedding. Code 16.1.2 freezes the encoder, fits a linear probe on the embeddings of a training split, and reports accuracy on a held-out split, alongside the accuracy of the same linear probe applied to the raw windows, the baseline that says how much the encoder actually added.

from sklearn.linear_model import LogisticRegression
import numpy as np

xp_te, yf_te, lab_te = make_dataset(400)                        # held-out evaluation windows
xp_te = (xp_te - xp_te.mean(1, keepdim=True)) / (xp_te.std(1, keepdim=True) + 1e-6)

with torch.no_grad():                                           # FREEZE the encoder
    z_tr = enc(xp).numpy();    z_te = enc(xp_te).numpy()        # embeddings, train and test

# Linear probe ON THE EMBEDDING (the representation-quality test).
probe = LogisticRegression(max_iter=1000).fit(z_tr, lab.numpy())
acc_emb = probe.score(z_te, lab_te.numpy())

# Same linear probe ON RAW WINDOWS (baseline: what a probe gets without the encoder).
raw = LogisticRegression(max_iter=1000).fit(xp.numpy(), lab.numpy())
acc_raw = raw.score(xp_te.numpy(), lab_te.numpy())

print("linear probe on embedding : %.3f" % acc_emb)
print("linear probe on raw window: %.3f" % acc_raw)

linear probe on embedding : 0.987
linear probe on raw window: 0.793

Read the two code blocks together and the whole section is enacted. Code 16.1.1 trained an encoder with no labels by a forecasting pretext, the train-once half of the paradigm; Code 16.1.2 reused that frozen encoder under a linear probe, the reuse half, and measured a large gain in linear separability over raw data. The encoder learned the right invariance (it discarded phase and amplitude, which were nuisances, while keeping frequency, which was signal) purely from the structure of the pretext task, not from labels, which is exactly the mechanism the next section systematizes.

# tsai: TSStandardize, the SSL callbacks, and encoder wrappers
from sklearn.linear_model import LogisticRegression

# A library encoder: define, self-supervise on unlabeled windows, and embed, in three calls.
encoder = build_ssl_encoder(d=16)            # schematic: see tsai SSL tutorials for the concrete class
encoder.fit(unlabeled_windows)               # self-supervised training (forecast/mask/contrast)
z_tr, z_te = encoder.encode(train_x), encoder.encode(test_x)   # frozen embeddings
# Probe is still one line, exactly as in Code 16.1.2.
acc = LogisticRegression(max_iter=1000).fit(z_tr, train_y).score(z_te, test_y)

We have defined what a temporal representation is, named the properties that make one good, traced the line from the linear factor models of Part II to the nonlinear encoders of Part IV, set out how to evaluate an encoder by a frozen linear probe and by transfer, and demonstrated the full train-once-reuse-everywhere pipeline on a controlled dataset where a label-free forecasting pretext made the target class linearly separable. One question was deferred at every turn and now takes center stage: where does the training signal come from when there are no labels? The worked example used forecasting as a pretext, and the wearables case study hinted at masking and contrast, but we never asked which pretext to choose or why. Section 16.2 answers exactly that, turning the design of self-supervised pretext tasks for time series, masked reconstruction, forecasting, contrastive pair construction, and the augmentations that instill phase, scale, and warping invariance, into a deliberate engineering discipline. The encoder of this section was the asset; the next section is the factory that builds it without labels.