Temporal Embeddings and Disentanglement

The limits and the critique. Honesty about this literature is mandatory. The influential analysis of Locatello and colleagues (2019) proved that unsupervised disentanglement is fundamentally unidentifiable without inductive biases or some supervision: for any disentangled representation there are infinitely many equally-good entangled ones related by a transformation the data cannot distinguish, so no purely unsupervised objective can be guaranteed to find the axes you want. In practice this means disentanglement results are sensitive to random seed and hyperparameters, the headline metrics vary widely across runs, and a little weak supervision or a structural bias (like the static/dynamic split above, which is an inductive bias) does far more than any amount of independence penalty alone. The working stance for 2026: treat strong unsupervised disentanglement claims with skepticism, lean on structural biases and weak labels when you need separated factors, and measure rather than assume.

4. Evaluating Embeddings Advanced

An embedding is only as good as what it lets you do, so evaluation is never a single number. Three complementary families of metric answer three different questions.

Probing asks: how much task-relevant information does the embedding contain? Freeze the encoder, train a deliberately simple head (linear or a shallow MLP) on top of the embeddings to predict a property of interest (a label, a regime, a future value), and report the head's accuracy. A linear probe is the standard because a high linear accuracy shows the information is not merely present but laid out in a linearly accessible way; if you need a deep probe to extract a property, the embedding has it but has not organized it well. Probing is the workhorse evaluation for self-supervised representations across the field.

Retrieval metrics ask: does distance agree with relevance? Using labels as ground truth, embed a query, rank the database by embedding distance, and score the ranking. Precision@k (fraction of the top $k$ that share the query's label), recall@k, and mean average precision (mAP) are the standard scores, and they directly measure the property subsection one depends on, that near in embedding space means similar in reality.

Disentanglement metrics ask: does each axis carry one factor? These require known generative factors (so they live on synthetic or carefully-labeled data). The Mutual Information Gap (MIG) measures, for each true factor, the gap in mutual information between its top-two most-informative latent dimensions: a large gap means one dimension dominates (good, one axis owns the factor), a small gap means the factor is smeared across dimensions (entangled). Written out, for a factor $v_k$ and latent coordinates $z_j$, MIG normalizes by the factor's entropy $H(v_k)$ and reads

so a score near one means each factor is captured by a single, clearly dominant latent and a score near zero means the information is split. The complementary SAP score and the DCI triple (Disentanglement, Completeness, Informativeness) measure the same idea from the angles of axis-to-factor and factor-to-axis mappings: DCI in particular separates disentanglement (each latent informs at most one factor) from completeness (each factor is informed by at most one latent), two properties that can fail independently and that a single scalar like MIG blurs together. Because subsection three warned that these scores swing with the seed, the honest protocol is to report them as a distribution over many runs, not a single best value, and to fix the random seed and hyperparameters before, not after, looking at the metric.

5. Worked Example: An Embedding Space, From Scratch and From a Library Advanced

We now build the full pipeline of subsection one end to end: train a tiny encoder, embed a database of windows, do nearest-neighbor retrieval, and visualize the space with t-SNE, all by hand so the structure is exposed, then collapse the embedding and retrieval to a few library lines. The data is a synthetic mixture of three regime types (a flat-noise regime, a trend regime, and a seasonal regime) so that "did the embedding separate the regimes?" has a checkable answer. Code 16.5.1 builds the data and a small contrastive-style encoder.

import numpy as np
import torch
import torch.nn as nn

rng = np.random.default_rng(0)
torch.manual_seed(0)
W, N_PER = 64, 200          # window width; windows per regime
t = np.linspace(0, 1, W)

def make_window(regime):
    """Three regime types: 0 = flat noise, 1 = trend, 2 = seasonal."""
    noise = rng.normal(0, 0.15, size=W)
    if regime == 0:
        return noise
    if regime == 1:
        slope = rng.uniform(1.0, 3.0)
        return slope * t + noise                 # rising trend
    freq = rng.uniform(3.0, 6.0)
    return np.sin(2 * np.pi * freq * t) + noise   # seasonal

X = np.stack([make_window(r) for r in range(3) for _ in range(N_PER)])
labels = np.array([r for r in range(3) for _ in range(N_PER)])  # ground-truth regime
Xt = torch.tensor(X, dtype=torch.float32)

class Encoder(nn.Module):
    """A small MLP encoder mapping a width-W window to a d-dim L2-normalized embedding."""
    def __init__(self, w=W, d=8):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(w, 32), nn.ReLU(), nn.Linear(32, d))
    def forward(self, x):
        z = self.net(x)
        return z / z.norm(dim=-1, keepdim=True).clamp_min(1e-8)  # unit sphere -> cosine

enc = Encoder()
print("data:", X.shape, "| embedding dim:", enc.net[-1].out_features)

data: (600, 64) | embedding dim: 8

We train the encoder with a from-scratch contrastive loss: build positive pairs by augmenting each window (add jitter, the simplest time-series augmentation) and pull each window's two views together while pushing different windows apart. Code 16.5.2 is the InfoNCE training loop written by hand so the objective is visible.

def augment(x):
    """Cheap positive-pair augmentation: add small Gaussian jitter."""
    return x + 0.10 * torch.randn_like(x)

def info_nce(za, zb, tau=0.2):
    """Contrastive loss: za[i] and zb[i] are a positive pair; all others are negatives."""
    logits = za @ zb.t() / tau                  # cosine sims (z already unit norm) / temperature
    targets = torch.arange(za.size(0))          # the matching index is the positive
    return nn.functional.cross_entropy(logits, targets)

opt = torch.optim.Adam(enc.parameters(), lr=1e-3)
for step in range(400):
    idx = torch.randint(0, Xt.size(0), (128,))  # a minibatch of windows
    batch = Xt[idx]
    za, zb = enc(augment(batch)), enc(augment(batch))   # two augmented views
    loss = info_nce(za, zb)
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 100 == 0:
        print("step %3d  contrastive loss = %.4f" % (step, loss.item()))

with torch.no_grad():
    Z = enc(Xt).numpy()                         # the embedding database

step   0  contrastive loss = 4.8742
step 100  contrastive loss = 0.6113
step 200  contrastive loss = 0.4359
step 300  contrastive loss = 0.3717

Now the two payoff queries, both written from scratch. Code 16.5.3 does nearest-neighbor retrieval by sorting cosine similarities, then measures retrieval quality (precision@10) and clustering quality (does $k$-means on the embeddings recover the three regimes?) against the held-out labels.

from sklearn.cluster import KMeans

# --- nearest-neighbor retrieval, from scratch ---
def retrieve(query_i, k=10):
    sims = Z @ Z[query_i]                         # cosine sims (Z is unit norm)
    order = np.argsort(-sims)                     # descending similarity
    return order[1:k + 1]                         # exclude the query itself

# precision@10 averaged over a sample of queries: fraction of neighbors sharing the label
p_at_10 = np.mean([
    np.mean(labels[retrieve(i)] == labels[i])
    for i in rng.choice(len(Z), size=100, replace=False)
])
print("retrieval precision@10 = %.3f" % p_at_10)

# --- clustering regimes, from scratch geometry handed to k-means ---
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(Z)
# purity: how cleanly each cluster maps to one true regime
purity = np.mean([
    np.bincount(labels[km.labels_ == c]).max() / np.sum(km.labels_ == c)
    for c in range(3)
])
print("cluster purity = %.3f" % purity)

# --- cosine similarity of two specific subsequences (numeric-example, in code) ---
i_trend, i_seasonal = np.where(labels == 1)[0][0], np.where(labels == 2)[0][0]
print("cos(trend, seasonal) = %.3f" % (Z[i_trend] @ Z[i_seasonal]))
print("cos(trend, trend')   = %.3f" % (Z[i_trend] @ Z[np.where(labels == 1)[0][1]]))

retrieval precision@10 = 0.981
cluster purity = 1.000
cos(trend, seasonal) = 0.118
cos(trend, trend')   = 0.964

Finally the visualization and the library pair. Code 16.5.4 projects the 8-dimensional embeddings to two dimensions with t-SNE for a picture of the space, then shows that the entire embed-and-retrieve pipeline of Codes 16.5.1 to 16.5.3 reduces, when a pretrained representation is acceptable, to a few lines.

from sklearn.manifold import TSNE

# --- visualize the structure (t-SNE projection to 2D) ---
emb2d = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(Z)
print("t-SNE projection:", emb2d.shape, "(plot emb2d colored by labels to see 3 islands)")

# --- the library pair: embedding + retrieval in a handful of lines ---
# tslearn provides distance-based time-series retrieval without a hand-built encoder;
# here we use a KNN over the learned embedding to make the line-count point concrete.
from sklearn.neighbors import NearestNeighbors
index = NearestNeighbors(n_neighbors=11, metric="cosine").fit(Z)   # build the index
_, nbrs = index.kneighbors(Z[[i_trend]])                            # query in one call
print("library retrieval neighbors share regime:",
      np.mean(labels[nbrs[0][1:]] == labels[i_trend]))

t-SNE projection: (600, 2) (plot emb2d colored by labels to see 3 islands)
library retrieval neighbors share regime: 1.0

Step back and read the four code blocks as one argument. Code 16.5.1 and 16.5.2 trained an encoder; Code 16.5.3 turned its output into retrieval, clustering, and a cosine comparison purely through distance; Code 16.5.4 visualized the space and showed the same retrieval in a couple of library lines. The whole of subsection one is in those numbers: a single embedding, four geometric uses, near-perfect because the encoder made distance mean similarity.

# Pretrained embedder + a vector index: no training loop, no hand-written retriever.
from momentfm import MOMENTPipeline        # a pretrained time-series foundation model
from sklearn.neighbors import NearestNeighbors

model = MOMENTPipeline.from_pretrained(    # frozen general-purpose encoder
    "AutonLab/MOMENT-1-large", model_kwargs={"task_name": "embedding"})
Z = model(x_enc=windows).embeddings        # (N, d) embeddings in one call
index = NearestNeighbors(metric="cosine").fit(Z)
neighbors = index.kneighbors(Z[[query_i]], n_neighbors=10)[1]   # retrieval in one call

6. Foundation Model Encoders as Temporal Representation Backbones Intermediate

The representation-learning strategies of Sections 16.2 through 16.4, TS-TCC, TNC, CoST, BTSF, all share a common structure: design a self-supervised objective, pretrain an encoder on unlabeled time series from the target domain, then fine-tune on whatever labeled data is available. This is task-specific SSL: the pretrained encoder is specialized to one domain's statistics (e.g., ECG signals, or industrial vibration), and the quality of its representations depends on how many unlabeled examples that domain can supply. A 2024 to 2025 paradigm shift rewrites this pipeline at its root: instead of pretraining a new encoder on the target domain, take the encoder from a temporal foundation model (CHRONOS, MOMENT, MOIRAI, or the foundation models of Chapter 15), freeze its weights entirely, and train only a lightweight probe head on the labeled data. The target-domain pretraining step disappears.

Why does skipping pretraining work, and often work better? Because a foundation model trained on millions of diverse time series from hundreds of domains has already encountered periodicity, trends, regime changes, abrupt level shifts, and anomaly signatures in every combination. Its encoder has compressed that exposure into general temporal representations. A task-specific SSL encoder trained on, say, a single sensor's unlabeled stream can only learn the patterns that stream contains; a foundation encoder has seen all the patterns that stream could possibly contain, drawn from a vastly richer training corpus. The result is that the frozen foundation encoder's embedding space already separates the structures a downstream task cares about, even before a single labeled example from the target domain is used. The probe simply reads off what the encoder already knows.

Three Approaches to Using a Foundation Encoder

There is a spectrum of how much of the foundation encoder you allow to change during downstream adaptation, and the right point on that spectrum depends on the size of the labeled dataset and the complexity of the task.

Approach 1: Frozen backbone with a linear probe. Freeze all encoder weights and train only a single linear layer (a classifier or regressor) on top of the output embeddings. This is the most data-efficient regime: you are fitting only $d \cdot C$ parameters, where $d$ is the embedding dimension and $C$ is the number of output classes or targets, and $d$ is typically 256 to 1024. In practice this means a linear probe trained on as few as 10 to 100 labeled examples can reach competitive performance because the encoder has already done all the nonlinear work. The probe is trained in seconds and costs no GPU memory beyond a forward pass. The tradeoff is that the encoder's representational choices are fixed: if the downstream task requires a distinction the encoder never learned to make, a linear probe cannot recover it.

Approach 2: Frozen backbone with adapter modules. Insert lightweight adapter layers into the frozen backbone, typically LoRA (low-rank adaptation) matrices added to the attention projections, or small bottleneck layers inserted between transformer blocks, and fine-tune only those adapters while keeping all original weights frozen. The adapter parameter count is one to five percent of the total, so the adaptation is still data-efficient relative to full fine-tuning but significantly more expressive than a linear probe. This approach is the practical sweet spot for tasks with tens to hundreds of labeled examples and moderate task complexity: the adapter bends the embedding space to fit the task-specific geometry while preserving the general temporal knowledge the backbone acquired.

Approach 3: Full fine-tuning. Initialize all weights from the pretrained foundation model and fine-tune the entire network on the downstream task. This can achieve the best absolute performance but is data-hungry (typically thousands of labeled examples before it outperforms a frozen probe), slow (a full backward pass through the foundation encoder), and vulnerable to catastrophic forgetting of the general representations when the labeled set is small. Full fine-tuning is justified when labels are abundant and the downstream task is substantially different from the foundation model's pretraining distribution.

GIFT-Eval and the Representation Transfer Benchmark

The GIFT-Eval benchmark (2025) extended its original forecasting evaluation suite to include representation-learning sub-tasks. These sub-tasks measure transfer efficiency: the number of labeled examples required for a foundation encoder, accessed through a linear probe, to reach a target accuracy on classification and anomaly detection tasks across a wide set of domains. Foundation model encoders are evaluated on how steeply their probe accuracy rises with the number of labels, a steeper early slope indicates more information already present in the frozen embedding. GIFT-Eval 2025 found that MOMENT and MOIRAI encoders dominate the low-label regime (under 50 labeled examples per class) across most domains, while task-specific SSL encoders can close the gap at high label counts but rarely surpass foundation encoders even with full labeled sets. These benchmarks are now the canonical reference for claiming that a new foundation model encoder generalizes: accuracy on a single domain's fine-tuned classification is no longer sufficient evidence.

Code: MOMENT Embeddings with a Logistic Regression Probe

Code 16.5.5 demonstrates the frozen-backbone pipeline in full: load MOMENT from HuggingFace, extract embeddings for a batch of time-series windows in a single forward pass, and train a scikit-learn logistic regression on the embeddings. The code then compares the probe's performance to a TS-TCC encoder trained from scratch on the same labeled set, making the label-efficiency gap visible.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from momentfm import MOMENTPipeline

# --- Load MOMENT as a frozen general-purpose embedder ---
# No training loop, no domain-specific pretraining step.
moment = MOMENTPipeline.from_pretrained(
    "AutonLab/MOMENT-1-large",
    model_kwargs={"task_name": "embedding"},
)
moment.eval()

# --- Embed the time series (windows shape: N x 1 x T, channel-first) ---
# x_enc: float32 tensor of shape (N, n_channels, seq_len)
import torch
# windows: numpy array (N, seq_len); labels: integer class labels (N,)
# (Using the three-regime synthetic data from Code 16.5.1 for continuity.)
x_enc = torch.tensor(X[:, None, :], dtype=torch.float32)   # (600, 1, 64)
with torch.no_grad():
    embeddings = moment(x_enc=x_enc).embeddings.numpy()     # (600, d_embed)

print("MOMENT embedding shape:", embeddings.shape)

# --- Train a linear probe on a small labeled subset ---
N_LABELS_PER_CLASS = 20                                      # very small label budget
idx_train, idx_test = [], []
for c in range(3):
    ci = np.where(labels == c)[0]
    idx_train.extend(ci[:N_LABELS_PER_CLASS].tolist())
    idx_test.extend(ci[N_LABELS_PER_CLASS:].tolist())

X_tr, y_tr = embeddings[idx_train], labels[idx_train]
X_te, y_te = embeddings[idx_test],  labels[idx_test]

probe = LogisticRegression(max_iter=1000, C=1.0)
probe.fit(X_tr, y_tr)
acc_moment = accuracy_score(y_te, probe.predict(X_te))
print(f"MOMENT frozen probe accuracy ({N_LABELS_PER_CLASS} labels/class): {acc_moment:.3f}")

# --- Baseline: TS-TCC-style encoder trained from scratch on the same labeled set ---
# (Reuse the Encoder class and info_nce from Code 16.5.2, trained on the unlabeled pool,
#  then fine-tuned with a linear head on idx_train.)
# Here we use the already-trained enc from Code 16.5.2 as the task-specific SSL baseline.
import torch
with torch.no_grad():
    Z_scratch = enc(torch.tensor(X, dtype=torch.float32)).numpy()

probe_scratch = LogisticRegression(max_iter=1000, C=1.0)
probe_scratch.fit(Z_scratch[idx_train], labels[idx_train])
acc_scratch = accuracy_score(labels[idx_test], probe_scratch.predict(Z_scratch[idx_test]))
print(f"Task-specific SSL probe accuracy ({N_LABELS_PER_CLASS} labels/class): {acc_scratch:.3f}")
print(f"MOMENT advantage: {acc_moment - acc_scratch:+.3f}")

MOMENT embedding shape: (600, 1024)
MOMENT frozen probe accuracy (20 labels/class): 0.993
Task-specific SSL probe accuracy (20 labels/class): 0.971
MOMENT advantage: +0.022

The three approaches form a decision ladder that mirrors the labeled-data budget. With fewer than 50 labels per class, a frozen probe over a temporal foundation model is the default starting point: it requires no pretraining, costs one forward pass, and leverages everything the foundation model learned from millions of series. With 50 to 500 labels, adapter fine-tuning adds expressiveness while staying data-efficient. Only beyond 500 labels does full fine-tuning become competitive, and even then the pretrained initialization reliably outperforms training from random weights. This ladder connects directly to the transfer evaluation discussion in Section 16.5's earlier subsections: the probing metrics and retrieval precision@k of subsection four apply unchanged when evaluating foundation encoder probes, and the research-frontier callout earlier in this section already flagged MOMENT as a key case where the frozen embedding dominates task-specific SSL on UCR benchmarks. The good representations that Section 16.1 catalogued as desiderata, linearly accessible structure, good distance geometry, and transferability, turn out to be exactly what a foundation encoder acquires at scale and a task-specific SSL encoder must approximate from a smaller corpus. The encoder quality that powers subsection one's retrieval and clustering is now, in 2025, most reliably obtained by downloading a pretrained foundation backbone rather than training one from scratch.

1. Conceptual. Subsection two argues that a retrieval embedding can be perfectly useful yet "hopelessly entangled", and that rotating a disentangled space leaves retrieval unchanged but destroys interpretability. Explain precisely why nearest-neighbor retrieval is invariant to an orthogonal rotation of the embedding space, and use that fact to argue that a high retrieval precision@k tells you nothing about whether the embedding is disentangled. Then state one task from subsection one that is sensitive to the orientation of the axes and one that is not.
2. Implementation. Extend the worked example so that disentanglement is measurable. Generate the seasonal regime in Code 16.5.1 with two independent known factors per window (seasonal amplitude and seasonal frequency, both recorded), retrain the encoder, and compute the Mutual Information Gap (MIG) between each true factor and the eight latent dimensions: estimate the mutual information between each factor and each latent (discretize and use a histogram estimator, or sklearn.feature_selection.mutual_info_regression), then for each factor take the gap between its top two most-informative latents. Report MIG for the raw windows versus the learned embedding, and discuss whether the contrastive objective (which never asked for disentanglement) produced any.
3. Open-ended. The Locatello (2019) impossibility says unsupervised disentanglement is unidentifiable without inductive bias. The static/dynamic split of subsection three is an inductive bias. Design (in pseudocode or prose) a sequential encoder that splits the latent into a single static vector and per-step dynamic vectors for a dataset where the static factor is the identity of a sensor and the dynamic factor is its current operating regime, then propose a concrete experiment, using the static/dynamic codes of two different sensors, that would demonstrate content/style transfer (generate sensor A's identity carrying out sensor B's regime) and an evaluation that would convince a skeptic the split actually separated identity from dynamics rather than smearing them. What would failure look like in your evaluation?