"For years they asked me only one question, again and again, on every new series: what comes next. I answered it, they threw me away, and they trained a fresh copy of me on the next series, and another on the one after that, a graveyard of single-use models. Then someone asked me a stranger thing. Not what comes next, but what does this window mean. Pull it close to the windows like it, push it away from the windows unlike it, hide half of it and make me rebuild the rest. I had no labels, only the series and its own structure, and from that alone I learned a vector that held the trend and the season and the regime. After that they stopped retraining me. They froze me, hung a thin linear layer off my embeddings, and I forecast, I classified, I flagged anomalies, all from the same representation. I had finally learned the one thing worth learning: not the answer, but the shape of the question."
A Frozen Encoder Whose Embeddings Outlived Every Task It Was Built For
Chapter Overview
Part IV begins here, and it changes the question the book has been asking. Through Parts II and III the workflow was always the same: take a series, train a model on that series for one task, and read off the answer. Even the foundation models that closed Part III in Chapter 15 were sold to you as forecasters, predictors of what comes next. This chapter steps underneath that framing. Instead of training a fresh model per task, you train an encoder once to turn a raw stretch of series into a reusable vector, a representation, that carries the trend, the seasonality, the regime, and the local shape of the window in a form that many downstream tasks can read. Forecasting, classification, clustering, and anomaly detection then become thin heads attached to one frozen encoder, rather than separate models built from scratch.
The engine that makes this work without labels is self-supervision: the series supplies its own training signal. Two families dominate the chapter. Contrastive methods construct pairs of views, augmented or differently timed slices of the data, and train the encoder to pull together views of the same thing while pushing apart views of different things; TS2Vec learns hierarchical contrastive representations at every timescale, and TF-C aligns a series in the time domain with itself in the frequency domain so the embedding respects both. Masked modeling, the second family, hides part of a series and trains the encoder to reconstruct it, the same masked-reconstruction objective that powers MOMENT and the masked encoders you met in Chapter 15. Both families learn from structure alone, and both produce an embedding space rather than a single prediction.
These representations are precisely the foundation beneath the foundation models. Pretraining a model on a billion series, as Chapter 15 described, is exactly the act of learning a temporal representation at scale; the contrastive and masked objectives of this chapter are the pretext tasks that make that pretraining possible. Read in order, Chapter 15 showed you the destination and Chapter 16 explains from underneath why it works at all: a model transfers to a never-seen series because it learned a representation, not a memorized forecast. The chapter closes on what a good representation should look like beyond raw accuracy, embeddings that are disentangled so that separate axes capture separate factors of variation (trend here, seasonality there, noise elsewhere), which makes the vectors interpretable and controllable rather than an opaque bundle.
This is also a chapter on the temporal thread the book has followed since Part II, now arriving at its learned form. The idea of compressing a series into a few informative coordinates is old: principal component analysis and factor models reduce a panel of series to a handful of latent factors that explain most of their movement, a linear, hand-derived representation. What this chapter does is replace the linear projection with a learned nonlinear encoder and replace the variance-maximizing objective with a self-supervised one, carrying the same ambition (find the few coordinates that matter) into deep, nonlinear, label-free territory. The classical factor becomes the learned embedding, and the rest of Part IV builds on the vectors you learn to make here.
Prerequisites
This chapter assumes the neural toolkit of Part III and the feature thinking of Part I. From Chapter 12: Attention and Transformers you need the Transformer encoder, patching, and above all the pretraining-then-transfer paradigm, because a temporal representation learner is an encoder pretrained with a self-supervised objective and then transferred, and the masked-modeling section is that paradigm applied to series. From Chapter 15: Temporal Foundation Models you need the pretrain-once-then-apply mindset and the masked-reconstruction objective behind MOMENT, since this chapter is the representation-learning machinery that those models rest on, explained from the ground up. From Chapter 2: Temporal Data Engineering you need hand-engineered temporal features and the augmentations (jitter, scaling, window slicing, masking) that contrastive methods turn into positive views, because a learned representation is the successor to those manual features and a careful split prevents leakage between pretext and probe. A comfort with embeddings and similarity in a vector space, and with the linear factor models of Part II, will help. Readers wanting a refresher on transformers, on temporal features and augmentation, or on factor and PCA reductions will find each referenced chapter and the appendices indexed in the Table of Contents.
If you keep one idea from this chapter, keep this: a temporal representation is a learned, reusable vector encoding of a stretch of series, trained without labels by a self-supervised pretext task (contrastive or masked), so that one frozen encoder serves many downstream tasks instead of a fresh model per task. Contrastive methods (TS2Vec, TF-C) pull together views of the same window and push apart views of different windows; masked modeling hides part of the series and reconstructs it; both learn from the series' own structure. The payoff is transfer, the same mechanism that makes the foundation models of Chapter 15 work, and the goal beyond accuracy is a disentangled embedding whose separate axes capture separate factors, the learned successor to the classical factor and PCA reductions of Part II.
Chapter Roadmap
- 16.1 Representation Learning Fundamentals What a representation is and why learning one beats training a fresh model per task: the encoder-and-head decomposition, transfer and linear probing, what makes a representation good, and the line from classical PCA and factor reductions to learned nonlinear temporal encoders.
- 16.2 Self-Supervised Pretext Tasks for Time Series How a series supplies its own labels: the pretext-task idea, the augmentation families (jitter, scaling, permutation, window slicing, masking) that create supervision from structure alone, and the taxonomy of contrastive versus generative-masked objectives that organizes the rest of the chapter.
- 16.3 Contrastive Learning (TS2Vec, TF-C, and variants) The contrastive family in depth: pulling positive views together and pushing negatives apart with the InfoNCE objective, TS2Vec's hierarchical contrasting across timescales, TF-C's time-frequency consistency, TS-TCC and triplet-loss variants, and the role of temporal neighborhoods as positives.
- 16.4 Masked Modeling for Time Series The generative-masked family: hide patches of a series and reconstruct them, the masked-autoencoder recipe carried from vision and language into temporal data, the design choices of mask ratio and patch size, and why this objective underlies MOMENT and the masked foundation encoders of Chapter 15.
- 16.5 Temporal Embeddings and Disentanglement What a good embedding looks like beyond accuracy: visualizing and probing the learned space, disentanglement so separate axes capture trend, seasonality, and noise, the beta-VAE route to factorized latents, and how interpretable, controllable embeddings set up the generative and event models of the rest of Part IV.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single experiment. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have built an encoder per Section 16.1, pretrained it with the masked objective of Section 16.4 and the contrastive objective of Section 16.3 using the pretext machinery of Section 16.2, and probed and inspected the resulting embeddings as Section 16.5 describes.
Hands-On Lab: Pretrain Once, Probe Everywhere
Objective
Prove to yourself the central promise of the chapter: that one encoder, pretrained once without labels, can serve many tasks better than a model trained from scratch for each. You will take an unlabeled corpus of series and pretrain a single encoder two ways, once with the masked-reconstruction objective of Section 16.4 and once with the contrastive objective of Section 16.3, using the augmentations and pretext machinery of Section 16.2 and the encoder-and-head decomposition of Section 16.1. You will then freeze each encoder and attach only a thin linear probe, fitting it for two downstream tasks in turn, forecasting and classification, so the representation does the work and the probe merely reads it. Against those you will run a supervised-from-scratch model trained end-to-end on each task with the same labels. The headline of the lab is the comparison: how close, or how far ahead, does a frozen self-supervised encoder with a linear probe land relative to a fully supervised model, and does contrastive or masked pretraining win on which task. You will finish by visualizing the learned embedding space, the inspection of Section 16.5, to see what the encoder actually captured.
What You'll Practice
- Building an encoder and separating it from task heads, the decomposition of Section 16.1.
- Constructing self-supervised pretext tasks and augmented views from unlabeled series, following Section 16.2.
- Pretraining an encoder contrastively with an InfoNCE-style objective, following Section 16.3.
- Pretraining an encoder by masked reconstruction, following Section 16.4.
- Linear-probing a frozen encoder for forecasting and classification and comparing to supervised-from-scratch.
- Visualizing and probing the learned embedding space for disentanglement, following Section 16.5.
Setup
You need an unlabeled corpus of series for pretraining and a smaller labeled set for the downstream probes; a Monash or UCR archive subset works well and ships many series at once. You need PyTorch for the encoder and the two pretraining objectives, and the tsai library, which packages TS2Vec, TS-TCC, and masked-modeling recipes so you can cross-check your from-scratch encoder against a reference implementation. Keep the pretraining corpus and the probe set disjoint, the split discipline of Chapter 2 applied to self-supervision, so the probe measures transfer and not memorization. The code below sketches the pretrain-then-freeze-then-probe skeleton the lab fills in: one encoder, a self-supervised loss to train it, then a frozen forward pass feeding a thin linear head.
encoder = TemporalEncoder() # the reusable backbone (Section 16.1)
# Pretrain once, no labels: masked reconstruction OR contrastive (Sections 16.3, 16.4)
for batch in unlabeled_loader:
loss = ssl_objective(encoder, augment(batch)) # InfoNCE or masked MSE
loss.backward(); opt.step(); opt.zero_grad()
encoder.eval() # freeze: the representation is now fixed
for p in encoder.parameters():
p.requires_grad = False
z = encoder(x_labeled).detach() # embeddings feed a thin linear probe per task
probe = LinearProbe(z.shape[-1], n_outputs) # forecasting head or classification head
Steps
Step 1: Pretrain an encoder by masked reconstruction
Mask patches of each unlabeled series and train the encoder to reconstruct them, the masked-modeling recipe of Section 16.4, sweeping the mask ratio briefly. Save the trained encoder weights; no labels are used anywhere in this step.
Step 2: Pretrain a second encoder contrastively
From the same corpus, build augmented positive views and contrast them against negatives with an InfoNCE-style loss, the TS2Vec and TF-C objective of Section 16.3, using the augmentations of Section 16.2. Save this second encoder so you have two representations to compare.
Step 3: Freeze and linear-probe both encoders
Freeze each encoder and fit only a linear head on its embeddings, once for a forecasting target and once for a classification target, the encoder-and-head split of Section 16.1. The encoder never updates here, so the probe measures the quality of the representation alone.
Step 4: Train a supervised-from-scratch baseline
For each downstream task, train an end-to-end supervised model on the same labels with no pretraining, the measuring stick. This is the contestant the frozen representation must approach or beat, and it gets the full supervised budget.
Step 5: Compare and visualize the embedding space
Tabulate frozen-probe versus supervised-from-scratch across both tasks and both pretraining objectives, then project the learned embeddings to two dimensions and inspect whether trend, season, and class separate cleanly, the disentanglement check of Section 16.5. Read off which objective transfers better to which task.
Expected Output
The lab produces one comparison table and one embedding picture. The table shows the frozen self-supervised encoder with a thin linear probe landing close to, and on the label-scarce setting often ahead of, the supervised-from-scratch model, the central promise of the chapter made concrete: a representation trained once and reused beats a fresh model per task when labels are limited. You will typically see contrastive pretraining favor the classification probe, where instance-level similarity matters, and masked pretraining favor the forecasting probe, where local reconstruction matters, though the exact split depends on the corpus and is itself a finding worth recording. The embedding visualization is the second deliverable: a two-dimensional projection in which series of the same class or regime cluster and the axes of variation begin to separate trend from season, the disentanglement that Section 16.5 argues a good representation should show. The reader finishes able to pretrain an encoder on unlabeled series, probe it for any new task in a few lines, and judge a representation by both its downstream accuracy and the structure of its embedding space.
The contrastive and masked recipes that this chapter derives from scratch, the augmentations, the InfoNCE loss, the hierarchical contrasting, the masked-reconstruction loop, are packaged in the tsai library as ready callbacks and model wrappers, so pretraining a TS2Vec or masked encoder and probing it drops from a few hundred lines to a handful: instantiate the model, attach the self-supervised callback, fit on the unlabeled loader, then fit a linear head on the frozen embeddings. The library handles the view construction, the loss, and the freeze-and-probe plumbing internally, and the GluonTS and HuggingFace ecosystems expose the pretrained MOMENT and masked encoders the same way. The chapter's discipline still holds: the few-line call is easy, but the judgment, keeping the pretraining and probe sets disjoint, choosing the objective that matches the downstream task, and reading the embedding space for what it actually learned, is the work the library cannot do for you. The shortcut makes the encoder cheap to train; it does not tell you whether the representation is any good.
Stretch Goals
- Add the TF-C time-frequency objective of Section 16.3 as a third encoder and test whether aligning time and frequency views yields a representation that transfers better than time-domain contrastive alone.
- Probe a pretrained foundation encoder, the MOMENT model of Chapter 15, on the same downstream tasks and compare its frozen embeddings to your own, seeing how far large-corpus pretraining pulls ahead.
- Train a beta-VAE on the same series and measure disentanglement explicitly, the route of Section 16.5, then test whether the more factorized latents probe as well as the contrastive ones.
What's Next?
This chapter opened Part IV: Temporal Representation Learning by teaching an encoder to turn a raw series into a reusable vector without labels, through contrastive and masked self-supervision, and to shape that vector toward disentangled, interpretable axes. Those representations describe a series; the next step is to generate one. Chapter 17: Generative Temporal Models takes the latent spaces you just learned to read and learns to sample from them, building models that synthesize realistic series: variational autoencoders and the latent variables that connect directly to the disentangled embeddings of Section 16.5, generative adversarial and diffusion models for time series, and the latent-variable view that returns the hidden Markov model and the Kalman filter of Part II in deep, learned form. Where this chapter asked what a stretch of series means, the next asks how to conjure new series that share that meaning, and the embeddings you learned here are the space those generators will sample from. The representations of Chapter 16 are the foundation; the generators of Chapter 17 are the first thing built on top.
Bibliography & Further Reading
Foundations of Representation Learning
Bengio, Y., Courville, A., Vincent, P. "Representation Learning: A Review and New Perspectives." IEEE TPAMI, 2013. arXiv:1206.5538. arxiv.org/abs/1206.5538
The foundational review of representation learning, defining what a good representation is and why learning one beats hand-engineering features, the conceptual spine of Section 16.1.
van den Oord, A., Li, Y., Vinyals, O. "Representation Learning with Contrastive Predictive Coding." 2018. arXiv:1807.03748. arxiv.org/abs/1807.03748
Contrastive Predictive Coding, which introduced the InfoNCE objective for learning representations by predicting future latents, the contrastive foundation underneath Section 16.3.
Chen, T., Kornblith, S., Norouzi, M., Hinton, G. "A Simple Framework for Contrastive Learning of Visual Representations (SimCLR)." ICML, 2020. arXiv:2002.05709. arxiv.org/abs/2002.05709
SimCLR, the augment-and-contrast recipe from vision that established the modern contrastive template the time-series methods of Section 16.3 adapt.
Contrastive Time-Series Representations
Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., Xu, B. "TS2Vec: Towards Universal Representation of Time Series." AAAI, 2022. arXiv:2106.10466. arxiv.org/abs/2106.10466
TS2Vec, hierarchical contrastive learning across timescales producing a universal representation usable for forecasting, classification, and anomaly detection, the central contrastive method of Section 16.3 and the lab.
Zhang, X., Zhao, Z., Tsiligkaridis, T., Zitnik, M. "Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency (TF-C)." NeurIPS, 2022. arXiv:2206.08496. arxiv.org/abs/2206.08496
TF-C, aligning a series in the time domain with itself in the frequency domain so the embedding respects both, the time-frequency consistency method of Section 16.3 and a lab stretch goal.
Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., Li, X., Guan, C. "Time-Series Representation Learning via Temporal and Contextual Contrasting (TS-TCC)." IJCAI, 2021. arXiv:2106.14112. arxiv.org/abs/2106.14112
TS-TCC, combining temporal and contextual contrasting with strong and weak augmentations, a key contrastive variant of Section 16.3 and a reference implementation in the lab.
Franceschi, J.-Y., Dieuleveut, A., Jaggi, M. "Unsupervised Scalable Representation Learning for Multivariate Time Series." NeurIPS, 2019. arXiv:1901.10738. arxiv.org/abs/1901.10738
An early triplet-loss approach using time-based negative sampling to learn unsupervised series representations, the triplet-loss variant cited in Section 16.3.
Tonekaboni, S., Eytan, D., Goldenberg, A. "Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding (TNC)." ICLR, 2021. arXiv:2106.00750. arxiv.org/abs/2106.00750
TNC, using temporal neighborhoods as positives to learn representations that respect the non-stationarity of clinical series, the neighborhood-as-positive idea of Section 16.3.
Masked Modeling
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., Girshick, R. "Masked Autoencoders Are Scalable Vision Learners (MAE)." CVPR, 2022. arXiv:2111.06377. arxiv.org/abs/2111.06377
The masked autoencoder, hiding a large fraction of patches and reconstructing them, the recipe Section 16.4 carries from vision into time series.
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., Dubrawski, A. "MOMENT: A Family of Open Time-Series Foundation Models." ICML, 2024. arXiv:2402.03885. arxiv.org/abs/2402.03885
MOMENT, a masked-reconstruction encoder pretrained for forecasting, classification, and anomaly detection, the time-series realization of masked modeling in Section 16.4 and a lab stretch goal.
Embeddings and Disentanglement
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A. "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework." ICLR, 2017. openreview.net/forum?id=Sy2fzU9gl
beta-VAE, the constrained variational objective that pressures latents toward factorized, disentangled axes, the route to interpretable embeddings in Section 16.5 and a lab stretch goal.
Tools
Oguiza, I. "tsai: State-of-the-Art Deep Learning for Time Series and Sequences." GitHub / timeseriesAI. timeseriesAI.github.io/tsai
The tsai library, packaging TS2Vec, TS-TCC, and masked-modeling recipes as ready callbacks and model wrappers, the toolkit behind the pretrain-and-probe shortcut in the lab.