"For years they only asked me what comes next, and I answered with one number and a polite shrug of uncertainty. Then someone asked the harder question: not what comes next, but what could have happened at all. Show me a thousand plausible mornings for this patient, a year of trading days that never traded, a sensor trace from a machine that was never built. So I stopped predicting a point and started learning the whole distribution of sequences, the shapes a series is allowed to take and the ones it never does. Now I dream in time. I produce histories that pass for real, fill the gaps where the data went missing, and flag the genuinely strange by noticing it is something I would never have dreamed. I am told this is generation. It feels more like remembering things that did not happen, carefully enough that you cannot tell."
A Generative Model Dreaming in Plausible Histories
Chapter Overview
Chapter 16 taught a network to encode a stretch of series into a representation, a compact vector that captures what the window means without ever predicting its future. This chapter turns that machinery around and asks it to generate. Once a model has learned the structure of a series well enough to compress it, the same understanding can run in reverse: sample from the learned structure and out comes a new series that never existed but looks like it could have. The shift from encoding to generating is the shift from "what does this mean" to "what else could the data have been", and it unlocks a set of capabilities that point forecasting alone cannot reach. The object of study is no longer a single best next value but the full probability distribution over sequences, $p(x_{1:T})$, and the four model families of this chapter are four different ways to learn that distribution and draw from it.
The chapter takes the four families in the order they reached time series. The autoencoder learns to reconstruct a sequence through a bottleneck, and a sequence autoencoder does the same for variable-length series with a recurrent or convolutional encoder and decoder; it is the workhorse for compression and the foundation the rest build on, though a plain autoencoder cannot truly sample. The variational autoencoder fixes that by placing a probability distribution over the latent space, turning the bottleneck into something you can sample from, and its sequential variants (the variational RNN and the deep Markov model) marry the VAE to a temporal latent process, making them the natural choice when you want a probabilistic latent state and tractable likelihoods. The generative adversarial network learns by contest rather than reconstruction, pitting a generator against a discriminator, and TimeGAN and its relatives adapt the idea to capture temporal dynamics, excelling at sharp, realistic samples when you care more about fidelity than likelihood. Diffusion models, the newest arrival, learn to reverse a gradual noising process and have become the strongest family for high-fidelity temporal synthesis, imputation, and probabilistic forecasting, with TimeGrad and CSDI as the landmark time-series instances.
These models earn their place through four practical uses that recur across the chapter. They generate synthetic data to augment scarce training sets or to share a realistic dataset without exposing the private records behind it. They perform imputation, filling the missing entries that plague real clinical and sensor series by conditioning the generative model on the observed values and sampling the rest. They deliver probabilistic forecasting, because a conditional generative model that produces whole future trajectories gives you a full predictive distribution rather than a single line, connecting this chapter directly to the uncertainty work of Part V. And they enable anomaly detection by likelihood: a point the model assigns low probability, or one it could never reconstruct or generate, is by definition something the normal data never taught it, the generative complement to the detectors of Chapter 8.
The temporal thread runs straight through this chapter, because every family here is a learned descendant of a classical generative model you already met. The hidden Markov model of Chapter 7, with its latent state evolving in time and emitting observations, returns as the sequential VAE and the deep Markov model, the same graphical structure with neural transition and emission functions and amortized variational inference doing the work the forward-backward algorithm once did. The denoising idea that underlies state-space smoothing returns, scaled up and iterated hundreds of times, as the diffusion model. One note on the field's trajectory is worth setting down now: GANs dominated temporal generation in the late 2010s, but training instability and mode collapse have steadily ceded ground to diffusion, which by the mid-2020s is the default first reach for high-fidelity time-series synthesis and the family this chapter spends the most care on.
Prerequisites
This chapter builds directly on the representation learning that precedes it and on the latent-state models of Part II. From Chapter 16: Learning Temporal Representations you need the idea of an encoder that maps a sequence to a latent vector and the self-supervised objectives that train one, because every generative model here is an encoder paired with a way to sample from or invert its latent space, and the representations that chapter learned are exactly what these models put a distribution over. From Chapter 7: State-Space Models and Filtering you need the latent-state view of a sequence, a hidden state that evolves in time and generates the observations, along with the hidden Markov model specifically, because the sequential VAE and the deep Markov model are that exact structure with neural networks in place of the linear-Gaussian or tabular transitions, and the inference they perform is the learned cousin of filtering and smoothing. You also need basic probability, the language the whole chapter speaks: a probability distribution over data, the difference between a likelihood and a sample, conditional distributions for imputation and forecasting, and the Kullback-Leibler divergence that the variational objective minimizes. Readers wanting a refresher on probability distributions and KL divergence, on latent-variable models, or on the encoder-decoder vocabulary will find the relevant chapters and the probability appendix indexed in the Table of Contents.
If you keep one idea from this chapter, keep this: a generative temporal model learns the full distribution over sequences $p(x_{1:T})$ rather than a single best forecast, and sampling from that distribution is what powers synthetic data, imputation, probabilistic forecasting, and likelihood-based anomaly detection. The four families differ in how they learn and sample that distribution: the autoencoder reconstructs through a bottleneck, the variational autoencoder puts a sampleable distribution on the latent space and its sequential variants give the latent a temporal dynamics that is the hidden Markov model reborn, the GAN learns by a generator-versus-discriminator contest and produces sharp samples, and the diffusion model learns to reverse a noising process and now leads the field for high-fidelity temporal synthesis. Choose by what you need: likelihoods and a probabilistic latent state favor the VAE family, raw sample fidelity once favored GANs, and diffusion has become the default first reach for both.
Chapter Roadmap
- 17.1 Autoencoders and Sequence Autoencoders The reconstruction-through-a-bottleneck idea and its sequence form: a recurrent or convolutional encoder compressing a variable-length series to a latent vector and a decoder rebuilding it, the foundation the generative families build on and the source of reconstruction-error anomaly scores, with the honest caveat that a plain autoencoder learns a code but not a distribution you can sample.
- 17.2 Variational Autoencoders and Sequential VAEs Turning the bottleneck into a probability distribution: the VAE's latent prior, the evidence lower bound, and the reparameterization trick, then the sequential variants (the variational RNN and the deep Markov model) that give the latent a temporal dynamics, the same hidden Markov model structure of Chapter 7 with neural transitions and amortized inference.
- 17.3 GANs for Time Series (TimeGAN and beyond) Generation by contest rather than reconstruction: the generator-discriminator adversarial game, why naive GANs miss temporal dynamics, and how TimeGAN, RCGAN, and DoppelGANger add a supervised or recurrent structure to capture step-to-step dependence, along with the training instabilities (mode collapse, non-convergence) that pushed the field toward diffusion.
- 17.4 Diffusion Models for Time Series (TimeGrad, CSDI) The family that learns to reverse a gradual noising process: the forward and reverse diffusion chains and the denoising objective, then their time-series landmarks, TimeGrad for autoregressive probabilistic forecasting, CSDI for conditional score-based imputation and forecasting, and SSSD using a structured state-space backbone, the current default for high-fidelity temporal synthesis.
- 17.5 Synthetic Temporal Data Generation and Evaluation How to use generated series and how to know whether they are any good: augmentation and privacy-preserving sharing as the why, then the evaluation triad of statistical fidelity (autocorrelation and spectral match), downstream utility (train on synthetic, test on real), and a privacy check against memorization, because a generative model is only as trustworthy as the audit that judges its samples.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single experiment. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have trained a sequential VAE from Section 17.2 and a diffusion model from Section 17.4, generated synthetic series as Section 17.1 frames the bottleneck, and judged the samples with the full evaluation triad of Section 17.5.
Hands-On Lab: Generate and Judge
Objective
Build two generative models for the same series, draw synthetic samples from each, and then do the part the field too often skips: judge whether the samples are any good. You will train a sequential VAE, the probabilistic-latent model of Section 17.2, and a diffusion model, the high-fidelity family of Section 17.4, on a single real series, treating each as an autoencoder-style generator in the sense of Section 17.1. You will then generate a batch of synthetic series from each model and put them through the three-part audit of Section 17.5: a statistical-fidelity check that compares the autocorrelation function and the power spectrum of synthetic against real, a downstream-utility score that trains a forecaster on the synthetic data and tests it on the real held-out data (the train-on-synthetic-test-on-real protocol), and a privacy check that confirms the model invented new series rather than memorizing and copying its training examples. The headline of the lab is the verdict: which model produces more faithful, more useful, and more private samples, and what each of the three axes reveals that the others miss.
What You'll Practice
- Training a sequence autoencoder and reading its reconstruction error, following Section 17.1.
- Training a sequential VAE and sampling synthetic series from its latent prior, following Section 17.2.
- Training a diffusion model and generating samples by running its reverse denoising chain, following Section 17.4.
- Measuring statistical fidelity by comparing autocorrelation and spectral content of synthetic against real, following Section 17.5.
- Computing a train-on-synthetic-test-on-real utility score to judge whether the synthetic data is usable downstream, following Section 17.5.
- Running a privacy check against memorization to confirm the samples are genuinely new, following Section 17.5.
Setup
You need PyTorch for both models, a single real univariate or low-dimensional multivariate series long enough to slice into many training windows (a finance returns series or a sensor trace works well), and the statsmodels and scipy.signal utilities for the autocorrelation and spectral comparisons of Section 17.5. For the diffusion model the CSDI reference implementation or a small custom denoising network is enough; for the VAE a recurrent encoder and decoder built in a few dozen lines suffice. The downstream-utility step needs any simple forecaster (a small recurrent model or a gradient-boosted regressor) trained twice, once on real and once on synthetic, and tested on the same real held-out window. The privacy check needs a nearest-neighbor distance from each synthetic sample to its closest training window, so that suspiciously tiny distances flag memorization. The code below sketches the generate-and-judge loop: sample from each trained model and score the samples on the three axes.
import numpy as np
from scipy.signal import welch
# vae and diffusion are two trained generative models (Sections 17.2 and 17.4)
synth_vae = vae.sample(n=512) # draw synthetic series from the latent prior
synth_diff = diffusion.sample(n=512) # draw synthetic series via the reverse denoising chain
def fidelity(real, synth): # statistical-fidelity axis (Section 17.5)
acf_gap = np.abs(acf(real) - acf(synth.mean(0))).mean() # autocorrelation match
_, psd_r = welch(real); _, psd_s = welch(synth.mean(0)) # spectral match
return acf_gap, np.abs(psd_r - psd_s).mean()
util = tstr_score(synth_diff, real_test) # train-on-synthetic, test-on-real utility
leak = nn_distance(synth_diff, train_set) # privacy: distance to nearest training window
Steps
Step 1: Train a sequential VAE on the series
Slice the series into overlapping windows and train a sequential VAE, the probabilistic-latent model of Section 17.2, to reconstruct them while regularizing the latent toward its prior. Confirm the evidence lower bound is rising and the reconstructions track the inputs, the sign that the latent has learned the series structure rather than collapsing.
Step 2: Train a diffusion model on the same windows
Train a diffusion model, the reverse-noising family of Section 17.4, on the identical window set so the two models are judged on equal footing. Watch the denoising loss fall and sanity-check a few reverse-chain samples by eye before trusting the quantitative scores.
Step 3: Generate synthetic batches from both
Draw a large batch of synthetic series from each model, sampling the VAE from its latent prior and the diffusion model by running its reverse chain, exactly the sample step the generative bottleneck of Section 17.1 makes possible. These two batches are the material the rest of the lab judges.
Step 4: Score statistical fidelity
Compare the autocorrelation function and the power spectrum of each synthetic batch against the real series, the fidelity axis of Section 17.5. A model that matches the marginal distribution but misses the autocorrelation has learned the values without the temporal structure, the most common failure of naive temporal generators.
Step 5: Score downstream utility and privacy
Train a forecaster on each synthetic batch and test it on the real held-out window, the train-on-synthetic-test-on-real utility score of Section 17.5, then run the nearest-neighbor privacy check to confirm the samples are not near-copies of the training windows. A model can score well on fidelity yet fail utility, or pass both yet fail privacy by memorizing, which is why all three axes are needed to reach the verdict.
Expected Output
The lab produces a three-by-two scorecard, three evaluation axes against two models, and a verdict read from it. In the common case the diffusion model wins on statistical fidelity, its autocorrelation and spectrum tracking the real series more closely than the VAE's, while the VAE trains faster and offers tractable likelihoods the diffusion model does not, the practical tradeoff that decides which family to reach for. The downstream-utility score reveals whether either model's samples are good enough to train a usable forecaster, the test that matters most when the point of generation was data augmentation, and it is sobering how often samples that look plausible to the eye still degrade a downstream model. The privacy check closes the audit by confirming the samples are genuinely new rather than memorized copies, the property that makes synthetic data safe to share. The reader finishes able to train either family, generate from it, and, most importantly, judge the output on fidelity, utility, and privacy together rather than trusting a generative model on the strength of samples that merely look right.
The full train-generate-and-judge pipeline of this lab, dozens of lines of model code plus the three-axis audit, is largely packaged in modern synthetic-data libraries. The Synthetic Data Vault (SDV) ecosystem ships sequential synthesizers (PAR and related models) that fit a temporal generator and sample new series in a handful of calls, and its companion SDMetrics computes a battery of fidelity, utility, and privacy diagnostics, including autocorrelation comparisons and memorization checks, that would otherwise be written by hand. The discipline of the chapter still holds: the library makes generating and scoring a few lines, but it does not choose the right metric for your use case or interpret a passing fidelity score against a failing utility score. Reach for SDV and SDMetrics to avoid reimplementing the plumbing, then spend your effort where it matters, on reading the three axes together and deciding whether the synthetic data is fit for the purpose you generated it for.
Stretch Goals
- Add a GAN-based generator, the TimeGAN of Section 17.3, to the comparison and see where the adversarial family lands against the VAE and diffusion on the same three axes, including whether it shows the mode collapse the section warns about.
- Recast the diffusion model as an imputer, the CSDI setting of Section 17.4, by masking part of each series and sampling the missing values conditioned on the observed ones, then score the imputation against the held-out truth.
- Use the trained generative model as a likelihood-based anomaly detector by flagging real windows it assigns low probability or cannot reconstruct, connecting back to the detectors of Chapter 8.
What's Next?
This chapter learned to model the distribution over whole sequences and to sample from it, treating a series as a single object drawn from a population of plausible series. Chapter 18: Event and Sequence Modeling narrows the lens to a different kind of temporal data, sequences not of regularly-sampled values but of discrete events scattered in continuous time: a patient's irregular clinical visits, a user's clicks, a machine's fault logs. It introduces temporal point processes and their neural forms, the models that ask not what value comes next but when the next event will happen and what kind it will be, and it brings in time-to-event and survival modeling and temporal graphs. The generative thread carries straight over, because a point process is itself a generative model of event sequences, and the latent-variable and likelihood ideas of this chapter are exactly the tools Chapter 18 uses to learn when things happen. From the distribution over sequences you arrive at the distribution over events in time.
Bibliography & Further Reading
Autoencoders and Variational Foundations
Kingma, D. P., Welling, M. "Auto-Encoding Variational Bayes." ICLR, 2014. arXiv:1312.6114. arxiv.org/abs/1312.6114
The paper introducing the variational autoencoder, the evidence lower bound, and the reparameterization trick, the foundation of the entire latent-variable family in Section 17.2.
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A., Bengio, Y. "A Recurrent Latent Variable Model for Sequential Data." NeurIPS, 2015. arXiv:1506.02216. arxiv.org/abs/1506.02216
The variational RNN (VRNN), which marries the VAE to a recurrent network so each step has its own latent variable, the canonical sequential VAE of Section 17.2.
Krishnan, R. G., Shalit, U., Sontag, D. "Deep Kalman Filters." 2015. arXiv:1511.05121. arxiv.org/abs/1511.05121
The deep Kalman filter, a neural latent state-space model that is the hidden Markov model of Chapter 7 with learned transitions, a direct realization of the temporal thread in Section 17.2.
Krishnan, R. G., Shalit, U., Sontag, D. "Structured Inference Networks for Nonlinear State Space Models." AAAI, 2017. arXiv:1609.09869. arxiv.org/abs/1609.09869
The deep Markov model and its structured amortized inference network, the learned cousin of filtering and smoothing that Section 17.2 builds on the HMM structure of Chapter 7.
GANs for Time Series
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y. "Generative Adversarial Nets." NeurIPS, 2014. arXiv:1406.2661. arxiv.org/abs/1406.2661
The paper that introduced the generator-versus-discriminator adversarial game, the foundation of the GAN family whose temporal adaptations open Section 17.3.
Esteban, C., Hyland, S. L., Ratsch, G. "Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs." 2017. arXiv:1706.02633. arxiv.org/abs/1706.02633
RCGAN, a recurrent conditional GAN for generating realistic medical time series, an early temporal GAN and a case study in privacy-preserving synthetic clinical data for Section 17.3.
Yoon, J., Jarrett, D., van der Schaar, M. "Time-series Generative Adversarial Networks." NeurIPS, 2019. papers.nips.cc/paper/2019
TimeGAN, which adds a supervised loss and a learned embedding so the adversarial game captures step-to-step temporal dynamics, the central GAN of Section 17.3.
Lin, Z., Jain, A., Wang, C., Fanti, G., Sekar, V. "Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions (DoppelGANger)." IMC, 2020. arXiv:1909.13403. arxiv.org/abs/1909.13403
DoppelGANger, a GAN designed for long, high-fidelity networked time series with metadata, the scaling-and-sharing case of Section 17.3.
Diffusion Models for Time Series
Ho, J., Jain, A., Abbeel, P. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020. arXiv:2006.11239. arxiv.org/abs/2006.11239
The DDPM paper that established the modern forward-and-reverse diffusion formulation and the denoising objective, the foundation of every diffusion model in Section 17.4.
Rasul, K., Seward, C., Schuster, I., Vollgraf, R. "Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting (TimeGrad)." ICML, 2021. arXiv:2101.12072. arxiv.org/abs/2101.12072
TimeGrad, the first prominent diffusion model for probabilistic forecasting, denoising each future step conditioned on a recurrent hidden state, the forecasting landmark of Section 17.4.
Tashiro, Y., Song, J., Song, Y., Ermon, S. "CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation." NeurIPS, 2021. arXiv:2107.03502. arxiv.org/abs/2107.03502
CSDI, a conditional score-based diffusion model for imputation and forecasting that conditions on observed values to sample the missing ones, the imputation landmark of Section 17.4 and the lab's diffusion contestant.
Alcaraz, J. M. L., Strodthoff, N. "Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models (SSSD)." TMLR, 2023. arXiv:2208.09399. arxiv.org/abs/2208.09399
SSSD, combining a diffusion model with a structured state-space backbone for long-range imputation and forecasting, linking Section 17.4 to the state-space models of Chapter 13.
Synthetic Data and Evaluation Tools
DataCebo. "SDV and SDMetrics: The Synthetic Data Vault, with sequential synthesizers and fidelity, utility, and privacy diagnostics." Documentation. docs.sdv.dev
The Synthetic Data Vault ecosystem, supplying sequential generators and the SDMetrics diagnostics for the statistical-fidelity, utility, and privacy audit of Section 17.5 and the lab's library shortcut.