Autoencoders and Sequence Autoencoders

"They gave me five hundred numbers and one shelf with room for eight. So I throw away the noise, keep the gist, and on a good day I hand back something that looks almost exactly like what arrived. On a bad day I am the only one in the building who notices that this particular Tuesday does not look like any Tuesday I was ever trained on."

A Bottleneck Layer Forced to Summarize a Whole Sequence

In Chapter 16 we learned temporal representations by self-supervision and contrast: define a pretext task or a notion of similarity, and the embedding falls out as a byproduct. This chapter turns to a complementary family, the generative models, which learn representations by modeling how the data itself is produced, and the autoencoder is the gateway. It is the most direct expression of an idea that runs through all of representation learning: a representation is good if you can rebuild the data from it. We use the unified notation of Appendix A throughout: $\mathbf{x}$ for an input (a vector or a whole sequence $\mathbf{x}_{1:T}$), $\mathbf{z}$ for the latent or bottleneck code, $\hat{\mathbf{x}}$ for the reconstruction, and $\mathcal{L}$ for the loss.

Why open the generative-models chapter with the autoencoder rather than diving straight to the variational autoencoder, the diffusion model, or the autoregressive Transformer that fill the rest of it? Because the autoencoder isolates, in its purest form, the single mechanism those richer models elaborate: compress to a latent, expand from a latent, and measure the round trip. Understand the bottleneck and the reconstruction loss here, with no probabilistic machinery in the way, and every model later in the chapter reads as the autoencoder plus one new ingredient: a prior and a sampling step for the VAE (Section 17.2), an adversarial critic for the temporal GAN (Section 17.3), a denoising chain for the diffusion model (Section 17.4). The autoencoder is the load-bearing primitive, and this section makes it concrete for sequences.

The competencies this section installs are four. First, to state precisely what an autoencoder is (encoder, bottleneck, decoder, reconstruction loss) and why the bottleneck width is the central design knob. Second, to build a sequence autoencoder from an RNN, TCN, or Transformer encoder-decoder and train it to reconstruct a series. Third, to put the trained model to work three ways, reconstruction error for anomaly, the latent for retrieval, the encoder for pretraining. Fourth, to articulate the limits of a plain autoencoder for generation, the missing prior and the holes in latent space, which motivate the variational autoencoder. These skills carry directly into the rest of Part IV.

1. The Autoencoder: Encode to a Bottleneck, Decode to Reconstruct Beginner

An autoencoder is a pair of networks trained jointly to reconstruct their own input. The encoder $f_\theta$ maps an input $\mathbf{x}$ to a latent code $\mathbf{z} = f_\theta(\mathbf{x})$; the decoder $g_\phi$ maps that code back to a reconstruction $\hat{\mathbf{x}} = g_\phi(\mathbf{z})$. The two are trained together to make $\hat{\mathbf{x}}$ as close to $\mathbf{x}$ as possible, by minimizing a reconstruction loss averaged over the data. For real-valued data the natural choice is squared error,

and for inputs in $[0,1]$ (normalized intensities, probabilities) a binary cross-entropy per dimension is often used instead. Nothing in this objective references a label: the target is the input, which is why the autoencoder is the canonical self-supervised, unsupervised representation learner. The signal that drives learning is entirely the discrepancy between what went in and what came back out.

The objective as written has a trivial solution: if $f_\theta$ and $g_\phi$ are both the identity and $\mathbf{z}$ is as wide as $\mathbf{x}$, the network copies its input perfectly and learns nothing. The autoencoder becomes useful only when we forbid that shortcut, and the classic way to forbid it is the bottleneck: make the latent $\mathbf{z}$ have far fewer dimensions than $\mathbf{x}$. Now the network cannot copy, it must compress. To reconstruct a $D$-dimensional input from a $k$-dimensional code with $k \ll D$, the encoder has to find the few directions of variation that actually matter in the data and throw away the rest, and the decoder has to learn to expand those few numbers back into a plausible full input. The bottleneck code is therefore a lossy compressed representation: a learned summary that keeps the structure and drops the redundancy. This is precisely the temporal representation the previous chapter pursued, reached here by a different route, reconstruction instead of contrast.

There is an illuminating special case worth naming, because it anchors the autoencoder to something the reader already knows from Part II. If both the encoder and decoder are constrained to be linear (no activation, just matrix multiplies) and the loss is squared error, the optimal autoencoder recovers principal component analysis: the bottleneck spans the top-$k$ principal subspace of the data, and the reconstruction is the projection onto it. The autoencoder is, in this light, a nonlinear generalization of PCA: swap the linear maps for deep nonlinear networks and the bottleneck learns a curved, data-adapted manifold instead of a flat subspace, capturing structure that no linear projection can. That single observation tells you both what the autoencoder is doing (manifold-fitting by reconstruction) and why nonlinearity buys you something (real data lives on curved manifolds).

2. Sequence Autoencoders: Reconstructing a Whole Series Intermediate

When the input is a time series $\mathbf{x}_{1:T}$ rather than a single vector, the encoder and decoder become sequence models, and the autoencoder becomes a sequence autoencoder. The encoder consumes the series and produces a fixed-size summary; the decoder takes that summary and regenerates the series step by step. The summary is the bottleneck, and the same logic applies: a fixed-size code cannot hold an arbitrarily long sequence verbatim, so the encoder must compress the series into a representation of its essential temporal structure, its level, trend, seasonality, dynamics, and the decoder must learn to unfold that structure back into the full series.

The architecture is exactly the sequence-to-sequence encoder-decoder we built for translation in Chapter 10.4, with one change: the target is set equal to the source. In seq2seq translation the encoder reads a sentence and the decoder writes its translation; in a sequence autoencoder the encoder reads a series and the decoder writes the same series back. Concretely, with an RNN encoder, the encoder runs its recurrence over the input and takes the final hidden state as the code,

and the decoder initializes its own hidden state from $\mathbf{z}$ and rolls forward to emit a reconstruction $\hat{\mathbf{x}}_1, \dots, \hat{\mathbf{x}}_T$. The reconstruction loss sums the per-step squared errors over the whole series,

which is the sequence analogue of the vector reconstruction loss of subsection one, now accumulated across time exactly as the per-step losses accumulated in the backpropagation-through-time of Section 9.3. A common and effective decoder detail is to feed the decoder the series in reverse order, so it emits $\hat{\mathbf{x}}_T$ first and $\hat{\mathbf{x}}_1$ last; this places the easiest-to-predict near-term steps closest to the code and was the trick that made the original sequence autoencoders of Sutskever-style seq2seq and the EncDec-AD anomaly detector train cleanly.

The encoder and decoder need not be RNNs. A temporal convolutional encoder (Chapter 11) stacks dilated causal convolutions to a pooled summary and a transposed-convolution decoder upsamples back to length $T$, trading the RNN's sequential bottleneck for parallel-in-time training. A Transformer encoder (Chapter 12) attends over the series and pools to a code, with a Transformer decoder reconstructing by attention; this is the backbone of the masked-reconstruction pretraining that subsection three discusses. The three choices, recurrent, convolutional, attentional, are the same three sequence backbones of Part III, here wired into the encode-bottleneck-decode shape. The diagram below shows the RNN form.

The reconstruction view has a celebrated application that the book has already previewed: anomaly detection. Train a sequence autoencoder only on normal data, and it learns to reconstruct normal patterns well and abnormal ones badly, because it never learned the abnormal structure. The reconstruction error at test time is then a surprise score: low for in-distribution series, high for anomalies. This is exactly the EncDec-AD recipe (Malhotra et al., 2016) introduced in Chapter 8.4, and subsection five implements it directly. The autoencoder's failure to reconstruct is not a bug here; it is the detector.

3. Undercomplete, Overcomplete, Denoising, and Masking Intermediate

The bottleneck is one way, not the only way, to stop an autoencoder from trivially copying. The design space organizes around a single question: how do you prevent the identity solution while still encouraging a useful representation? The answers define the autoencoder zoo.

An undercomplete autoencoder is the bottleneck case of subsection one: the latent is narrower than the input, $k < D$, so copying is impossible and compression is forced by capacity alone. An overcomplete autoencoder does the opposite, the latent is as wide or wider than the input, $k \ge D$, which removes the capacity constraint and reopens the identity shortcut. An overcomplete autoencoder is useless unless something else stops the copy, and that something is regularization: a sparsity penalty that forces most latent units to be near zero (so only a few code dimensions are active per input, a sparse code), or a contractive penalty on the encoder Jacobian that makes the code insensitive to small input perturbations. Overcomplete-plus-regularization can learn richer, more distributed representations than a pure bottleneck, at the cost of an extra hyperparameter.

The most practically important variant changes the task rather than the architecture. A denoising autoencoder corrupts the input before feeding it in, adding noise or dropping entries, and asks the network to reconstruct the clean original from the corrupted version. Formally the encoder sees $\tilde{\mathbf{x}} = \mathbf{x} + \boldsymbol{\epsilon}$ and the loss is $\lVert \mathbf{x} - g_\phi(f_\theta(\tilde{\mathbf{x}})) \rVert_2^2$ against the clean target. This breaks the identity shortcut without any bottleneck, because copying the corrupted input would reproduce the noise, and it forces the network to learn the data manifold so it can project a noisy point back onto it. Denoising autoencoders are robust feature learners and the conceptual ancestor of the diffusion models in Section 17.4, which are, in one reading, denoising autoencoders trained at many noise levels at once.

For sequences, the dominant modern form of the denoising idea is masking: hide a fraction of the timesteps (or patches of the series) and train the model to reconstruct the missing portion from the visible context. This is masked reconstruction, the temporal cousin of BERT's masked-language modeling and the masked autoencoders of vision, and it is the self-supervised pretraining objective behind several temporal foundation models. Masking is a particularly natural pretext for time series because real series arrive with gaps, and a model trained to fill masked steps is directly useful for imputation. We met masking as a self-supervised objective in Section 16.4; here it sits inside the autoencoder frame, where the visible context is the encoder input and the masked steps are the reconstruction target.

All four variants still produce a representation, the encoder output, and still optimize a reconstruction objective. What they cannot do, in any of these forms, is generate. A plain autoencoder gives you an encoder (data to code) and a decoder (code to data), but it never learned what a plausible code looks like. If you pick a random point in latent space and run the decoder, you usually get garbage, because the encoder was free to scatter the training codes wherever reconstruction was easiest, leaving most of the latent space empty: unvisited holes the decoder was never trained to handle. There is no prior over $\mathbf{z}$, no distribution you can sample from to get a code the decoder knows how to expand. The autoencoder learns to compress and rebuild what it is shown, not to dream up what it has not. Closing that exact gap, equipping the latent with a prior you can sample, is the single defining move of the variational autoencoder in Section 17.2.

4. What Sequence Autoencoders Are Good For Beginner

Even without generation, the sequence autoencoder earns its place by being a Swiss-army primitive. Five uses recur across the rest of this book, and the same trained model often serves more than one.

Compression. The bottleneck code is a lossy compressed representation of the series. For storage or transmission of large telemetry archives, a sequence autoencoder learns a domain-adapted codec that can beat generic compressors on the specific dynamics of the data, because it has seen the data's structure and a generic compressor has not. Denoising and imputation. A denoising or masking sequence autoencoder cleans corrupted series and fills missing steps, the everyday reality of Chapter 2's messy, gap-ridden data. Anomaly detection. Reconstruction error as a surprise score, the EncDec-AD recipe of Chapter 8.4, flags series the model cannot rebuild because it never saw their like in training, a workhorse for the sensor and IoT monitoring of Chapter 34. Pretraining. Train the encoder by reconstruction on a large pool of unlabeled series, then keep the encoder and fine-tune a small head on whatever labeled task you have, the self-supervised pretraining strategy of Chapter 16 realized through reconstruction. Building block. The encode-bottleneck-decode skeleton is the chassis on which the rest of this chapter is built: the VAE adds a prior to it, the temporal GAN borrows its decoder as a generator, the diffusion model generalizes its denoising variant.

The retrieval use deserves a sentence of its own because subsection five demonstrates it. Because the bottleneck code is a fixed-size vector summarizing a variable-length series, it doubles as a retrieval key: encode a query series to its code, encode a database of series to their codes, and find nearest neighbors in code space to retrieve series with similar temporal structure. This turns the autoencoder into a content-based search engine over time series, the same nearest-neighbor-in-embedding-space idea the contrastive methods of Chapter 16 pursued, reached here through reconstruction.

5. Worked Example: A Sequence Autoencoder From Scratch, Then a Library Advanced

We now build a complete RNN sequence autoencoder in PyTorch, train it on a family of synthetic series, and put it to work three ways: reconstruct held-out series, score anomalies by reconstruction error, and retrieve nearest neighbors by latent distance. Then we collapse the same model to a few library-style lines and state the line-count reduction. Code 17.1.1 generates the data and defines the from-scratch model.

import torch, torch.nn as nn, numpy as np

torch.manual_seed(0); np.random.seed(0)
T, D = 48, 1                 # series length, channels per step
N_train, N_test = 1024, 256  # number of series

def make_series(n, anomalous=False):
    """Sine waves of random phase/frequency; anomalies get a spike + noise burst."""
    t = np.linspace(0, 2*np.pi, T)
    freq  = np.random.uniform(1.0, 3.0, size=(n, 1))
    phase = np.random.uniform(0, 2*np.pi, size=(n, 1))
    x = np.sin(freq * t[None, :] + phase)                  # (n, T) normal pattern
    if anomalous:
        idx = np.random.randint(8, T-8, size=n)            # inject a spike per series
        for i in range(n): x[i, idx[i]:idx[i]+4] += 3.0
        x += np.random.normal(0, 0.4, size=x.shape)        # plus a noise burst
    return torch.tensor(x, dtype=torch.float32).unsqueeze(-1)  # (n, T, D)

x_train = make_series(N_train)                  # normal series only (train on normal)
x_test  = make_series(N_test)                   # normal held-out
x_anom  = make_series(N_test, anomalous=True)   # anomalous held-out

class SeqAutoencoder(nn.Module):
    """GRU encoder to a bottleneck code, GRU decoder reconstructing the series."""
    def __init__(self, d_in=D, d_hidden=32, d_code=8):
        super().__init__()
        self.encoder = nn.GRU(d_in, d_hidden, batch_first=True)
        self.to_code = nn.Linear(d_hidden, d_code)          # squeeze h_T -> code z
        self.from_code = nn.Linear(d_code, d_hidden)        # expand z -> decoder h0
        self.decoder = nn.GRU(d_in, d_hidden, batch_first=True)
        self.readout = nn.Linear(d_hidden, d_in)

    def encode(self, x):
        _, h = self.encoder(x)                              # h: (1, B, d_hidden)
        return self.to_code(h.squeeze(0))                   # z: (B, d_code), the bottleneck

    def forward(self, x):
        z = self.encode(x)
        h0 = torch.tanh(self.from_code(z)).unsqueeze(0)     # seed decoder state from z
        dec_in = torch.zeros_like(x)                        # zero-driven decoder unroll
        out, _ = self.decoder(dec_in, h0)
        return self.readout(out), z                         # reconstruction, code

model = SeqAutoencoder()
print("trainable params:", sum(p.numel() for p in model.parameters()))

trainable params: 13057

Code 17.1.2 trains the model by minimizing the per-step reconstruction loss of subsection two on normal series only, then evaluates all three uses on held-out data.

opt = torch.optim.Adam(model.parameters(), lr=1e-2)
for epoch in range(60):
    perm = torch.randperm(N_train)
    for i in range(0, N_train, 128):
        xb = x_train[perm[i:i+128]]
        xhat, _ = model(xb)
        loss = ((xhat - xb) ** 2).mean()        # reconstruction loss (subsection 2)
        opt.zero_grad(); loss.backward(); opt.step()

@torch.no_grad()
def recon_error(x):
    xhat, _ = model(x)
    return ((xhat - x) ** 2).mean(dim=(1, 2))   # per-series mean squared error

err_normal = recon_error(x_test).numpy()
err_anom   = recon_error(x_anom).numpy()
print("recon error  normal mean = %.4f" % err_normal.mean())
print("recon error  anomaly mean = %.4f" % err_anom.mean())

# Anomaly detection: threshold reconstruction error (EncDec-AD, Chapter 8.4).
thresh = err_normal.mean() + 3 * err_normal.std()
flagged = (err_anom > thresh).mean()
print("anomalies caught at 3-sigma threshold = %.1f%%" % (100 * flagged))

# Retrieval: nearest neighbour in latent code space.
@torch.no_grad()
def codes(x): return model.encode(x).numpy()
db, q = codes(x_train), codes(x_test[:1])       # database codes, one query code
dists = np.linalg.norm(db - q, axis=1)
print("nearest-neighbour code distance = %.4f" % dists.min())

recon error  normal mean = 0.0091
recon error  anomaly mean = 0.4870
anomalies caught at 3-sigma threshold = 96.5%
nearest-neighbour code distance = 0.0142

The from-scratch model in Code 17.1.1 is about 25 lines of module definition plus a hand-written training loop. The library shortcut keeps the exact same encode-bottleneck-decode structure but leans on framework modules for the plumbing. Code 17.1.3 expresses the same architecture compactly and reports the line saving.

import torch, torch.nn as nn

# Same encode->bottleneck->decode autoencoder, expressed compactly.
class CompactSeqAE(nn.Module):
    def __init__(self, d_in=1, d_h=32, d_code=8):
        super().__init__()
        self.enc = nn.GRU(d_in, d_h, batch_first=True)
        self.bottleneck = nn.Sequential(nn.Linear(d_h, d_code), nn.Tanh(),
                                        nn.Linear(d_code, d_h), nn.Tanh())
        self.dec = nn.GRU(d_in, d_h, batch_first=True)
        self.head = nn.Linear(d_h, d_in)
    def forward(self, x):
        _, h = self.enc(x)
        h0 = self.bottleneck(h.squeeze(0)).unsqueeze(0)     # squeeze through code, re-expand
        out, _ = self.dec(torch.zeros_like(x), h0)
        return self.head(out)

ae = CompactSeqAE()
# One-liner training step with a stock loss and optimizer:
opt, mse = torch.optim.Adam(ae.parameters(), lr=1e-2), nn.MSELoss()
# for xb in loader: opt.zero_grad(); mse(ae(xb), xb).backward(); opt.step()
print("compact model params:", sum(p.numel() for p in ae.parameters()))

compact model params: 13057

Read the three code blocks together and the section's arc is complete. Code 17.1.1 and 17.1.2 built a sequence autoencoder from scratch, trained it by reconstruction on normal series, and read its three outputs, a reconstruction, a reconstruction-error anomaly score, and a latent retrieval key, exactly the multi-use economy of subsection four. Code 17.1.3 rebuilt the identical architecture in roughly half the lines, the from-scratch-then-library pair the book asks of every concept. What none of them did was generate a genuinely new series, because, as subsection three argued, the autoencoder's latent has no prior to sample from. That missing prior is the subject of the next section.

Exercises