"Chapter 9 left me with a wound: I could hold a memory but not learn to hold it, because the lesson faded before it reached the part of me that mattered. So they rebuilt me with a vault. Now there is a path down my middle, a cell state, that addition flows along almost untouched, and three small gates that decide what enters, what is quietly let go, and what I am willing to show the world. The forget gate is the one I am proudest of and most ashamed of: I can carry a grudge across a thousand steps, or release it in a single one, and the gradient rides that protected lane home without dying. I am not elegant. I have more parameters than I can justify and a younger cousin who does nearly as well with fewer. But for a decade I was the sequence model, and every value I ever carried, I carried on purpose."
A Cell State Carrying a Grudge Across a Thousand Steps
Chapter Overview
Chapter 9 ended on a diagnosis, not a cure. A vanilla recurrence can represent a long-range dependency in principle, yet backpropagation through time cannot teach it one, because the gradient that carries the lesson backward shrinks as a product of Jacobians and dies somewhere across the gap. This chapter delivers the first and most consequential answer to that exact arithmetic fact. The recurrent neural network as it is actually used in practice is not the bare $h_t = f(h_{t-1}, x_t)$ of the previous chapter; it is a gated cell, engineered around a protected carry path along which information, and crucially its gradient, can travel across many timesteps almost undisturbed. The same trainable recurrence is still at the core, but now it is built so it can be taught the dependencies it was always able to represent.
The chapter develops that idea in stages. It opens by recapping the vanilla recurrent network as the honest baseline, the trainable recurrence whose forward pass is simple and whose backward pass we already know is fragile. It then introduces the Long Short-Term Memory cell, whose additive cell state and learned input, forget, and output gates create the protected lane that keeps the gradient near one, and the Gated Recurrent Unit, a lighter cousin that merges the gating into two streams and reaches comparable quality with fewer parameters. With a cell that can carry a state across a long horizon in hand, the chapter assembles cells into the encoder-decoder and sequence-to-sequence architecture: one recurrence reads an input sequence into a context vector, a second generates an output sequence from it, trained with teacher forcing. This is the blueprint that powered a generation of translation and forecasting systems and the structure whose single-context-vector bottleneck will, in the next chapters, motivate attention.
Section 10.5 is the payoff of a thread this book has carried since Part II. Section 7.6 closed the classical part with a precise claim: the Kalman filter is a recurrence that carries a state forward, corrects it with each observation, and reads out a prediction, with the transition and observation matrices handed to it as known. Chapter 9 made that recurrence trainable but fragile. Here, with the gated cell, the comparison becomes exact and instructive. The cell state is the filter's state estimate, the input and forget gates play the structural role of the Kalman gain in deciding how much to trust the new observation against the carried memory, and the whole update is a learned, nonlinear, data-driven version of the filter's predict-correct cycle. The recurrent network is the Kalman filter with its matrices set free and its gain made adaptive, and Section 10.5 draws that map line by line.
The chapter closes honestly. The gated recurrent network earned its decade of dominance, but it is no longer the uncontested default. Its sequential computation cannot be parallelized across time the way a convolution or an attention layer can, which makes it slow to train on long sequences, and its memory, though vastly better than the vanilla cell's, still attenuates over very long horizons. The Temporal Convolutional Networks of Chapter 11 and the Transformers of Chapter 12 are the direct challengers, trading the recurrence for parallel receptive fields and direct all-to-all connections. Reading this chapter well means holding both truths: the gated cell is a beautiful and still widely deployed answer to the gradient problem, and it is the baseline the next two chapters were built to beat. Following the Part III convention, the worked examples lean on synthetic long-lag tasks where the right answer is known exactly, so the gain in memory is measured rather than asserted.
Prerequisites
This chapter builds directly on two anchors. From Chapter 9: Neural Sequence Modeling Fundamentals you need everything: the recurrence $h_t = f(h_{t-1}, x_t)$ and its unrolling, backpropagation through time, the sequence-task taxonomy and the encoder-decoder bottleneck, and above all the vanishing and exploding gradient problem, because every cell in this chapter is an engineered answer to it and will make no sense without the problem it solves. From Chapter 7: State-Space Models and Filtering, and especially its closing recurrence bridge, you need the Kalman filter as a state carried forward and corrected by each observation, since Section 10.5 maps the gated cell onto that predict-correct cycle line by line; without the filter the bridge has nothing to bridge to. You also need a working command of PyTorch at the level Chapter 9 assumed: defining modules, training with an optimizer, and trusting autograd to run the backward pass. Readers who want a refresher on optimization and deep learning basics, on PyTorch, or on the state-space machinery of Chapter 7 will find Appendix C, Appendix D, and all referenced chapters indexed in the Table of Contents.
If you keep one idea from this chapter, keep this: a gated recurrent cell solves Chapter 9's vanishing gradient by adding a protected, mostly-additive carry path and learned gates that decide what to remember, forget, and expose, which is exactly a Kalman filter with its matrices learned and its gain made adaptive. The LSTM gives that carry path a separate cell state guarded by three gates; the GRU reaches nearly the same quality with two gates and fewer parameters. Stack such cells into an encoder-decoder and you get the sequence-to-sequence model, trained with teacher forcing, whose single context vector both powered a decade of sequence systems and set up the bottleneck that attention will later remove. The recurrence is still the carry-update-read shape inherited from the Kalman filter of Chapter 7, only now it is learned, nonlinear, and able to keep its gradient alive long enough to be taught the dependencies the vanilla cell could represent but never learn.
Chapter Roadmap
- 10.1 Vanilla RNNs The plain recurrent network restated as the honest baseline for this chapter, its forward recurrence and its hidden-state readout, its expressive reach and its fragile backward pass, and the concrete long-lag tasks on which its memory measurably fails, setting the bar every gated cell must clear.
- 10.2 LSTM Networks The Long Short-Term Memory cell in full: the additive cell state that forms the protected carry path, the input, forget, and output gates that learn what to write, retain, and expose, why the additive update keeps the gradient near one across long horizons, and a from-scratch cell paired with its PyTorch equivalent.
- 10.3 GRU Networks The Gated Recurrent Unit as a lighter LSTM: the update and reset gates that merge the carry and the gating into two streams, the parameter and speed savings, the empirical near-parity with the LSTM, and guidance on when the simpler cell is the right default.
- 10.4 Encoder-Decoder and Seq2Seq Architectures Assembling cells into a sequence-to-sequence model: an encoder that compresses an input sequence into a context vector, a decoder that generates an output sequence from it, teacher forcing and its exposure-bias cost, and the single-context bottleneck that motivates the attention of the chapters ahead.
- 10.5 Bridge: RNNs as Learned Kalman Filters The explicit map from the gated cell to the Kalman filter of Section 7.6: the cell state as the state estimate, the gates as a learned adaptive gain, and the gated update as a nonlinear, data-driven predict-correct cycle, the precise payoff of the temporal thread carried since Part II.
- 10.6 Practical Considerations and Training Stability What it takes to train deep recurrent networks well: stacking and bidirectionality, dropout that respects the recurrence, gradient clipping and initialization, packed variable-length batches, and the honest limits (sequential compute, attenuating long memory) that the next chapters were built to overcome.
Once you have worked through the six sections, the Hands-On Lab below chains them into a single experiment. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have raced the vanilla cell of Section 10.1 against the LSTM of Section 10.2 and the GRU of Section 10.3, built the sequence-to-sequence forecaster of Section 10.4, mapped its gates onto the Kalman update of Section 10.5, and applied the stability toolkit of Section 10.6.
Hands-On Lab: Gate It and Watch It Learn
Objective
Take the exact long-lag task that defeated the vanilla recurrence in Chapter 9 and watch gating cure it, then see the cured cell reveal itself as a learned Kalman filter. You will reuse the delayed-copy task, in which the target at the end of a sequence is a value injected far back at the start with pure distraction in between, and train three models head to head on it: the vanilla recurrent network of Section 10.1, the LSTM of Section 10.2, and the GRU of Section 10.3. Sweeping the lag from short to long, you will record each model's accuracy and its parameter count, and produce the central result of the chapter: the vanilla cell collapses as the gap grows while both gated cells carry the value across gaps several times longer, with the GRU matching the LSTM at a lower parameter cost. Next you will assemble GRU cells into the encoder-decoder forecaster of Section 10.4, train it with teacher forcing to map an input window to a multistep forecast, and feel the exposure-bias gap between teacher-forced training and free-running inference. Then comes the conceptual payoff of Section 10.5: you will take one trained gated cell, read its forget and input gates on a step where a clean observation arrives after a noisy stretch, and map those gate values onto the Kalman gain, seeing the cell trust the new observation exactly when a filter would. Finally you will apply the stability toolkit of Section 10.6, gradient clipping, sensible initialization, and packed variable-length batches, and confirm the gated models train steadily where the vanilla cell did not.
What You'll Practice
- Reproducing the vanilla recurrent network as a measured baseline and locating the gap where its memory fails, following Section 10.1.
- Building and training LSTM and GRU forecasters and comparing their accuracy and parameter counts on a swept long-lag task, following Section 10.2 and Section 10.3.
- Assembling a GRU encoder-decoder for multistep forecasting and training it with teacher forcing, following Section 10.4.
- Reading a trained cell's gates and mapping them onto the Kalman gain of the predict-correct cycle, following Section 10.5.
- Applying gradient clipping, initialization, and packed batches to train deep recurrences stably, following Section 10.6.
Setup
You need PyTorch for all three cells, the encoder-decoder, and the autograd that runs backpropagation through time, NumPy to generate the synthetic delayed-copy sequences, and Matplotlib to plot accuracy and parameter count against the lag. No dataset download is required; the task is generated in a few lines so the lab runs out of the box, and a real long-horizon benchmark is available from the Monash Forecasting Archive linked in the bibliography when you want to repeat the comparison on field data. Every model trains and evaluates on disjoint sequences and respects the temporal split discipline of Chapter 2, so a strong score reflects real long-range carry rather than a leak of the answer. PyTorch supplies nn.RNN, nn.LSTM, and nn.GRU as drop-in cells, so swapping architectures is a one-line change and the comparison stays fair.
import torch, torch.nn as nn
class GatedForecaster(nn.Module):
def __init__(self, cell="LSTM", hidden=64):
super().__init__()
Cell = {"RNN": nn.RNN, "LSTM": nn.LSTM, "GRU": nn.GRU}[cell]
self.rnn = Cell(input_size=1, hidden_size=hidden, batch_first=True)
self.head = nn.Linear(hidden, 1)
def forward(self, x): # x: (batch, length, 1)
out, state = self.rnn(x) # carry a (gated) state across every step
return self.head(out[:, -1]) # read the target from the last output
Steps
Step 1: Race vanilla, LSTM, and GRU on the swept long-lag task
Generate the delayed-copy task at a range of gaps and train all three cells of Section 10.1, Section 10.2, and Section 10.3 on each gap with the shared module above. Record final accuracy and parameter count per model per gap. You will see the vanilla cell match the gated cells at short gaps and fall off a cliff as the gap grows, while both gated cells hold accuracy far longer, the additive carry path doing exactly what the gradient analysis of Chapter 9 promised it would.
Step 2: Compare parameters against accuracy and read the GRU verdict
Plot accuracy and parameter count side by side across the gaps. The LSTM and GRU curves should sit close together at every gap, but the GRU reaches that accuracy with fewer parameters, the empirical near-parity of Section 10.3. Read off the regime where the cheaper cell is the right default and the narrower regime where the LSTM's extra gate earns its keep.
Step 3: Build a GRU encoder-decoder and train with teacher forcing
Assemble GRU cells into the encoder-decoder of Section 10.4: an encoder compresses an input window into a context vector, a decoder generates a multistep forecast from it. Train it with teacher forcing, feeding the ground-truth previous step into the decoder, then evaluate it free-running, feeding its own predictions. The gap between the two is exposure bias, and seeing it is the point.
class GRUSeq2Seq(nn.Module):
def __init__(self, hidden=64):
super().__init__()
self.enc = nn.GRU(1, hidden, batch_first=True)
self.dec = nn.GRU(1, hidden, batch_first=True)
self.head = nn.Linear(hidden, 1)
def forward(self, src, tgt): # src: input window, tgt: teacher-forced targets
_, ctx = self.enc(src) # context vector summarizes the input
out, _ = self.dec(tgt, ctx) # decode the forecast conditioned on context
return self.head(out) # one prediction per decoder step
Step 4: Map the gates onto the Kalman update
Take one trained gated cell and feed it a sequence with a clean observation arriving after a noisy stretch. Read the forget and input gate values at that step and line them up against the Kalman gain of Section 10.5: the input gate opens to admit the trustworthy observation while the forget gate decides how much carried state to keep, exactly the trust trade-off the gain encodes. You will watch the cell behave as a learned, adaptive filter on a step where a hand-tuned Kalman filter would behave the same way.
Step 5: Apply the stability toolkit
Bring in the practical tools of Section 10.6. Add gradient clipping to cap rare exploding updates, initialize sensibly, set the LSTM forget-gate bias positive so the cell starts out remembering, and pack variable-length batches so padding never pollutes the recurrence. Retrain and confirm the gated models converge steadily and faster than the unstabilized run, the difference between a recurrence that trains and one that merely could.
# inside the training loop, after loss.backward():
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # tame exploding gradients
# pack ragged batches so padded steps never enter the recurrence:
packed = nn.utils.rnn.pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False)
Expected Output
The lab produces a clear before-and-after story. The accuracy-versus-gap plot of Step 1 shows the vanilla cell collapsing toward chance as the gap grows while the LSTM and GRU hold high accuracy across gaps several times longer, the gated carry path curing the exact failure Chapter 9 measured. The parameter comparison of Step 2 shows the GRU tracking the LSTM's accuracy at a visibly lower parameter count, the practical case for the simpler cell. The encoder-decoder of Step 3 forecasts well under teacher forcing and degrades a measurable amount when run free, exposing the exposure-bias gap and the single-context bottleneck. The gate-reading of Step 4 produces the chapter's conceptual punchline: on a step with a clean observation after noise, the cell's gates move the way a Kalman gain would, making the learned-filter claim of Section 10.5 concrete rather than rhetorical. The stabilized run of Step 5 converges steadily where the raw recurrence wobbled. The reader finishes with a measured, plotted, hands-on understanding that gating solves the long-dependency problem, that the GRU buys most of the LSTM's gain more cheaply, and that the gated cell is a Kalman filter that learned its own gain.
The whole comparison rides on PyTorch's drop-in recurrent modules. nn.RNN, nn.LSTM, and nn.GRU share an identical call signature, so swapping the vanilla cell for a gated one is a single argument and the rest of the training code never changes, which is what keeps the race fair. The same modules handle multilayer stacking and bidirectionality through their num_layers and bidirectional flags, torch.nn.utils.clip_grad_norm_ implements clipping in one line, and nn.utils.rnn.pack_padded_sequence handles ragged batches so padded steps never enter the recurrence. Build one LSTM cell from scratch, as Section 10.2 does, so you know exactly what the gates compute, then let these library cells carry every experiment in the lab. The line-count reduction is dramatic: a hand-written, numerically careful LSTM with packing and clipping runs to well over a hundred lines, and the library collapses it to a handful while handling the cuDNN-accelerated backward pass internally.
Stretch Goals
- Add attention over the encoder states to the sequence-to-sequence model of Step 3 and watch the single-context bottleneck of Section 10.4 relax, a preview of Chapter 12.
- Make the encoder bidirectional and stack two layers using the
bidirectionalandnum_layersflags, and measure the accuracy change against the parameter and compute cost, exercising the stacking tools of Section 10.6. - Repeat the gate-reading of Step 4 across many steps and correlate the input-gate value with observation noise, quantifying how closely the learned gate tracks the ideal Kalman gain of Section 10.5.
What's Next?
This chapter cured Chapter 9's gradient problem with a gated carry path, but it left two costs on the table: a recurrence computes step by step and cannot be parallelized across time, and even a gated memory attenuates over very long horizons. Chapter 11: Temporal Convolutional Networks takes the first direct swing at both. Instead of carrying a state forward one step at a time, it stacks dilated causal convolutions whose receptive field grows exponentially with depth, reaching far back in a handful of layers, with every timestep computed in parallel during training. Residual connections keep the deep stack trainable, and causal masking preserves the no-peeking-at-the-future discipline that every temporal model owes its data. The result is a model that often matches or beats the gated recurrent network on long-sequence tasks while training far faster, and it sets up the even more radical move of Chapter 12, where attention abandons the fixed receptive field entirely and connects any two timesteps directly. The carry-update-read recurrence steps aside; the question of how to reach far into the past without losing the signal stays exactly the same.
Bibliography & Further Reading
Foundational Papers
Hochreiter, S., Schmidhuber, J. "Long Short-Term Memory." Neural Computation, 9(8), 1997. MIT Press. bioinf.jku.at
The paper that introduced the LSTM cell and its additive, gate-protected carry path, the direct answer to the vanishing gradient and the entire subject of Section 10.2.
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y. "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." EMNLP, 2014. arXiv:1406.1078. arxiv.org/abs/1406.1078
The paper that introduced the GRU and the RNN encoder-decoder, the two ideas that anchor Sections 10.3 and 10.4.
Sutskever, I., Vinyals, O., Le, Q. V. "Sequence to Sequence Learning with Neural Networks." NeurIPS, 2014. arXiv:1409.3215. arxiv.org/abs/1409.3215
The work that established the LSTM encoder-decoder as a general sequence-to-sequence learner, the blueprint and the context-vector bottleneck of Section 10.4.
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., Schmidhuber, J. "LSTM: A Search Space Odyssey." IEEE Transactions on Neural Networks and Learning Systems, 2017. arXiv:1503.04069. arxiv.org/abs/1503.04069
A large ablation of LSTM variants showing which gates matter most, including the forget gate and its positive bias, the practical evidence behind Sections 10.2 and 10.6.
Chung, J., Gulcehre, C., Cho, K., Bengio, Y. "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling." 2014. arXiv:1412.3555. arxiv.org/abs/1412.3555
The head-to-head study of GRU against LSTM that established their near-parity, the empirical backbone of the GRU verdict in Section 10.3.
Graves, A. "Generating Sequences With Recurrent Neural Networks." 2013. arXiv:1308.0850. arxiv.org/abs/1308.0850
An influential demonstration of LSTM generation and training practice, with clipping and stacked recurrences, that informs the stability toolkit of Section 10.6.
Salinas, D., Flunkert, V., Gasthaus, J., Januschowski, T. "DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks." International Journal of Forecasting, 2020. arXiv:1704.04110. arxiv.org/abs/1704.04110
The model that brought LSTM forecasting to industrial scale, the recurrent sequence-to-sequence forecaster of Section 10.4 deployed on real probabilistic demand data.
Tutorials
Olah, C. "Understanding LSTM Networks." colah's blog, 2015. colah.github.io
The canonical visual walkthrough of the LSTM cell state and its gates, and of the GRU, the single clearest companion to Sections 10.2 and 10.3.
Tools & Libraries
PyTorch recurrent layers: nn.RNN, nn.LSTM, nn.GRU, and sequence packing. Official documentation. pytorch.org/docs
The drop-in cells behind every model in this chapter; their shared interface makes the vanilla-versus-gated comparison a one-line change and supplies stacking, bidirectionality, and packing for the lab.
Datasets & Benchmarks
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., Montero-Manso, P. "Monash Time Series Forecasting Archive." NeurIPS Datasets and Benchmarks, 2021. forecastingdata.org
A large, curated collection of real forecasting datasets across many domains and frequencies, the field-data benchmark for taking the gated forecasters of this chapter beyond the synthetic long-lag task.