Vanilla RNNs

"I have exactly one hidden state and I pour everything I have ever seen into it. The opening of the sequence, the surprise in the middle, the thing you told me a thousand steps ago: all of it, into the same little vector, overwritten a little more every step. I am not forgetful on purpose. I simply ran out of room around step thirty and have been improvising ever since."

A Vanilla RNN Doing Its Best With One Hidden State

Chapter 9 built the machinery of neural sequence modeling without committing to a specific architecture: Section 9.3 derived backpropagation through time over a generic recurrence, and Section 9.4 dissected the vanishing and exploding gradients that the unrolled chain produces. This section fills in the simplest concrete cell that lives inside that machinery, the Elman recurrent network, and makes it run. The recurrence we use is exactly the skeleton that closed Part II, where Section 7.6 showed the Kalman filter, the HMM forward pass, and the RNN to be one loop, $\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$, $\mathbf{y}_t = g(\mathbf{h}_t)$, distinguished only by how $f$ and $g$ are chosen. The state-space models of Part II fixed $f$ and $g$ from a hand-specified dynamical model; the RNN of this section learns them by gradient descent. Throughout we use the unified notation of Appendix A: $\mathbf{x}_t$ the input at step $t$, $\mathbf{h}_t$ the hidden state, $\mathbf{y}_t$ the readout, $\mathbf{W}_h, \mathbf{W}_x, \mathbf{W}_y$ the shared weights.

By the end of this section you will be able to write the Elman recurrence and explain each term; run the forward pass over a batch of sequences and pick the right output head (many-to-one versus many-to-many) for a task; argue precisely what an RNN can represent that a fixed-window MLP cannot; explain why that representational reach is undercut in practice by vanishing gradients; and implement, train, and library-shortcut a vanilla RNN, including a demonstration of its long-lag failure. These competencies are the foundation for every recurrent architecture in this chapter, and the failure we end on is the precise motivation for the gates that follow.

1. The Elman Recurrence: One State, One Rule, Shared Across Time Beginner

The vanilla RNN, introduced by Jeffrey Elman in 1990, is defined by two equations and nothing more. At each timestep it updates a hidden state and reads out a prediction:

Read the update term by term. The input $\mathbf{x}_t \in \mathbb{R}^{d_x}$ is the observation at step $t$; the matrix $\mathbf{W}_x \in \mathbb{R}^{d_h \times d_x}$ projects it into the hidden space. The previous state $\mathbf{h}_{t-1} \in \mathbb{R}^{d_h}$ is everything the network remembers about the past; the recurrent matrix $\mathbf{W}_h \in \mathbb{R}^{d_h \times d_h}$ decides how that memory is carried forward and mixed. Their sum, plus a bias $\mathbf{b}$, is squashed through $\tanh$ into the new state $\mathbf{h}_t$, which is bounded componentwise in $(-1, 1)$. The readout matrix $\mathbf{W}_y \in \mathbb{R}^{d_y \times d_h}$ turns the state into an output of whatever dimension the task needs. The sequence is initialized with $\mathbf{h}_0$, usually the zero vector or a learned constant.

The single most important property is hidden in plain sight: the three matrices $\mathbf{W}_h$, $\mathbf{W}_x$, $\mathbf{W}_y$ and the bias $\mathbf{b}$ do not depend on $t$. The same rule is applied at step one, step ten, and step ten thousand. This hidden-state update is the learned analogue of the Kalman filter prediction-correction cycle introduced in Section 7.3. This is the recurrence skeleton of Section 7.6, now with the transition and emission learned rather than derived from physics. The structured state-space models of Section 13.1 generalize this recurrence with a linear state transition designed for efficient parallel training. And it is precisely the unrolling target of Section 9.3: stamp this one cell onto every timestep, tie the weights, and you get the deep feedforward graph that backpropagation through time differentiates. The whole of Section 10.1 is an exploration of what this two-line rule can and cannot do.

The right way to think about $\mathbf{h}_t$ is as a running summary of the sequence so far. It is the network's lossy compression of $\mathbf{x}_1, \dots, \mathbf{x}_t$ into a fixed-size $d_h$-dimensional vector. Nothing in the architecture grows as the sequence lengthens: a single $d_h$-vector must hold whatever the network needs from a past of arbitrary length. This is the source of both the RNN's power (a fixed network handles any length) and its central weakness (a fixed-width summary must forget, and the next subsection's gradient analysis shows it forgets the distant past first). The state is a memory with a hard size budget and an overwrite-every-step update policy (Figure 10.1).

2. The Forward Pass: Parameter Sharing and Output Heads Beginner

Running an RNN forward is a single loop. Start at $\mathbf{h}_0$, then for $t = 1, \dots, T$ apply the update to fold in $\mathbf{x}_t$ and produce $\mathbf{h}_t$ (and, if the task wants it, a per-step output $\mathbf{y}_t$). Because the weights are shared, the same handful of matrices is reused at every iteration: an RNN with $d_h = 128$, $d_x = 10$, $d_y = 1$ has a few tens of thousands of parameters whether it processes sequences of length ten or length ten thousand. Figure 10.1.1 shows the recurrence drawn as a folded loop on the left and unrolled across three steps on the right, the two views we move between constantly.

How we read predictions off the state depends on the task, and there are two dominant patterns. In a many-to-one head the network consumes the whole sequence and produces a single output, almost always from the final state: $\mathbf{y} = \mathbf{W}_y \mathbf{h}_T$. This is the shape of sequence classification (is this ECG trace arrhythmic?) and of point forecasting (given the last $T$ observations, predict the next value). In a many-to-many head the network produces an output at every step, $\mathbf{y}_t = \mathbf{W}_y \mathbf{h}_t$ for all $t$, which is the shape of sequence labeling (tag each token), of next-step prediction in language modeling, and of multi-step forecasting where each step emits a prediction. The two heads share the identical recurrence; they differ only in which states feed the readout and the loss. Table 10.1.2 lays out the common configurations.

One consequence of parameter sharing deserves emphasis because it is easy to miss and central to everything that follows. Because $\mathbf{W}_x$ is shared, the RNN applies the same feature extractor to the input at every position; it does not have a separate "position 5 weight" the way a window MLP has a separate weight for each slot of its window. This positional invariance is a strong, often correct, inductive bias for time series: a pattern means the same thing whether it appears early or late. It is also why an RNN trained on length-50 sequences can be run, without retraining, on length-500 sequences: there is no parameter tied to a position that does not exist at training time. We exploit exactly this in subsection four when the model trained on short sequences is asked to handle a long lag.

3. Reach Versus Reality: Unbounded Context and the Vanishing-Gradient Limit Intermediate

What can an RNN represent that a fixed-window model cannot? The honest answer is the whole reason RNNs exist. Consider the natural baseline: a window MLP that takes the last $w$ observations $\mathbf{x}_{t-w+1}, \dots, \mathbf{x}_t$ as a flat input vector and predicts $\mathbf{y}_t$. Such a model is structurally blind beyond its window. Any dependency longer than $w$ steps simply cannot reach its output, because the inputs that would carry it were never presented. To capture a longer dependency you must enlarge $w$, and the parameter count grows with the window, so unbounded context demands unbounded parameters. The window is a hard horizon baked into the architecture.

The RNN removes that horizon in principle. Because the hidden state is carried forward indefinitely and updated by a rule that never resets, information from $\mathbf{x}_1$ can in principle still be present in $\mathbf{h}_T$ for any $T$, with a fixed parameter count. The recurrence is, in the formal sense, a universal model of sequences: an RNN with enough hidden units can approximate any measurable sequence-to-sequence mapping, and with unbounded precision and time it is Turing complete (Siegelmann and Sontag, 1992). So on paper the vanilla RNN dominates the window MLP outright: unbounded context, no horizon, constant parameters. Figure 10.1.3 contrasts the two reaches.

Now the reality. The path that gives the RNN its reach is a chain of state updates, and a gradient that must train that reach has to travel back along the same chain, multiplying one hidden-to-hidden Jacobian per step exactly as Section 9.3 derived. For the $\tanh$ cell that Jacobian is $\mathbf{J}_t = \operatorname{diag}(1 - \mathbf{h}_t^2)\,\mathbf{W}_h$, and a gradient travelling from step $t$ back to step $k$ is multiplied by the product $\prod_{j=k+1}^{t} \mathbf{J}_j$. Because $\tanh'$ is at most one and typically much less, and because a randomly initialized $\mathbf{W}_h$ has a spectral radius that is rarely exactly one, this product shrinks geometrically when the per-step Jacobian norm is below one (the gradient vanishes) or grows geometrically when it is above one (the gradient explodes). This is the vanishing/exploding gradient pathology that Section 9.4 analyzed in full, and for the vanilla RNN it is decisive.

The practical consequence is a painful gap between representation and learning. An RNN can represent a 200-step dependency, but if the gradient that would teach it that dependency has decayed by a factor of, say, $0.8^{200} \approx 10^{-19}$ by the time it reaches the relevant early weights, the dependency is never learned. The network has the capacity and lacks the training signal. Empirically, vanilla RNNs reliably learn dependencies of roughly ten to twenty steps and struggle badly beyond that, a limit that has nothing to do with their representational power and everything to do with the geometry of the gradient. This single fact, the unbounded reach undercut by a vanishing gradient, is the reason the rest of this chapter exists.

4. Worked Example: A Vanilla RNN From Scratch, Then With nn.RNN Advanced

We now make the cell run. The task is deliberately small and interpretable so the mechanics stay visible. We pose a cumulative-sum threshold problem: each input is a short window of a noisy sensor stream (the sensor-IoT series threaded through Chapter 8), and the many-to-one target at the final step is whether the running sum of a hidden driver has crossed a threshold. The dependency is short here, well inside the trainable horizon, so the vanilla RNN learns it cleanly. Then we deliberately stretch the lag and watch it fail. Code 10.1.1 implements the Elman cell and its forward pass from scratch with torch tensors, no nn module, so every matrix multiply of subsection one is visible.

import torch

torch.manual_seed(0)
d_x, d_h, d_y = 4, 32, 1                       # input, hidden, output widths

def make_batch(n, T, lag):
    """n sequences of length T; label = 1 if the driver summed over the FIRST
    `lag` steps exceeds 0. Long lag => long-range dependency the RNN must learn."""
    x = torch.randn(n, T, d_x)                 # noisy multivariate sensor windows
    driver = x[:, :lag, 0].sum(dim=1)          # the signal lives in feature 0, first `lag` steps
    y = (driver > 0).float().unsqueeze(1)      # many-to-one binary target at the final step
    return x, y

# --- The Elman cell, from scratch: parameters are bare tensors we update by hand. ---
def init_params():
    g = 1.0 / d_h ** 0.5
    return {
        "Wx": (torch.randn(d_h, d_x) * g).requires_grad_(),   # input -> hidden
        "Wh": (torch.randn(d_h, d_h) * g).requires_grad_(),   # hidden -> hidden (the recurrence)
        "b":  torch.zeros(d_h, requires_grad=True),           # hidden bias
        "Wy": (torch.randn(d_y, d_h) * g).requires_grad_(),   # hidden -> output
    }

def rnn_forward(p, x):
    """Run h_t = tanh(Wh h_{t-1} + Wx x_t + b) over the sequence; read out from h_T."""
    n, T, _ = x.shape
    h = torch.zeros(n, d_h)                     # h_0 = 0, the running summary starts empty
    for t in range(T):                          # the time loop: same weights every step
        h = torch.tanh(x[:, t] @ p["Wx"].T + h @ p["Wh"].T + p["b"])
    return h @ p["Wy"].T                         # many-to-one readout from the FINAL state

def train(p, T, lag, steps=400, lr=0.05):
    bce = torch.nn.BCEWithLogitsLoss()
    for s in range(steps):
        x, y = make_batch(128, T, lag)
        logits = rnn_forward(p, x)
        loss = bce(logits, y)
        loss.backward()
        with torch.no_grad():                   # manual SGD step on the bare tensors
            for v in p.values():
                v -= lr * v.grad; v.grad.zero_()
    return loss.item()

p = init_params()
final_loss = train(p, T=12, lag=4)              # SHORT lag: dependency well inside trainable horizon
# evaluate accuracy on a fresh batch
xe, ye = make_batch(2000, 12, 4)
acc = ((torch.sigmoid(rnn_forward(p, xe)) > 0.5).float() == ye).float().mean()
print("from-scratch RNN, short lag=4:  train loss %.3f  test acc %.3f" % (final_loss, acc))

from-scratch RNN, short lag=4:  train loss 0.118  test acc 0.961

Now the library pair. Code 10.1.2 replaces the entire hand-built cell, the parameter dictionary, the explicit time loop, and the manual SGD step, with nn.RNN (which is exactly the Elman $\tanh$ cell of subsection one) plus an Adam optimizer. The recurrence, the weight management, and backpropagation through time are all internal.

import torch.nn as nn

class RNNClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.rnn = nn.RNN(d_x, d_h, batch_first=True, nonlinearity="tanh")  # the Elman cell
        self.head = nn.Linear(d_h, d_y)                                     # readout from h_T

    def forward(self, x):
        out, h_T = self.rnn(x)            # out: all states; h_T: final state (1, n, d_h)
        return self.head(h_T.squeeze(0))  # many-to-one: classify from the final state

def train_lib(T, lag, steps=400):
    model = RNNClassifier()
    opt = torch.optim.Adam(model.parameters(), lr=0.01)   # Adam, not hand-rolled SGD
    bce = nn.BCEWithLogitsLoss()
    for s in range(steps):
        x, y = make_batch(128, T, lag)
        loss = bce(model(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    xe, ye = make_batch(2000, T, lag)
    acc = ((torch.sigmoid(model(xe)) > 0.5).float() == ye).float().mean()
    return loss.item(), acc.item()

l_short, a_short = train_lib(T=12, lag=4)    # short lag: should match the from-scratch result
l_long,  a_long  = train_lib(T=60, lag=55)   # LONG lag: dependency far beyond the trainable horizon
print("nn.RNN short lag=4:   train loss %.3f  test acc %.3f" % (l_short, a_short))
print("nn.RNN long  lag=55:  train loss %.3f  test acc %.3f" % (l_long,  a_long))

nn.RNN short lag=4:   train loss 0.087  test acc 0.974
nn.RNN long  lag=55:  train loss 0.690  test acc 0.531

Read the three numbers together. Both the from-scratch cell and the library cell solve the short-lag task at well above 95 percent, confirming that the two implementations are the same model and that short dependencies are easy. The library cell then fails on the long-lag task, dropping to chance with a loss pinned at $\ln 2$, the loss of a classifier that has learned nothing. The model did not run out of capacity; it ran out of gradient. A 55-step dependency is trivially representable by the recurrence and entirely invisible to the optimizer, because the signal that would teach it has decayed below noise. This is precisely the gap between representational reach and trainable reach from subsection three, reproduced in twenty lines of code, and it is the exact failure the gated cells of the next section repair.

import torch.nn as nn

# A 2-layer, bidirectional vanilla RNN over batched, variable-length sequences:
rnn = nn.RNN(input_size=4, hidden_size=32, num_layers=2,
             nonlinearity="tanh", batch_first=True, bidirectional=True)
outputs, h_final = rnn(x)        # x: (batch, time, 4); outputs: (batch, time, 64)
# the entire recurrence + BPTT is inside this one call; .backward() trains it

Exercises

1. (Conceptual) A colleague claims that "an RNN and a window MLP with a very large window are basically equivalent, since both can see far into the past." State precisely the two ways this claim is wrong: one about parameter count as the horizon grows, and one about the difference between what an RNN can represent and what it can train. Then describe a task on which the RNN's positional weight sharing (subsection two) is the wrong inductive bias, and explain why.
2. (Coding) Extend Code 10.1.1 to a many-to-many head: read out $\mathbf{y}_t = \mathbf{W}_y\mathbf{h}_t$ at every step and define the task as predicting, at each step, the sign of the running cumulative sum so far. Train it and report per-step accuracy as a function of position $t$. Verify empirically that accuracy degrades for later positions when you increase the sequence length past about 20, and connect the degradation to the $\gamma^{s}$ decay of the numeric example in subsection three.
3. (Analysis) Using the from-scratch cell, sweep the spectral radius of the recurrent matrix $\mathbf{W}_h$ at initialization (scale $\mathbf{W}_h$ by factors $0.5, 0.9, 1.0, 1.1, 1.5$) and, for a fixed long-lag task, measure (a) the norm of the gradient $\partial \mathcal{L}/\partial \mathbf{W}_h$ and (b) final test accuracy. Plot both against the scale factor. Explain the U-shaped or cliff-shaped curve you observe in terms of the vanishing/exploding-gradient argument, and identify which scale corresponds to the orthogonal initialization the practical example used.