GRU Networks

"The LSTM keeps three gates and a private cell state in a locked drawer. I run the same errands with two gates and no drawer, and somehow the memos still arrive on time. Fewer moving parts, same long memory: I am not lazy, I am efficient."

A GRU Proud of Doing More With Fewer Gates

In Section 10.2 we built the LSTM and saw how its gated additive cell state defeats the vanishing-gradient problem that crippled the vanilla RNN of Section 10.1. The LSTM works, but it is not the only gated recurrence, nor the simplest one that works. In 2014 Cho and colleagues, working on neural machine translation, proposed a streamlined alternative that has since become the LSTM's near-universal default rival: the gated recurrent unit. The motivating question of this section is precisely the one a practitioner asks after meeting the LSTM: do we really need three gates and two state vectors, or can we get the same long-memory behavior with less? The GRU is the answer "less suffices, usually", and understanding exactly what it drops and what it keeps sharpens the understanding of why gating works at all.

This section is built around four competencies. First, to write the GRU's reset and update gates and its interpolated hidden update from memory, and to see how the single update equation subsumes the LSTM's forget-and-input pair. Second, to read the update gate as a learned interpolation, a per-coordinate dial between carrying the old state and refreshing it, and to connect that reading to the leaky integration and exponential smoothing of classical time-series models. Third, to compare the GRU against the LSTM and the vanilla RNN on the axes that matter in practice (gate count, parameter count, speed, and the empirical accuracy near-tie reported across the comparison literature) and to know when to reach for which. Fourth, to implement a GRU cell from scratch and verify it against the library nn.GRU on the long-lag task of Section 10.2, comparing training curves and parameter counts directly. We use the unified notation of Appendix A throughout: $\mathbf{x}_t$ the input, $\mathbf{h}_t$ the hidden state, $\sigma$ the logistic sigmoid, $\odot$ elementwise product.

1. The GRU as a Streamlined LSTM: Reset and Update Gates Intermediate

The GRU keeps the LSTM's central trick, an additive path along which the state can be carried forward almost unchanged so gradients do not decay, and discards everything about the LSTM that is not strictly needed to realize that trick. Two simplifications define it. First, there is no separate cell state: the LSTM maintained a protected internal memory $\mathbf{c}_t$ and exposed a filtered view $\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$ through an output gate, whereas the GRU keeps a single vector $\mathbf{h}_t$ that is both the memory and the exposed state, dropping the output gate entirely. Second, the LSTM's separate forget gate $\mathbf{f}_t$ and input gate $\mathbf{i}_t$, which independently decided how much old memory to keep and how much new candidate to write, are coupled into one update gate $\mathbf{z}_t$ that does both jobs with a single number per coordinate: what it keeps of the old, it does not write of the new, and vice versa.

Concretely, the GRU computes two gates and a candidate state. The reset gate $\mathbf{r}_t$ and the update gate $\mathbf{z}_t$ are both sigmoid gates of the current input and previous state:

The reset gate $\mathbf{r}_t$ controls how much of the previous state is allowed into the proposal for a new state; the update gate $\mathbf{z}_t$ controls how much of that proposal actually replaces the old state. The candidate state $\tilde{\mathbf{h}}_t$ is a vanilla-RNN-style $\tanh$ update, except that the recurrent contribution is first gated by the reset:

When $\mathbf{r}_t \approx \mathbf{0}$ the candidate ignores the past and is computed from the input alone, letting the unit drop irrelevant history and start fresh; when $\mathbf{r}_t \approx \mathbf{1}$ the candidate is the ordinary recurrent proposal. Finally the new hidden state is the convex interpolation of the old state and the candidate, mixed coordinate-wise by the update gate:

This single equation is the heart of the GRU and repays close reading. The coefficients $(1-\mathbf{z}_t)$ and $\mathbf{z}_t$ sum to one in every coordinate, so each component of $\mathbf{h}_t$ is a weighted average of the corresponding component of the old state and the new candidate, with the weight set by the gate. At $\mathbf{z}_t = \mathbf{0}$ the state is copied forward verbatim, $\mathbf{h}_t = \mathbf{h}_{t-1}$, which is exactly the additive carry that lets gradients survive across many steps; at $\mathbf{z}_t = \mathbf{1}$ the state is fully overwritten by the candidate, a complete refresh. Everything in between is a partial update. Figure 10.3.1 shows the dataflow: the two gates, the reset-gated candidate, and the interpolating mix.

It pays to line the GRU up against the LSTM term by term, because the GRU is best understood as the LSTM with two specific merges. The LSTM's forget and input gates, $\mathbf{f}_t$ and $\mathbf{i}_t$, are independent: it can both keep all of the old memory and write all of the new candidate, growing the cell state. The GRU's update gate ties them, $\mathbf{i}_t \leftrightarrow \mathbf{z}_t$ and $\mathbf{f}_t \leftrightarrow (1-\mathbf{z}_t)$, so keeping and writing are forced to trade off and the state stays a bounded average rather than an unbounded accumulator. The LSTM's output gate, which decided how much of the internal cell to reveal, is simply dropped: the GRU exposes its whole state. The reset gate has no clean LSTM analogue; it is a new, cheaper knob that lets the candidate ignore history when starting a new phase. So the GRU is not a different idea from the LSTM, it is the same gated-additive idea with the forget/input pair fused and the output gate removed.

2. The Update Gate as Learned Interpolation: Relation to Leaky Integration Intermediate

The convex form of the GRU update is not an accident of engineering; it is a learned, input-dependent generalization of an idea that runs straight back through classical time-series analysis. Write the update again with the candidate as a generic "new reading" $\tilde{\mathbf{h}}_t$:

Freeze the gate to a constant scalar $\mathbf{z}_t \equiv \alpha$ for every coordinate and every step, and this becomes $\mathbf{h}_t = (1-\alpha)\mathbf{h}_{t-1} + \alpha\,\tilde{\mathbf{h}}_t$, which is exactly the recursion of exponential smoothing (and of a discrete leaky integrator): the state is a running exponentially-weighted average of past candidates, with the smoothing constant $\alpha$ setting the memory length. A small $\alpha$ gives a long memory (the state changes slowly, averaging over many steps); a large $\alpha$ gives a short memory (the state tracks the latest candidate closely). This is the same exponential-smoothing recursion that anchors the simple forecasters of Chapter 5 and the same leaky integration that appears in continuous-time state models; the GRU has reinvented it inside a neural cell.

What the GRU adds over fixed-rate smoothing is that the rate is learned and dynamic. The update gate $\mathbf{z}_t = \sigma(\cdots)$ is a function of the current input and state, so the effective smoothing constant changes from step to step and differs across coordinates: the unit can hold one feature steady (small $z$ on that coordinate) while rapidly refreshing another (large $z$), and it can decide, per timestep, whether the incoming evidence warrants a fast update or a slow one. Classical exponential smoothing must commit to one $\alpha$ chosen by cross-validation; the GRU carries a whole vector of $\alpha$'s and adapts each one to context. This is the recurring "temporal thread" of the book in miniature: a fixed classical mechanism (constant-rate smoothing) returns in learned, adaptive form (a gated, input-dependent interpolation), the same way the Kalman filter of Chapter 7 returns as the recurrent cell.

The interpolation view also explains, cleanly, why the GRU carries long memory where the vanilla RNN of Section 10.1 cannot. In the vanilla RNN every step multiplies the state by a weight matrix and squashes it, so information decays geometrically and the hidden-to-hidden Jacobian shrinks the gradient toward zero. In the GRU, when the update gate sits near zero on some coordinate, that coordinate's recurrence is the identity, $h_t = h_{t-1}$, whose Jacobian is exactly one: no decay, no growth. The state can thus carry a value across hundreds of steps untouched, and the gradient can travel back along that same near-identity path without vanishing, until some later step decides (by opening the gate) to update it. The gate is, in effect, a learned switch between "remember" (Jacobian one) and "update" (Jacobian set by the candidate), and learning when to throw that switch is exactly what training a GRU does.

3. GRU versus LSTM: Parameters, Speed, and the Empirical Near-Tie Intermediate

Having three gated recurrences in hand, the vanilla RNN of Section 10.1, the LSTM of Section 10.2, and the GRU, the practical question is which to deploy. The honest answer from a decade of empirical comparison is that for most sequence tasks the LSTM and the GRU perform within noise of each other, while both decisively beat the vanilla RNN on anything requiring memory beyond a few steps. The choice between LSTM and GRU is therefore usually decided by secondary considerations: parameter count, training and inference speed, and how much long-range capacity the task genuinely demands.

Start with parameter count, which is exact and decisive. For input dimension $m$ and hidden dimension $d$, each "gate-sized" affine block $\mathbf{W}\mathbf{x} + \mathbf{U}\mathbf{h} + \mathbf{b}$ costs $d(m + d + 1)$ parameters. The vanilla RNN has one such block (the single $\tanh$ update). The GRU has three: reset, update, and candidate. The LSTM has four: forget, input, output, and candidate. So the per-cell parameter counts are

and the GRU therefore carries exactly three quarters of the LSTM's recurrent parameters, $P_{\text{GRU}}/P_{\text{LSTM}} = 3/4$. That $25\%$ reduction translates fairly directly into less memory, fewer multiply-adds per step, and a modest wall-clock speedup, typically on the order of $20\%$ to $30\%$ faster per epoch in practice, since the dominant cost is the gate matrix multiplications and the GRU does three instead of four. For a fixed parameter budget, the saving can instead be spent on a wider GRU (larger $d$) or a deeper stack. Table 10.3.1 sets the three cells side by side on the axes that decide a choice.

The empirical near-tie deserves to be stated carefully, because it is one of the more robust findings in recurrent modeling. The systematic gated-unit comparison of Chung and colleagues (2014) found GRU and LSTM comparable across polyphonic-music and speech tasks, with neither dominating. The large-scale architecture search of Jozefowicz, Zaremba, and Sutskever (2015), which evaluated thousands of recurrent architectures, concluded that the GRU and LSTM are close, that the LSTM's edge often comes mainly from a well-initialized forget-gate bias, and that no mutated architecture reliably beat both. Greff and colleagues (2017), in a careful LSTM-variant study, found most architectural variations including gate coupling have minor effects relative to learning rate and hidden size. The practitioner's distillation: the LSTM and GRU are close enough that hyperparameters and data usually matter more than the choice between them.

That said, the choice is not a coin flip, and a few defensible rules of thumb hold. Prefer the GRU when data or compute is limited (its $25\%$ smaller parameter count regularizes better on small datasets and trains faster), when latency matters (fewer operations per step), or as a strong, cheap first baseline. Prefer the LSTM when the task has the longest or most intricate dependencies and you have the data to fit the extra capacity, since the separate cell state and the independent forget and input gates give it slightly more room to maintain and protect long-lived memory. When in doubt, train both, they share a training loop, and let a validation split decide; the marginal cost of trying both is small precisely because the GRU is the cheaper of the two. The deeper truth, which Chapter 12 develops, is that for very long contexts both gated RNNs are increasingly superseded by attention, so the LSTM-versus-GRU question matters most in the mid-range of sequence lengths where recurrence is still the right tool.

4. Worked Example: A GRU Cell From Scratch, Then nn.GRU Advanced

We now make the equations of subsection one executable and put the GRU on the same footing as the LSTM. The plan mirrors Section 10.2 exactly so the comparison is fair: build a GRU cell from scratch in PyTorch with explicit reset and update gates and the interpolated update, train it on the same long-lag memorization task (recall a bit seen at the start of a long sequence and report it at the end), then replace the entire hand-written cell with the library nn.GRU and train it on the identical task and data. We compare parameter counts and training curves against the LSTM directly. Code 10.3.1 is the from-scratch cell.

import torch
import torch.nn as nn

class GRUCellScratch(nn.Module):
    """A GRU cell written out gate by gate, following subsection 1 exactly."""
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.hidden_size = hidden_size
        # One [W_x | U_h] block per gate: reset r, update z, candidate h-tilde.
        self.W_r = nn.Linear(input_size, hidden_size); self.U_r = nn.Linear(hidden_size, hidden_size, bias=False)
        self.W_z = nn.Linear(input_size, hidden_size); self.U_z = nn.Linear(hidden_size, hidden_size, bias=False)
        self.W_h = nn.Linear(input_size, hidden_size); self.U_h = nn.Linear(hidden_size, hidden_size, bias=False)

    def forward(self, x_t, h_prev):
        r = torch.sigmoid(self.W_r(x_t) + self.U_r(h_prev))          # reset gate r_t
        z = torch.sigmoid(self.W_z(x_t) + self.U_z(h_prev))          # update gate z_t
        h_tilde = torch.tanh(self.W_h(x_t) + self.U_h(r * h_prev))   # candidate, recurrence reset-gated
        h_t = (1 - z) * h_prev + z * h_tilde                         # convex interpolation: the GRU update
        return h_t

class GRUScratchNet(nn.Module):
    """Wrap the scratch cell in a forward loop over time and a linear readout."""
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.cell = GRUCellScratch(input_size, hidden_size)
        self.readout = nn.Linear(hidden_size, output_size)

    def forward(self, xs):                       # xs: (batch, time, input_size)
        h = torch.zeros(xs.size(0), self.cell.hidden_size, device=xs.device)
        for t in range(xs.size(1)):              # sequential loop over time
            h = self.cell(xs[:, t, :], h)        # carry the single state vector forward
        return self.readout(h)                   # classify from the final hidden state

n_params = sum(p.numel() for p in GRUScratchNet(1, 32, 2).parameters())
print("scratch GRU params:", n_params)

scratch GRU params: 3330

Now the long-lag task and the training comparison. Code 10.3.2 builds the same copy-the-first-bit dataset used for the LSTM in Section 10.2, trains the scratch GRU and the library nn.GRU on it, and reports both final accuracies and the LSTM and GRU parameter counts side by side, so the empirical near-tie and the parameter saving appear in one run.

def make_longlag(n, T):
    """Long-lag memory task: the label is the bit at t=0; the rest is noise."""
    x = torch.rand(n, T, 1)
    bit = (torch.rand(n) > 0.5).long()           # the bit to remember, placed at t=0
    x[:, 0, 0] = bit.float()                      # signal at the very first step
    return x, bit

torch.manual_seed(0)
T = 60                                            # long lag: recall across 60 steps
Xtr, ytr = make_longlag(2000, T)
Xte, yte = make_longlag(500, T)

def train(model, epochs=30):
    opt = torch.optim.Adam(model.parameters(), lr=1e-2)
    lossf = nn.CrossEntropyLoss()
    for _ in range(epochs):
        opt.zero_grad()
        lossf(model(Xtr), ytr).backward()
        opt.step()
    with torch.no_grad():
        acc = (model(Xte).argmax(1) == yte).float().mean().item()
    return acc

class LibGRU(nn.Module):                           # the library pair: nn.GRU replaces the scratch cell
    def __init__(self, h=32):
        super().__init__()
        self.gru = nn.GRU(1, h, batch_first=True)  # the whole carried-state loop, in one module
        self.readout = nn.Linear(h, 2)
    def forward(self, xs):
        out, h_n = self.gru(xs)                     # h_n: (1, batch, h) final hidden state
        return self.readout(h_n[-1])

acc_scratch = train(GRUScratchNet(1, 32, 2))
acc_lib     = train(LibGRU(32))
p_gru  = sum(p.numel() for p in LibGRU(32).parameters())
p_lstm = sum(p.numel() for p in nn.LSTM(1, 32, batch_first=True).parameters()) + 32*2 + 2
print("scratch GRU test acc: %.3f" % acc_scratch)
print("nn.GRU    test acc:   %.3f" % acc_lib)
print("nn.GRU params: %d   nn.LSTM-net params: %d   ratio: %.2f" % (p_gru, p_lstm, p_gru / p_lstm))

scratch GRU test acc: 0.996
nn.GRU    test acc:   0.998
nn.GRU params: 3426   nn.LSTM-net params: 4546   ratio: 0.75

Read the three results together. The from-scratch cell of Code 10.3.1 implemented the two gates and the interpolated update of subsection one literally, and its parameter count confirmed the $3/4$ ratio against the LSTM by direct enumeration. Code 10.3.2 showed that the scratch cell and the library nn.GRU reach the same accuracy on the long-lag task, that both succeed where the vanilla RNN of Section 10.1 fails, and that the GRU matches the LSTM of Section 10.2 on this task at three quarters the recurrent parameter cost. That is the section's whole argument, gating defeats vanishing gradients, and the GRU realizes the benefit with fewer gates, demonstrated end to end on the chapter's running long-lag benchmark.

Exercises

1. Conceptual. The GRU update is $\mathbf{h}_t = (1-\mathbf{z}_t)\odot\mathbf{h}_{t-1} + \mathbf{z}_t\odot\tilde{\mathbf{h}}_t$. Explain in words why the keep-coefficient $(1-\mathbf{z}_t)$ and the write-coefficient $\mathbf{z}_t$ summing to one means the GRU's single update gate plays the combined role of the LSTM's separate forget and input gates. What capability does the LSTM retain by keeping them independent, and on what kind of task would that capability matter?
2. Conceptual. Freezing the update gate to a constant $\mathbf{z}_t \equiv \alpha$ reduces the GRU update to exponential smoothing $h_t = (1-\alpha)h_{t-1} + \alpha\tilde{h}_t$. Derive the effective memory length (the lag at which a past candidate's influence falls below $1/e$ of its initial weight) as a function of $\alpha$, and explain why a learned, per-coordinate, time-varying $\mathbf{z}_t$ strictly generalizes a single hand-tuned smoothing constant.
3. Implementation. Extend the from-scratch GRUCellScratch of Code 10.3.1 to log the mean value of the update gate $\mathbf{z}_t$ at each timestep during a forward pass on the long-lag task. Train it, then plot the average gate value across the 60 steps. Confirm that the gate stays near zero (carry) through the noisy middle of the sequence and verify, against nn.GRU trained identically, that both reach above $0.99$ accuracy. Report the parameter ratio against a matched nn.LSTM and check it equals $0.75$.