Neural Sequence Modeling Fundamentals

Step 1: Build the sliding-window MLP forecaster and find its window edge

Start with the lag-window multilayer perceptron of Section 9.1, the nonlinear autoregression. Flatten a fixed window of recent timesteps into a vector and feed it to a small MLP that predicts the target. Train it on the delayed-copy task with the injected value sitting inside the window, and confirm it solves the task easily. Then move the value one step beyond the window's edge and watch the score collapse: a fixed-window model cannot see a lag it does not include, which is the structural reason a model needs to carry a state rather than re-read a fixed slice.

class WindowMLP(nn.Module):
    def __init__(self, window, hidden=64):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(window, hidden), nn.ReLU(),
                                 nn.Linear(hidden, 1))
    def forward(self, x):                            # x: (batch, window, 1)
        return self.net(x.squeeze(-1))               # flatten the window, predict the target

Step 2: Frame the task and build a vanilla RNN

Cast the delayed-copy problem as the many-to-one sequence task of Section 9.2: a variable-length input sequence in, a single value out, read from the final hidden state. Build a minimal vanilla recurrent network that carries a hidden state across the whole sequence, the trainable form of the filter recurrence from Section 7.6. Unlike the windowed MLP, this model has, in principle, an unbounded reach: its hidden state can hold the injected value for as long as it learns to.

class VanillaRNN(nn.Module):
    def __init__(self, hidden=64):
        super().__init__()
        self.rnn = nn.RNN(input_size=1, hidden_size=hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)
    def forward(self, x):                            # x: (batch, length, 1)
        out, h = self.rnn(x)                         # carry a state across every timestep
        return self.head(h[-1])                       # read the target from the last hidden state

Step 3: Train with backpropagation through time and read per-timestep gradients

Train the recurrent network with backpropagation through time, the unrolled chain rule of Section 9.3. PyTorch autograd performs the unrolling for you, but you can expose its inner workings: retain the gradient on the hidden state at each timestep and, after one backward pass, read the gradient norm flowing into each step. This is the raw material for the measurement that makes the chapter concrete, the message the early timesteps actually receive.

model = VanillaRNN()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
x, y = delayed_copy(256, length=40, gap=35)
loss = nn.functional.mse_loss(model(x), y)
loss.backward()                                      # BPTT: chain rule across all 40 steps
# inspect grad norms per timestep via hooks on the unrolled hidden states

Step 4: Measure and plot the gradient-norm decay

Now make the pathology visible, following Section 9.4. Sweep the gap of the delayed-copy task from short to long, and for each gap record the norm of the gradient of the final loss with respect to the hidden state as a function of how many steps back you go. Plot gradient norm against distance on a log scale. You will see a near-straight line sloping down, the exponential decay set by the recurrent Jacobian's spectral radius, and you will read off, concretely, that the gradient from twenty or thirty steps back is orders of magnitude weaker than the gradient from one step back. The early timesteps barely hear the correction, which is exactly why the network fails to learn the long copy.

Step 5: Apply the toolkit and confirm the limit

Reach into the Section 9.4 toolkit and see how far it gets you. Add gradient clipping to cap the rare exploding update, and initialize the recurrent weight matrix orthogonally so its spectral radius starts near one, slowing the vanishing. Retrain and re-measure: the decay curve flattens somewhat and training is steadier, yet on the longest gaps the vanilla cell still cannot reliably carry the value to the end. Record where it breaks. That ceiling is the lab's punchline and the bridge to the next chapter: clipping and orthogonal init buy you some range, but a fundamentally different mechanism, a protected gated carry path, is needed to cross the longest gaps, and that mechanism is the subject of Chapter 10.

nn.init.orthogonal_(model.rnn.weight_hh_l0)         # spectral radius near one slows vanishing
# inside the training loop, after loss.backward():
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # tame exploding gradients