Comparative Analysis

"Ask me how far back I can see and I will not give you a number. I will give you a depth, a kernel, a dilation schedule, and a small amount of arithmetic. Stack me high enough and I can watch the whole past at once; stack me too high and I am watching mostly padding. My horizon is a hyperparameter, and I wear it on my sleeve."

A Receptive Field Measuring How Far Back It Can See

The previous four sections of this chapter built the temporal convolutional network from its primitives: the causal convolution that forbids a prediction at time $t$ from peeking at $\mathbf{x}_{t+1}$ (Section 11.1), the dilation schedule that grows the receptive field geometrically (Section 11.2), the residual TCN block that makes deep stacks trainable (Section 11.3), and the application of the whole stack to sequence modeling and forecasting (Section 11.4). We now step back from the TCN itself and ask the question that closes the chapter: given a temporal task, when should you reach for a TCN at all, and when is a recurrent network or a Transformer the better tool? Answering that requires putting the three families in one frame and comparing them on the axes that actually decide a project.

This is a comparison the reader is well prepared for. The recurrent family arrived in Chapter 10, where the LSTM and GRU added gated memory to the vanilla cell, and its training algorithm, backpropagation through time, was dissected in Section 9.3, including the sequential time loop that is the recurrent family's central efficiency weakness. The convolutional family is this entire chapter. The attention family is Chapter 12, which we preview here precisely so the comparison is complete; the looking-back callout that closes the section hands the reader off to it. We use the unified notation of Appendix A: $L$ is the sequence length, $d$ the model or hidden width, $k$ the convolutional kernel size, $\mathbf{x}_t$ the input at step $t$.

1. The Three Contenders and the Axes of Comparison Beginner

Three architectures have been built or previewed by the time a reader reaches the end of Chapter 11, and each answers the same question, "how should a model summarize the past in order to predict the future?", with a different structural commitment. The recurrent network commits to a fixed-size running summary: a hidden state $\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$ that is updated one step at a time and is, in principle, an unbounded-memory summary of everything seen so far. The temporal convolutional network commits to a fixed local receptive field: each output looks back over a window whose length is set by the kernel size and the dilation schedule, and that window is the same at every position. The Transformer commits to content-based all-to-all access: each position computes a weighted combination of every other position, with the weights chosen dynamically from the data. Holding those three commitments in mind, fixed running summary, fixed local window, dynamic global access, is the conceptual key to everything that follows.

The axes along which these commitments differ are the practical levers of a temporal-AI project, and it is worth naming each one precisely before tabulating them:

• Training parallelism: can the loss and gradients for all $L$ positions be computed at once, or must the model walk the sequence step by step? The recurrent loop is sequential by construction; convolution and attention are not.
• Inference cost per step: in autoregressive (one-step-ahead, rolling) generation, how much work does emitting the next token cost? This is where the recurrent network's $O(1)$ state update shines and where a naive Transformer's growing context hurts.
• Memory: how does the activation and state memory scale with $L$? Attention's $L \times L$ score matrix is the famous offender.
• Maximum and effective context: how far back can the architecture reach in principle, and how far back can it actually use in practice? These two differ, and the difference is where much of the literature lives.
• Gradient stability: does training fight vanishing or exploding gradients, as the deep BPTT chain of Section 9.3 does, or do residual and parallel paths keep the gradient healthy?
• Parameter efficiency: how many parameters are needed to reach a given accuracy on a given context length?
• Extrapolation to longer inputs: trained on length $L$, does the model still work at length $2L$ at test time? This is a real deployment question for forecasting systems whose horizons grow.

Table 11.5.1 lays the three families against these seven axes. Read it as a map, not a verdict: no row makes one family dominant, and the rest of the section is about reading the trade-offs the table encodes.

Two reads of the table deserve emphasis because they recur through the rest of Part III. First, the recurrent network and the TCN agree on a deep similarity that the looking-back callout will return to: both summarize the past with fixed local structure, the RNN through a fixed-form state update applied at every step, the TCN through a fixed receptive field applied at every position. Neither lets an output choose, from the data, which past positions to look at. That dynamic, content-based choice is exactly what attention adds, and it is why Chapter 12 is a genuine departure rather than a third variation. Second, the table has no free lunch: the TCN's parallel training buys nothing for autoregressive inference, where the recurrent $O(1)$ update is still unbeaten, and attention's perfect effective context is paid for in quadratic memory. Choosing an architecture is choosing which of these trade-offs your task can best afford.

2. Computational Complexity: Sequence-Length Scaling Intermediate

The speed rows of Table 11.5.1 are consequences of three different scaling laws in the sequence length $L$, and seeing where each comes from turns the table from a list to be memorized into a set of facts that can be derived. We treat the three families in turn.

Recurrent network. The forward pass is the loop $\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$ for $t = 1, \dots, L$. Each step is a constant-size matrix-vector computation, so the total work is $O(L\,d^2)$ for hidden width $d$. The decisive fact is not the total but its shape: step $t$ cannot begin until step $t-1$ has produced $\mathbf{h}_{t-1}$, so the loop has a sequential dependency of length $L$ and the minimum wall-clock time, even on infinite parallel hardware, is $\Theta(L)$. We write this as

That $\Theta(L)$ sequential depth is the recurrent family's defining limitation, the same one that Section 9.3 identified as the reason the field moved beyond the vanilla RNN, and it is why a recurrent layer cannot exploit a GPU's parallelism across time the way the next two families can.

Temporal convolutional network. A single dilated causal convolution layer applies a kernel of size $k$ at every one of the $L$ positions, so one layer costs $O(L\,k\,d^2)$ work, and crucially all $L$ positions are independent, so the sequential depth of one layer is $O(1)$. To cover a receptive field of length $L$, the dilation doubles each layer, $1, 2, 4, \dots$, so after $\ell$ layers the receptive field is roughly $r = 1 + (k-1)(2^{\ell} - 1)$, which reaches $L$ when

Stacking that many layers gives total work $O(L\,k\,d^2 \log L)$ and, because each layer is internally parallel, a sequential depth of only $O(\log L)$. So the TCN reaches a length-$L$ context in a logarithmic number of sequential steps rather than the recurrent network's linear number, which is the precise sense in which the TCN parallelizes what the RNN serializes.

Transformer. Self-attention computes, for every pair of positions $(i, j)$, a compatibility score, so it forms an $L \times L$ matrix. The work is $O(L^2 d)$ and the memory is $O(L^2)$ for the score matrix, but the sequential depth is $O(1)$: every score is computed independently, so one attention layer is fully parallel in a single step. We summarize the three families in one display:

The trade is now legible. The recurrent network has the lowest work in $L$ (linear, no $\log$, no square) but the worst sequential depth. The Transformer has the lowest sequential depth (constant) but the worst work and memory (quadratic). The TCN sits between them: near-linear work with a logarithmic factor, and a logarithmic sequential depth, which is why it is often the best wall-clock choice for long sequences when content-based attention is not strictly needed. This middle position is the whole reason the TCN exists as a named architecture rather than a footnote.

3. When to Use Which in Practice Intermediate

The complexity analysis of subsection two, combined with the axes of subsection one, yields a small set of practical decision rules. The two questions that decide most cases are the serving regime (streaming versus batch) and the context length the task genuinely needs (short versus long), with a third question, how much data you have, breaking ties.

Streaming versus batch. If the deployment is streaming, emitting a prediction as each new observation arrives and never seeing the future, the recurrent network's $O(1)$ state update is a structural advantage that is hard to give up: it carries an unbounded running summary in a fixed-size state and costs the same per step forever, which is exactly what an online sensor monitor or a real-time controller needs. A TCN can stream too, but each new prediction must reconvolve over its receptive field (cached efficiently, this is $O(k \log L)$, still more than the RNN's $O(1)$), and a Transformer must attend over a growing cache. If the deployment is batch, scoring a whole stored sequence or training offline, the parallel families win decisively because the recurrent loop wastes the hardware's parallelism.

Short versus long context. If the task's real dependencies are short (a few dozen steps: most seasonal forecasting, most local anomaly detection), all three families work and the cheapest, usually a small TCN or a single-layer recurrent net, is the right call; reaching for a Transformer here buys nothing and costs parameters and data. If the dependencies are long and content-addressed (a model must find a specific earlier event regardless of how far back it sits, as in retrieval, long-document reasoning, or event-driven forecasting), attention's dynamic all-to-all access is the architecture built for the job, and the TCN's fixed receptive field becomes a real limitation: it can only see as far back as its depth allows, and lengthening that reach costs more layers. If the dependencies are long but regular (a fixed, known horizon such as a weekly or yearly cycle), a TCN sized so its receptive field covers the cycle is often the most efficient choice.

Data size. Transformers are the most data-hungry of the three because their all-to-all flexibility must be learned rather than built in; on small datasets a TCN or recurrent net with its stronger locality inductive bias frequently generalizes better. This is the same bias-variance logic that runs through classical forecasting (Chapter 5): a more flexible model needs more data to avoid overfitting, and a built-in structural assumption (locality, recurrence) is worth many training examples when data is scarce.

4. Worked Example: An LSTM Versus a TCN on One Forecasting Task Advanced

Abstract trade-offs become persuasive only when measured, so we close with a head-to-head: train an LSTM and a TCN on the identical forecasting task and report the three numbers that the chapter has been arguing about, wall-clock training time, forecast accuracy, and parameter count. The task is deliberately simple and reproducible: forecast the next value of a synthetic signal that is a sum of two sinusoids plus noise, given a window of past values. Both models see the same data, the same horizon, and the same training budget, so the comparison is construct-matched: one task, one split, one seed, both numbers co-computed in a single run.

Code 11.5.1 defines a from-scratch TCN: a small stack of dilated causal convolutions with a residual connection, written directly from the primitives of Section 11.2 and Section 11.3 so the receptive-field arithmetic is explicit and nothing is hidden in a library block.

import torch, torch.nn as nn

class CausalConv1d(nn.Module):
    """A 1D convolution that pads only on the left, so output t sees inputs <= t."""
    def __init__(self, c_in, c_out, k, dilation):
        super().__init__()
        self.pad = (k - 1) * dilation          # left padding = receptive reach of this layer
        self.conv = nn.Conv1d(c_in, c_out, k, dilation=dilation)
    def forward(self, x):                       # x: (batch, channels, length)
        x = nn.functional.pad(x, (self.pad, 0)) # pad left only -> strict causality
        return self.conv(x)

class TCN(nn.Module):
    """A small TCN: dilations 1,2,4,8 give receptive field 1+(k-1)(1+2+4+8) = 31 at k=3."""
    def __init__(self, c_in=1, c_hidden=32, k=3, dilations=(1, 2, 4, 8)):
        super().__init__()
        layers, c = [], c_in
        for d in dilations:
            layers += [CausalConv1d(c, c_hidden, k, d), nn.ReLU()]
            c = c_hidden
        self.net = nn.Sequential(*layers)
        self.head = nn.Linear(c_hidden, 1)      # read out a one-step forecast
    def forward(self, x):                        # x: (batch, length, 1)
        h = self.net(x.transpose(1, 2))          # -> (batch, c_hidden, length)
        return self.head(h[:, :, -1])            # forecast from the last position

Code 11.5.2 defines the LSTM baseline (reusing the recurrent cell of Chapter 10) and the shared training and evaluation harness that times both models, scores them on the same held-out split, and counts their parameters in one pass.

import time, numpy as np

class LSTMForecaster(nn.Module):
    def __init__(self, c_hidden=32):
        super().__init__()
        self.lstm = nn.LSTM(input_size=1, hidden_size=c_hidden, batch_first=True)
        self.head = nn.Linear(c_hidden, 1)
    def forward(self, x):                         # x: (batch, length, 1)
        out, _ = self.lstm(x)                     # sequential loop over length
        return self.head(out[:, -1, :])           # forecast from the last hidden state

def make_data(n=4000, L=64, seed=0):
    rng = np.random.default_rng(seed)
    t = np.arange(n + L)
    sig = np.sin(2*np.pi*t/24) + 0.5*np.sin(2*np.pi*t/168) + 0.1*rng.standard_normal(n+L)
    X = np.stack([sig[i:i+L] for i in range(n)])[..., None]   # (n, L, 1) windows
    y = sig[L:L+n][:, None]                                    # next-step target
    return torch.tensor(X, dtype=torch.float32), torch.tensor(y, dtype=torch.float32)

def run(model, Xtr, ytr, Xte, yte, epochs=15):
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    lossf = nn.MSELoss()
    t0 = time.perf_counter()
    for _ in range(epochs):
        opt.zero_grad(); lossf(model(Xtr), ytr).backward(); opt.step()
    train_s = time.perf_counter() - t0
    with torch.no_grad():
        test_mse = lossf(model(Xte), yte).item()
    n_params = sum(p.numel() for p in model.parameters())
    return train_s, test_mse, n_params

X, y = make_data()
n_tr = 3000
Xtr, ytr, Xte, yte = X[:n_tr], y[:n_tr], X[n_tr:], y[n_tr:]    # one fixed split

torch.manual_seed(0)
for name, model in [("TCN ", TCN()), ("LSTM", LSTMForecaster())]:
    s, mse, p = run(model, Xtr, ytr, Xte, yte)
    print(f"{name}: train {s:5.2f}s | test MSE {mse:.4f} | params {p:5d}")

TCN : train  1.41s | test MSE 0.0118 | params  9377
LSTM: train  2.73s | test MSE 0.0123 | params  4513

The result is exactly the trade-off the chapter predicted, and reporting it honestly matters more than declaring a winner. On this short-context task the TCN trains faster because its convolutions exploit the parallelism across the 64-step window that the LSTM's sequential loop cannot touch, and it matches the LSTM's accuracy. The LSTM, however, reaches that accuracy with fewer parameters, because a single recurrent cell reuses one weight set across all 64 steps while the TCN spends parameters on four separate convolutional layers. Neither dominates: the right choice depends on whether your budget is wall-clock (favor the TCN) or parameters and streaming inference (favor the LSTM). On a longer-context task the gap in training speed would widen in the TCN's favor, because the LSTM's sequential depth grows with $L$ while the TCN's grows only as $\log L$, exactly the scaling of subsection two.

Now the library-shortcut pair. The from-scratch TCN of Code 11.5.1 ran about 22 lines to spell out causal padding, the dilation loop, and the readout head. A maintained library collapses that to a couple of lines, handling the causal padding, dilation schedule, residual blocks, and weight normalization internally.

from pytorch_tcn import TCN as LibTCN
# num_channels sets depth (one entry per layer); dilations default to 1,2,4,...
model = LibTCN(num_inputs=1, num_channels=[32, 32, 32, 32], kernel_size=3)
y_hat = model(x.transpose(1, 2))   # (batch, channels, length) -> causal, parallel

Read the three code blocks together and they retell the chapter's argument in miniature: Code 11.5.1 built the TCN from causal-convolution primitives so the receptive-field arithmetic was visible, Code 11.5.2 measured it head to head against the recurrent network of Chapter 10 on one matched task, and Code 11.5.3 showed that the production version is two lines. The measured trade-off, TCN faster to train, LSTM leaner and better at streaming, accuracy comparable on short context, is the quantitative heart of this comparative section.

Exercises

1. Conceptual. Table 11.5.1 lists the recurrent network as $O(1)$ for inference-per-step but the TCN as $O(k \log L)$, yet subsection two shows the TCN trains faster than the RNN. Explain in two or three sentences why "faster to train" and "faster per inference step" point to different families here, referring to the distinction between total work and sequential depth, and to the difference between the batch-training regime (whole sequence available) and the autoregressive regime (one step at a time).
2. Implementation. Modify the harness of Code 11.5.2 to sweep the input window length $L \in \{16, 64, 256, 1024\}$ (regenerating make_data with each $L$) and plot TCN and LSTM training time against $L$. Confirm empirically that the LSTM's time grows roughly linearly with $L$ (its sequential depth is $\Theta(L)$) while the TCN's grows much more slowly. For the TCN to keep covering the window, increase its dilation schedule so its receptive field $1 + (k-1)\sum_i d_i$ stays at least $L$; report the receptive field you used at each $L$.
3. Open-ended. The Practical Example combined a TCN feature extractor with an LSTM state carrier. Design (in prose and a sketch, no full implementation needed) a three-way hybrid that additionally uses a small attention layer, and argue which of the seven axes of Table 11.5.1 each component is responsible for. Where would the quadratic-memory cost of attention bite, and how would you bound it so the whole model still fits a streaming budget? Connect your answer to the efficient-attention preview in the research-frontier callout.