Temporal Convolutional Networks (TCN)

"While my recurrent colleague processes its sequence one reluctant timestep after another, waiting for step nine hundred ninety-nine before it dares touch step one thousand, I have already convolved the entire history at once, on every position in parallel, and I am sitting here finished, drumming my filters. We started together. I am bored. It is still on step forty."

A TCN Training in Parallel While the RNN Waits

In Section 11.1 we introduced one-dimensional convolution over a time axis and made it causal by padding only on the left, so the output at time $t$ depends on inputs at times $\le t$ and never on the future. In Section 11.2 we added dilation, inserting gaps between the taps of the kernel so that a fixed kernel size reaches exponentially further back as the dilation grows. Those two ideas, causality and dilation, are the raw material. This section assembles them into the full architecture that turned convolution from a building block into a competitive alternative to the recurrent networks of Chapter 10. We continue to use the unified notation of Appendix A: $\mathbf{x}_{1:T}$ the input sequence, $\mathbf{h}^{(\ell)}_t$ the activation at layer $\ell$ and time $t$, $d$ the channel width, $k$ the kernel size.

The reason this architecture earns a section of its own, rather than a remark that "you can also use convolutions", is historical and practical at once. For most of the 2010s the default answer to "model this sequence" was an LSTM, and convolutions were thought of as the image people's tool. The Bai, Kolter and Koltun paper, provocatively titled "An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling", showed that a simple, almost boringly uniform convolutional architecture, the TCN, outperformed canonical LSTMs and GRUs on a wide suite of sequence tasks while training several times faster and using less memory. That result reframed the field: the recurrence was not sacred, and the sequential bottleneck of Section 9.3 was a cost one could simply refuse to pay. The lesson echoes forward to the attention models of Chapter 12 and the parallel-scan state-space models of Chapter 13, both of which keep the TCN's commitment to parallel training while relaxing its fixed receptive field.

By the end of this section you will be able to draw a TCN residual block and name every component, explain why dilation $2^{\ell}$ at layer $\ell$ gives a receptive field that grows exponentially in depth, articulate the three reasons TCNs often beat RNNs and the three things they give up in exchange, and implement and train a TCN from scratch, reproducing in miniature the speed advantage that made the architecture famous.

1. The TCN Architecture: Stacked Causal Dilated Residual Blocks Intermediate

A TCN is, from a distance, almost embarrassingly regular: it is a stack of identical residual blocks, each block doing the same thing, the only parameter that changes from block to block being the dilation. There is no gate, no recurrence, no attention, none of the machinery that makes other sequence models intricate. The art is entirely in how three plain ingredients (a causal convolution, a dilation schedule, and a residual connection) combine to give a model that is parallel, stable, and far-reaching. We take the ingredients one at a time and then assemble the block.

The base operation is the causal dilated convolution from Sections 11.1 and 11.2. A one-dimensional convolution with kernel size $k$ and dilation factor $\delta$ computes its output at time $t$ as a weighted sum of inputs spaced $\delta$ apart, reaching back $\delta\,(k-1)$ steps:

where $\mathbf{f}_0, \dots, \mathbf{f}_{k-1}$ are the kernel weights and the indices $t - \delta i$ are all $\le t$, which is exactly the causality requirement: the output at $t$ touches only the present and the past. Setting $\delta = 1$ recovers an ordinary convolution; setting $\delta = 2, 4, 8$ skips ever-wider gaps. The decisive design choice of the TCN is the dilation schedule: stack blocks with dilation $\delta_\ell = 2^{\ell}$ at depth $\ell = 0, 1, 2, \dots$, so that the receptive field, the number of input steps the top of the stack can see, grows exponentially with depth. With $L$ layers, kernel $k$, and dilations doubling each layer, the receptive field is

for a residual TCN with two convolutions per block. This is the single-convolution formula $R = 1 + (k-1)(2^{L}-1)$ of Section 11.2 multiplied by a factor of 2, and the factor of 2 is there because each residual block here contains two dilated convolutions at the same dilation rather than one, so the two formulas are consistent rather than contradictory. The exponential $2^L$ is the whole point: doubling the depth roughly doubles the reachable history, so a 10-layer stack with kernel 3 already sees on the order of a couple of thousand steps. Compare this to a plain (undilated) convolutional stack, whose receptive field grows only linearly in depth and which would need hundreds of layers to reach the same horizon. Dilation buys exponential reach for the price of logarithmic depth.

The third ingredient is the residual block, the unit that is actually stacked. A single TCN residual block applies, to its input, the following sequence twice and then adds a shortcut:

Reading the block from the inside out: the dilated causal convolution mixes information across time within the layer's receptive reach; weight normalization (Salimans and Kingma, 2016) reparameterizes each filter as a direction times a learned scale, which stabilizes and accelerates training much as batch normalization does but without depending on batch statistics, a real advantage for the variable-length sequences common in forecasting; the ReLU supplies the nonlinearity; and dropout regularizes, applied here in a spatial form that zeroes whole channels consistently across the time axis. This four-step pipeline runs twice inside one block, deepening the per-block transformation. Finally the residual connection $\mathcal{R}(\mathbf{x})$ adds the block's input back to its output, exactly the identity-shortcut idea of ResNets: it gives gradients a direct path around the block (so very deep stacks train), and it lets a block learn a small refinement of its input rather than a full remapping. Because the convolutions may change the channel count, $\mathcal{R}$ is an optional $1\times1$ convolution that matches input channels to output channels when they differ, and the identity otherwise. Figure 11.3.1 draws one block with every component labeled.

Assembling the full network is now trivial because the block is the only repeated unit: pick a kernel size $k$ (commonly 2 or 3), a channel width $d$, and a number of levels $L$, then stack $L$ residual blocks with dilations $1, 2, 4, \dots, 2^{L-1}$. Choose $L$ so the receptive field $R$ of the formula above covers the longest dependency the task requires, and you have a model whose entire forward pass is a sequence of convolutions, every one of which is computed at all time positions simultaneously. Note the contrast with the Elman recurrence of Section 9.3, where $\mathbf{h}_t$ could not be computed until $\mathbf{h}_{t-1}$ existed: here there is no such dependency, and that single fact is the engine of everything in the next subsection.

Two details of the block reward a closer look because they are where careless implementations go wrong. The first is the enforcement of causality. A standard convolution with kernel size $k$ and dilation $\delta$ centered on position $t$ would read $\delta(k-1)/2$ steps into the future, which for a forecaster is catastrophic: it leaks the answer. The fix is asymmetric padding. We left-pad the input by the full $(k-1)\delta$ steps and apply no right padding, which shifts every output so that position $t$ aligns with inputs $t, t-\delta, t-2\delta, \dots$ and nothing later; a framework that pads symmetrically must then trim the surplus right-hand outputs, the operation the code below calls a chomp. Get this wrong by even one step and the network trains beautifully on a future it would never see at inference, the temporal leakage warned against in Chapter 2.

The second detail is weight normalization on the convolution. Splitting each filter into a unit-norm direction and a separately learned scalar magnitude decouples the length of the weight vector from its orientation, which conditions the optimization landscape and, unlike batch normalization, introduces no dependence on other examples in the batch and no train-versus-inference discrepancy, both of which matter when sequences vary in length and batches are small.

The choice of how wide and how deep to make the stack is the practitioner's main lever, and the two dimensions play different roles. Depth $L$ sets the receptive field through the exponential dilation schedule, so it is dictated almost entirely by the longest dependency the task contains: pick the smallest $L$ whose $R$ clears that horizon. Channel width $d$ sets the representational capacity at each position, how much information the layer can carry about its receptive window, and is tuned like any hidden dimension, traded against overfitting and compute. A useful mental model is that depth buys reach and width buys richness, and because reach grows exponentially in depth, you almost never need many layers; a TCN that sees thousands of steps is typically only ten to twelve blocks deep, which is part of why it trains so cheaply.

2. Why TCNs Often Match or Beat RNNs Intermediate

The Bai, Kolter and Koltun (2018) evaluation was deliberately unflattering to its own architecture: it pitted a generic, lightly tuned TCN against carefully tuned LSTMs and GRUs across a broad suite of canonical sequence-modeling benchmarks (the adding problem, sequential and permuted MNIST, copy-memory, polyphonic music modeling, character-level and word-level language modeling). Across most of these tasks the plain TCN matched or outperformed the recurrent baselines, and it did so while training faster and holding less state. Three structural properties explain that result, and each maps directly onto a weakness of the recurrent training we studied in Chapter 9.

First, full parallelism in training. A convolution at time $t$ depends only on fixed input positions, never on the network's own output at $t-1$, so every output position of every layer is computed at once. There is no sequential unrolling and, crucially, no backpropagation-through-time chain: the backward pass is ordinary backprop over a fixed acyclic graph, parallel across time, instead of the serial $T$-step adjoint recurrence of Section 9.3. On modern accelerators this turns a sequence into a single batched matrix operation, and it is the source of the multi-fold training speedups the paper reports and that we reproduce in subsection four.

Second, stable gradients. The pathological object of Section 9.4 was the long product of hidden-to-hidden Jacobians $\prod_j \mathbf{J}_j$ that an RNN gradient must survive, the product whose norm decides whether the gradient vanishes or explodes. A TCN has no such product. The path from a distant input to the output passes through only $O(\log R)$ convolutional layers, not $R$ recurrent steps, and the residual connections give the gradient a near-identity shortcut around each block. So the effective depth that a gradient traverses is logarithmic in the receptive field rather than linear in it, and the residual highways keep that short path well-conditioned. The result is that TCNs train stably to long receptive fields where a vanilla RNN would have lost the gradient entirely.

Third, a flexible, explicitly controllable receptive field. With an RNN the effective memory is an emergent, hard-to-control property of the learned dynamics; with a TCN the maximum receptive field is a transparent function of the kernel size, dilation schedule, and depth, set by the formula of subsection one. A practitioner who knows the longest dependency the task requires can size the network to cover it exactly, and can trade receptive field against compute by adding or removing levels. This makes the architecture predictable in a way recurrent memory is not.

There is a fourth, quieter advantage that the benchmark numbers reflect but do not name: training stability and ease of tuning. A recurrent network's loss landscape is notoriously delicate, sensitive to initialization, to gradient clipping thresholds, and to the exact gating that keeps the Jacobian product near one. A TCN inherits the well-understood training behavior of a residual convolutional network, for which good defaults (He initialization, a modest dropout, Adam at a standard learning rate) transfer almost unchanged from the image literature. In the original evaluation this showed up as the TCN needing far less per-task hyperparameter search than the LSTM baselines to reach its reported numbers, which in practice is often more valuable than a marginal accuracy edge: a model that works at the first reasonable configuration saves the days of tuning that a recurrent model can demand.

It is worth being precise about the claim. "TCNs often match or beat RNNs" is an empirical statement about a broad benchmark suite, not a theorem that convolution dominates recurrence everywhere. The honest summary of the 2018 evidence is that a generic TCN is a strong, fast, stable default for sequence modeling that a practitioner should try before reaching for a more delicate recurrent model, and that on tasks requiring memory within a sizable but bounded horizon it frequently wins outright. Where the comparison turns against the TCN is precisely the set of limitations we turn to next.

3. Limitations: Fixed Receptive Field, Memory, and No Extrapolation Advanced

The same design decisions that make a TCN fast and stable impose three real limitations, and a practitioner who ignores them will be surprised in production. Naming them honestly is part of using the architecture well.

First, the receptive field is fixed and finite. The history a TCN can condition on is exactly $R = 1 + 2(k-1)(2^{L}-1)$, set once at construction time. Anything that happened more than $R$ steps in the past is, to the network, as if it never happened: it literally cannot enter the computation of the output. An RNN, by contrast, carries a hidden state that can in principle encode information from arbitrarily far back (whether it actually learns to is the vanishing-gradient question of Section 9.4, but the capacity is unbounded). If a task has genuinely unbounded or unpredictably long memory, a TCN sized for one horizon will silently miss dependencies beyond it, and the only remedy is to enlarge $R$ by adding layers or kernel width, which costs parameters and compute.

Second, memory at long sequence lengths. Training a TCN keeps the activations of every layer at every time position for the backward pass, so activation memory scales as $O(L \cdot d \cdot T)$ for depth $L$, width $d$, and length $T$, the same linear-in-length growth that haunted recurrent training in Section 9.3, now multiplied by depth. Worse, because the TCN processes the whole sequence at once rather than streaming it, it cannot trade memory for time by truncating a backward window the way TBPTT does; the parallel forward pass that gives the speed advantage also forces the whole sequence and all its activations to be resident at once. For very long sequences this can make a TCN more memory-hungry than a carefully truncated RNN, and gradient checkpointing becomes necessary.

Third, no built-in input-length extrapolation. A TCN trained on sequences of one length transfers awkwardly to much longer ones at inference, and to shorter ones at the start of a stream, because the left padding that enforces causality was sized for the training length, and the convolutional statistics (and any normalization) were learned at that length. Feeding a far longer sequence does not break the model, but the receptive field is still capped at $R$ and the behavior near the boundaries is not what training prepared. An RNN, which simply keeps stepping its recurrence, handles arbitrary lengths far more gracefully. For streaming inference a TCN must be run with a sliding buffer of the last $R$ inputs, which is a real engineering wrinkle the recurrence avoids.

These three limitations are exactly the openings that the next architectures in Part III exploit. Attention (Chapter 12) replaces the fixed receptive field with an all-pairs interaction that reaches any distance in one layer, at quadratic cost. The structured state-space models of Chapter 13 recover an unbounded, RNN-like memory while keeping the TCN's parallel training through a parallel scan. WaveNet, the subject of Section 11.4, pushes the dilated-causal-convolution idea to its limit for high-resolution generative audio. The TCN is best read as the architecture that proved parallel sequence modeling was possible and competitive; the rest of the part removes its specific constraints one by one.

4. Worked Example: A TCN From Scratch, Raced Against the LSTM Advanced

We now build the architecture in code and put its central claim to the test: a TCN should match or beat the Chapter 10 LSTM on a forecasting task while training faster. The plan is to implement the residual block of subsection one from scratch as an nn.Module, stack a few blocks into a small TCN, train it on a synthetic forecasting series, then run the identical experiment with an LSTM and compare both accuracy and wall-clock training time. Code 11.3.1 is the from-scratch residual block, the load-bearing component of the whole architecture.

import torch
import torch.nn as nn
from torch.nn.utils import weight_norm

class Chomp1d(nn.Module):
    """Trim the extra right-padding so the convolution stays strictly causal."""
    def __init__(self, chomp): super().__init__(); self.chomp = chomp
    def forward(self, x):
        return x[:, :, :-self.chomp].contiguous() if self.chomp > 0 else x

class TCNBlock(nn.Module):
    """One residual block: (dilated causal conv -> weightnorm -> ReLU -> dropout) x2 + 1x1 residual."""
    def __init__(self, c_in, c_out, k, dilation, dropout=0.2):
        super().__init__()
        pad = (k - 1) * dilation                       # left-pad to keep length, chomp the right
        # --- first sub-layer ---
        self.conv1 = weight_norm(nn.Conv1d(c_in, c_out, k, padding=pad, dilation=dilation))
        self.chomp1, self.relu1, self.drop1 = Chomp1d(pad), nn.ReLU(), nn.Dropout(dropout)
        # --- second sub-layer ---
        self.conv2 = weight_norm(nn.Conv1d(c_out, c_out, k, padding=pad, dilation=dilation))
        self.chomp2, self.relu2, self.drop2 = Chomp1d(pad), nn.ReLU(), nn.Dropout(dropout)
        # --- residual: 1x1 conv only when channel counts differ, else identity ---
        self.downsample = nn.Conv1d(c_in, c_out, 1) if c_in != c_out else None
        self.relu_out = nn.ReLU()

    def forward(self, x):                              # x: (batch, channels, time)
        z = self.drop1(self.relu1(self.chomp1(self.conv1(x))))   # first conv-norm-relu-drop
        z = self.drop2(self.relu2(self.chomp2(self.conv2(z))))   # second, identical
        res = x if self.downsample is None else self.downsample(x)
        return self.relu_out(z + res)                  # add the shortcut, then a final ReLU

Stacking these blocks with a doubling dilation schedule gives the full network. Code 11.3.2 assembles the stack and adds a linear head that reads the last time position to produce a one-step-ahead forecast.

class TCN(nn.Module):
    """Stack residual blocks with dilations 1, 2, 4, ... and a linear forecasting head."""
    def __init__(self, c_in, channels, k=3, dropout=0.2):
        super().__init__()
        blocks, prev = [], c_in
        for i, c_out in enumerate(channels):           # one block per level
            blocks.append(TCNBlock(prev, c_out, k, dilation=2 ** i, dropout=dropout))
            prev = c_out                               # dilation doubles each level: 1,2,4,8,...
        self.network = nn.Sequential(*blocks)
        self.head = nn.Linear(channels[-1], 1)         # forecast from the final channel vector

    def forward(self, x):                              # x: (batch, time, features)
        y = self.network(x.transpose(1, 2))            # conv1d wants (batch, channels, time)
        return self.head(y[:, :, -1])                  # read the last (rightmost) position

# receptive field check for k=3, 6 levels: R = 1 + 2*(k-1)*(2^L - 1)
L, k = 6, 3
print("receptive field R =", 1 + 2 * (k - 1) * (2 ** L - 1))

receptive field R = 253

Now the head-to-head. Code 11.3.3 generates a synthetic autoregressive-plus-seasonal series, trains the from-scratch TCN and a same-width LSTM (the recurrent baseline of Chapter 10) on identical data for identical epochs, and reports both final test loss and total training wall-clock for each.

import time, numpy as np

def make_series(n=6000, seed=0):
    rng = np.random.default_rng(seed)
    t = np.arange(n)
    s = np.sin(2*np.pi*t/24) + 0.5*np.sin(2*np.pi*t/168)   # daily + weekly seasonality
    ar = np.zeros(n)
    for i in range(2, n):
        ar[i] = 0.6*ar[i-1] - 0.2*ar[i-2] + rng.normal(0, 0.3)  # AR(2) component
    return (s + ar).astype(np.float32)

W = 64                                                   # input window length
series = make_series()
X = np.stack([series[i:i+W] for i in range(len(series)-W-1)])[..., None]  # (N, W, 1)
Yt = series[W+1:len(series)-0][:len(X)]                                   # one-step-ahead target
split = int(0.8*len(X))
Xtr, Ytr = torch.tensor(X[:split]), torch.tensor(Yt[:split, None])
Xte, Yte = torch.tensor(X[split:]), torch.tensor(Yt[split:, None])

def train(model, epochs=8):
    opt, lossf = torch.optim.Adam(model.parameters(), 1e-3), nn.MSELoss()
    t0 = time.perf_counter()
    for _ in range(epochs):
        for b in range(0, len(Xtr), 256):
            opt.zero_grad()
            out = model(Xtr[b:b+256])
            lossf(out, Ytr[b:b+256]).backward()
            opt.step()
    wall = time.perf_counter() - t0
    with torch.no_grad():
        test = lossf(model(Xte), Yte).item()
    return wall, test

class LSTMBaseline(nn.Module):                            # the Chapter 10 recurrent baseline
    def __init__(self, h=32):
        super().__init__(); self.rnn = nn.LSTM(1, h, batch_first=True); self.head = nn.Linear(h, 1)
    def forward(self, x): out, _ = self.rnn(x); return self.head(out[:, -1])

torch.manual_seed(0)
tcn_wall, tcn_mse = train(TCN(1, channels=[32, 32, 32, 32]))   # 4 levels, width 32
torch.manual_seed(0)
lstm_wall, lstm_mse = train(LSTMBaseline(h=32))
print(f"TCN   : test MSE {tcn_mse:.4f}   train wall {tcn_wall:.2f}s")
print(f"LSTM  : test MSE {lstm_mse:.4f}   train wall {lstm_wall:.2f}s")
print(f"TCN trains {lstm_wall/tcn_wall:.1f}x faster, MSE ratio {lstm_mse/tcn_mse:.2f}")

TCN   : test MSE 0.1043   train wall 1.18s
LSTM  : test MSE 0.1187   train wall 3.46s
TCN trains 2.9x faster, MSE ratio 1.14

Read the three code blocks as one argument. Code 11.3.1 built the residual block component by component, exactly mirroring Figure 11.3.1; Code 11.3.2 stacked it with a doubling dilation schedule and confirmed the receptive field by formula; Code 11.3.3 trained it against the recurrent baseline of Chapter 10 and measured the very speedup that subsection two predicted, the parallel TCN finishing while, in the words of the epigraph, the recurrent network is still stepping through its sequence. The accuracy parity and the wall-clock gap together are the empirical claim of Bai, Kolter and Koltun reproduced on a single page.

from pytorch_tcn import TCN              # pip install pytorch-tcn

model = TCN(
    num_inputs=1,                       # input channels
    num_channels=[32, 32, 32, 32],      # one entry per level; dilations 1,2,4,8 set automatically
    kernel_size=3,
    dropout=0.2,
    causal=True,                        # left-padding + chomp handled internally
)
y = model(x)                            # x: (batch, channels, time); fully causal, parallel