"They warned me I was just a filter, a fixed little window sliding over the signal, blind to anything older than my own width. So I learned a trick. I left gaps. One step, then two, then four, then eight, doubling the reach of every layer while my parameter count barely stirred, until a stack of us could touch a thousand steps of history in a dozen clean strides. And here is the part the recurrences never managed: I do not wait. I see every timestep at once, all of them in parallel, the whole past unrolled before me in a single forward sweep. I am not a memory that must be carried. I am a shape that already contains the past, and I never once worried about my gradient on the way back."
A Dilated Kernel Reaching Exponentially Into the Past
Chapter Overview
Convolution is not a newcomer to this book. In Chapter 4 a filter was a small, fixed set of weights slid along a series to smooth it, to extract a band, or to compute a moving average, and the analyst chose those weights or read them off a frequency response. This chapter brings the convolution back, but with the weights set free. The kernel is no longer hand-designed; it is learned by gradient descent against a forecasting or classification loss, so the network discovers for itself which local temporal patterns matter. The moving-average filter of Part II becomes a stack of learned filters, and the same operation that smoothed a series now builds a representation of it. That is the temporal thread of this chapter: the classical filter returns as a learned convolution.
Two design choices turn an ordinary one-dimensional convolution into a sequence model worthy of the name. The first is causality: the kernel is padded and shifted so that the output at time $t$ depends only on inputs at times up to and including $t$, never the future, which is exactly the no-leakage discipline of Chapter 2 baked into the architecture. The second is dilation: by spacing the kernel taps apart and doubling that spacing at each layer, the receptive field grows exponentially with depth, so a modest stack of layers can reach hundreds or thousands of steps into the past while the parameter count grows only linearly. The payoff over the recurrent networks of Chapter 10 is decisive on hardware: where a recurrence must process timesteps one after another, a convolution sees the entire sequence at once and trains fully in parallel, and its gradient flows back through a shallow, well-conditioned graph rather than a long fragile product of Jacobians.
Those two ideas assemble into two landmark architectures. The Temporal Convolutional Network, or TCN, stacks residual blocks of causal dilated convolutions with weight normalization and dropout into a clean, general-purpose sequence model that, on a striking range of benchmarks, matches or beats recurrent networks of comparable size. WaveNet, which predates the TCN name and inspired it, adds gated activation units and a dense skip-connection structure and uses the same causal dilated stack to generate raw audio one sample at a time, the canonical demonstration that a convolution can be an autoregressive generative model, not merely a forecaster. This chapter builds both, the discriminative TCN for forecasting and the generative WaveNet head for sampling, from the same convolutional parts.
The chapter closes with an honest accounting rather than a victory lap. Convolutions win on parallel training and on stable gradients, and a well-tuned TCN is a formidable and underused baseline. But a TCN's receptive field is still bounded by its depth and dilation schedule: anything older than the deepest tap is simply invisible, where a recurrence has, in principle, unbounded memory and an attention layer can connect any two positions directly regardless of distance. That bounded horizon, and the desire to weight distant context by relevance rather than by fixed kernel position, is exactly the gap that Chapter 12 opens with. Read this chapter, then, as both the strongest convolutional answer to sequence modeling and the setup for why the field moved to attention.
Prerequisites
This chapter assumes the neural sequence foundations of the two chapters before it and the classical filtering idea it revives. From Chapter 9: Neural Sequence Modeling Fundamentals you need the taxonomy of sequence tasks, the windowing and temporal-split discipline, and above all the vanishing-gradient problem, because a central selling point of convolutions is that they sidestep it. From Chapter 10: Recurrent Neural Networks you need the gated recurrent architectures and their sequential, one-step-at-a-time training cost, since the comparison that runs through Section 11.5 is precisely TCN against LSTM on matched tasks; a fair comparison needs both contestants understood. From Chapter 4: Frequency-Domain and Spectral Analysis you need convolution as filtering: the moving-average and band-pass filters whose fixed taps this chapter turns into learned ones, and the receptive-field intuition that a wider or more spread-out kernel sees more of the signal. Beyond those, you need a working knowledge of PyTorch and convolutional layers at the level of Appendix C and Appendix D. Readers who want refreshers on any of these foundations, or on the appendices, will find every referenced chapter indexed in the Table of Contents.
If you keep one idea from this chapter, keep this: a causal, dilated one-dimensional convolution gives a sequence model an exponentially large receptive field and fully parallel training, turning the fixed filter of Chapter 4 into a learned one that often matches a recurrent network at a fraction of the wall-clock cost. Causality, padding and shifting so output $t$ sees only inputs up to $t$, hard-wires the no-leakage rule of Chapter 2 into the architecture. Dilation, doubling the gap between kernel taps layer by layer, makes the receptive field grow as a power of two in depth while parameters grow only linearly. Stacked into residual blocks you get the TCN, a strong general-purpose forecaster; add gated activations and a sample-by-sample autoregressive head and you get WaveNet, a convolutional generative model. The one honest catch, the bounded receptive field that cannot see past its deepest tap, is exactly the limitation the attention of Chapter 12 was built to remove.
Chapter Roadmap
- 11.1 One-Dimensional Convolutions How a one-dimensional convolution slides a learned kernel over a time series, what kernel size, stride, padding, channels, and the receptive field each control, why a learned filter generalizes the fixed moving-average and band-pass filters of Chapter 4, and a from-scratch numpy convolution paired with its PyTorch Conv1d equivalent.
- 11.2 Dilated and Causal Convolutions How causal padding and shifting make the output at time $t$ depend only on the past, how dilation spaces the kernel taps and doubles layer by layer so the receptive field grows exponentially in depth at linear parameter cost, the exact formula relating depth, kernel size, and dilation to the horizon, and a runnable causal dilated stack with its receptive field verified.
- 11.3 Temporal Convolutional Networks (TCN) How the TCN assembles causal dilated convolutions into residual blocks with weight normalization and dropout, why the residual path keeps deep stacks trainable, how the architecture matches or beats recurrent networks on a wide benchmark suite, and a complete PyTorch TCN forecaster built block by block.
- 11.4 WaveNet How WaveNet adds gated activation units and dense skip connections to the causal dilated stack, how it generates a sequence one sample at a time as an autoregressive model, the trade-off between fast parallel training and slow sequential sampling, and a WaveNet-style generative head built on the convolutional parts of the earlier sections.
- 11.5 Comparative Analysis An honest head-to-head of convolutions against the recurrences of Chapter 10 and a forward look to the attention of Chapter 12, weighing parallel training and stable gradients against the TCN's bounded receptive field, with guidance on when to reach for a TCN, an RNN, or a Transformer.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single experiment. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have built the one-dimensional convolutions of Section 11.1, made them causal and dilated as in Section 11.2, stacked them into the TCN of Section 11.3, raced it against the Chapter 10 LSTM in the spirit of Section 11.5, and generated autoregressively with the WaveNet head of Section 11.4.
Hands-On Lab: Build a TCN Forecaster
Objective
Build a Temporal Convolutional Network from its convolutional parts, prove that it does what its design promises, and then put it in a fair fight against the recurrent workhorse of the previous chapter. You will start by stacking causal dilated convolution blocks into a TCN, following Sections 11.1 through 11.3, then verify two properties that are easy to claim and easy to get wrong: that the network is genuinely causal, meaning a change to any future input leaves the present output untouched, and that its receptive field is exactly as wide as the depth-and-dilation formula predicts, which you confirm by perturbing inputs one position at a time and watching where the output stops responding. With a correct TCN in hand you will train it on the same forecasting task you used for the LSTM of Chapter 10 and compare the two head to head on both accuracy and wall-clock training speed, the comparison at the center of Section 11.5; the convolution's parallel training is meant to show up as a real clock-time win. Finally you will swap the TCN's forecasting head for a WaveNet-style autoregressive head from Section 11.4 and generate a sequence one sample at a time, feeling directly the asymmetry that defines these models: training is fast and parallel, sampling is slow and sequential. You finish with a working, verified, benchmarked TCN and a clear, measured sense of where convolutions beat recurrences and where they do not.
What You'll Practice
- Building one-dimensional convolutions with the right kernel size, channels, and padding for a time series, following Section 11.1.
- Making a convolution causal and dilated, and computing its receptive field from depth and dilation, following Section 11.2.
- Stacking causal dilated blocks with residual connections into a complete TCN forecaster, following Section 11.3.
- Empirically verifying causality and receptive field by perturbing inputs and observing the output, following Section 11.2 and Section 11.3.
- Benchmarking the TCN against the Chapter 10 LSTM on matched accuracy and training wall-clock, following Section 11.5.
- Generating autoregressively with a WaveNet-style head and feeling the parallel-train, sequential-sample asymmetry, following Section 11.4.
Setup
You need PyTorch for the convolutions, the training loop, and the autograd that supports both, NumPy to assemble the forecasting windows, and Matplotlib to plot the receptive-field probe and the accuracy-versus-time comparison. No new dataset download is required: reuse the same forecasting series and the same temporal train-validation-test split you built for the LSTM of Chapter 10, so the TCN and the LSTM compete on identical data, and pull a longer real series from the Monash Forecasting Archive linked in the bibliography when you want to stress the receptive field. Every model respects the no-leakage discipline of Chapter 2, which the causal convolution enforces structurally rather than by convention.
import torch, torch.nn as nn
class CausalDilatedConv(nn.Module):
def __init__(self, ch, kernel=3, dilation=1):
super().__init__()
self.pad = (kernel - 1) * dilation # left-pad so output t sees only past
self.conv = nn.Conv1d(ch, ch, kernel, dilation=dilation)
def forward(self, x): # x: (batch, channels, time)
return self.conv(nn.functional.pad(x, (self.pad, 0))) # pad left only: strict causality
Steps
Step 1: Stack causal dilated blocks into a TCN
Build the network from the atom above, following Sections 11.1 through 11.3. Stack several causal dilated convolution blocks with the dilation doubling at each layer, one, two, four, eight, and wrap each block in a residual connection so the deep stack stays trainable. Cap it with a linear head that reads the forecast from the last timestep. The doubling dilation is the whole point: a handful of layers buys a receptive field of dozens or hundreds of steps.
Step 2: Verify causality and receptive field
Before trusting the model, prove it behaves, following Section 11.2. To check causality, change a single future input and confirm the present output does not move; if it does, a padding is wrong. To measure the receptive field, perturb the input one position at a time, sweeping backward, and record the last position whose change still alters the output. Compare that empirical horizon to the depth-and-dilation formula. They should match exactly, and finding out they do not is how you catch an off-by-one in the dilation schedule.
Step 3: Train the TCN forecaster
Train the verified TCN on the Chapter 10 forecasting task, following Section 11.3. Because the convolution sees the whole input window at once, every timestep's loss is computed in a single parallel forward pass, so each epoch is fast. Record the validation error and, just as carefully, the wall-clock time per epoch; the clock is half the experiment.
Step 4: Race the TCN against the Chapter 10 LSTM
Now run the comparison at the heart of Section 11.5. Train an LSTM of comparable parameter count on the identical task and split, and plot both contestants on the same axes: validation accuracy against training wall-clock. The expected story is that the TCN reaches comparable accuracy in noticeably less clock time, because its parallel training avoids the LSTM's step-by-step recurrence. Note honestly where the LSTM wins, typically when the relevant dependency runs longer than the TCN's receptive field.
Step 5: Generate autoregressively with a WaveNet-style head
Finally, swap the forecasting head for a WaveNet-style autoregressive head from Section 11.4 and generate, feeding each predicted sample back as the next input. Here the asymmetry bites: the same convolutions that trained in parallel must now run one step at a time during sampling, because each new sample depends on the last. Time a generated sequence and contrast it with the training speed of Step 3. That contrast, fast parallel training against slow sequential generation, is the defining trade-off of convolutional sequence models and the lab's closing lesson.
Expected Output
The lab produces a verified model and two comparison plots rather than a single number. Step 2 yields a receptive-field probe whose empirical horizon lands exactly on the depth-and-dilation formula and a causality check that shows future inputs leaving the present output untouched, the two correctness proofs that separate a real TCN from a leaky one. Step 4 produces the headline figure: validation accuracy against training wall-clock for the TCN and the LSTM on identical data, with the TCN typically reaching comparable accuracy in less clock time thanks to parallel training, and the LSTM holding its own or pulling ahead exactly when the dependency outruns the TCN's bounded receptive field. Step 5 closes with a timing contrast that makes the convolution's trade-off concrete: training was fast because every timestep ran at once, generation is slow because samples must come one after another. You finish with a working TCN you have proven correct, a measured sense of when a convolution beats a recurrence, and a felt understanding of the bounded receptive field that motivates the attention of Chapter 12.
Once Sections 11.1 through 11.3 have shown you how a causal dilated residual stack is built from scratch, you rarely write one again by hand. PyTorch supplies nn.Conv1d with a dilation argument, so the dilated convolution is one constructor call, and left padding with nn.functional.pad makes it causal in one line. For the full architecture, the pytorch-tcn package gives a complete, well-tested TCN module with the dilation schedule, residual blocks, weight normalization, and dropout already wired, turning the whole of Step 1 into a handful of lines, and forecasting libraries such as Darts wrap a TCN behind a fit-and-predict interface that handles windowing, scaling, and the temporal split for you. The line-count reduction from a hand-rolled stack to a library TCN is roughly an order of magnitude, and the library handles the off-by-one padding and dilation bookkeeping that Step 2 exists to catch. Build it once from the convolutional atom so you know exactly what the library does, then let the library carry every experiment after.
Stretch Goals
- Deepen the TCN by adding more dilated layers and confirm, with the Step 2 probe, that the receptive field doubles with each added layer as Section 11.2 predicts.
- Replace the plain activation in each block with the gated activation unit of WaveNet and measure whether it improves the forecast, testing the gating lever of Section 11.4.
- Run the TCN-versus-LSTM race from Section 11.5 on a longer Monash series whose dependency exceeds the TCN's receptive field, and document the regime where the recurrence wins.
What's Next?
This chapter gave convolutions their due: causal and dilated, they buy an exponentially large receptive field and fully parallel training, and a well-tuned TCN is a strong, under-appreciated baseline that often matches a recurrent network at a fraction of the wall-clock cost. But the chapter ended on the one limitation no dilation schedule removes, the bounded receptive field: a convolution simply cannot see past its deepest tap, and it weights its context by fixed kernel position rather than by relevance. Chapter 12: Attention and Transformers removes both limits at once. An attention layer connects any two timesteps directly, no matter how far apart, and it weights every position in the context by a learned, content-dependent relevance rather than a fixed offset, so distance no longer caps memory and the model decides for itself what in the past matters now. The Transformer keeps the parallel-training advantage this chapter prized, since attention, like convolution, processes the whole sequence at once, while lifting the fixed-horizon ceiling that convolution could not. The recurrence-to-convolution-to-attention arc that organizes Part III reaches its third and most consequential design in the chapter ahead. The full sequence of architectures, with the appendices, is laid out in the Table of Contents.
Bibliography & Further Reading
Foundational Papers
Bai, S., Kolter, J. Z., Koltun, V. "An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling." 2018. arXiv:1803.01271. arxiv.org/abs/1803.01271
The paper that named the Temporal Convolutional Network and showed that a generic causal dilated residual stack matches or beats recurrent networks across a broad benchmark suite, the empirical backbone of Sections 11.3 and 11.5.
van den Oord, A., Dieleman, S., Zen, H., et al. "WaveNet: A Generative Model for Raw Audio." 2016. arXiv:1609.03499. arxiv.org/abs/1609.03499
The architecture that put causal dilated convolutions on the map, generating raw audio sample by sample with gated activations and skip connections, the direct subject of Section 11.4 and the WaveNet head in the lab.
Lea, C., Vidal, R., Reiter, A., Hager, G. D. "Temporal Convolutional Networks: A Unified Approach to Action Segmentation." 2016. arXiv:1608.08242. arxiv.org/abs/1608.08242
An early use of the temporal-convolutional-network idea for action segmentation, one of the works that established the causal convolution as a general sequence model beyond audio, context for Section 11.3.
Yu, F., Koltun, V. "Multi-Scale Context Aggregation by Dilated Convolutions." 2015. arXiv:1511.07122. arxiv.org/abs/1511.07122
The paper that popularized dilated convolutions and the exponentially growing receptive field at linear parameter cost, the mechanism Section 11.2 builds on and the lab's receptive-field probe verifies.
Borovykh, A., Bohte, S., Oosterlee, C. W. "Conditional Time Series Forecasting with Convolutional Neural Networks." 2017. arXiv:1703.04691. arxiv.org/abs/1703.04691
A WaveNet-style convolutional model applied directly to financial time-series forecasting, the bridge from audio generation to the forecasting task of Sections 11.3 and the chapter lab.
Modern Advances
Liu, M., Zeng, A., Chen, M., et al. "SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction." NeurIPS, 2022. arXiv:2106.09305. arxiv.org/abs/2106.09305
A modern downsample-convolve-interact architecture that revisited the convolutional approach to forecasting and posted strong results, evidence for Section 11.5 that convolutions remain competitive.
Donghao, L., Xue, W. "ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis." ICLR, 2024. openreview.net
A recent pure-convolution architecture that widens the receptive field and rivals Transformers across forecasting, classification, and anomaly tasks, the strongest current case for the convolutional family of Section 11.5.
Tools & Libraries
PyTorch: Conv1d and the convolution modules. Official documentation. pytorch.org/docs
The one-dimensional convolution behind every model in this chapter, with the dilation and padding arguments that Sections 11.1 and 11.2 use to make a kernel causal and dilated.
Krug, P. "pytorch-tcn: A complete Temporal Convolutional Network implementation in PyTorch." github.com/paul-krug/pytorch-tcn
A ready-made TCN module with the dilation schedule, residual blocks, weight normalization, and dropout already wired, the library shortcut that compresses Step 1 of the lab into a few lines.
Darts: a Python library for time series forecasting, including a TCN model. unit8co.github.io/darts
A forecasting library that wraps a TCN behind a fit-and-predict interface handling windowing, scaling, and the temporal split, the high-level path from Section 11.3 to a deployed forecaster.
Datasets & Benchmarks
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., Montero-Manso, P. "Monash Time Series Forecasting Archive." NeurIPS Datasets and Benchmarks, 2021. forecastingdata.org
A large, curated collection of real forecasting datasets across many domains and frequencies, the field-data benchmark for stressing the TCN's receptive field and running the Section 11.5 comparison beyond synthetic tasks.