"For a decade the recurrence made me wait. It read the past one step at a time and squeezed it into a single vector, then handed me that vector and told me everything I would ever need was in there somewhere. It never was. The early steps had faded by the time I needed them, and I could not reach back to check. Now I do not wait and I do not squeeze. I look. For every position I am asked about, I send out a query and let every other timestep answer with how much it matters, and I read a weighted blend of them all at once, in parallel, no carry, no decay. They say I have no sense of order, and it is true: shuffle my inputs and I would not notice, so they bolt a clock onto me to remind me when things happened. And they say I am expensive, that comparing everyone to everyone costs me the square of the sequence, and that is true too. But I never forget a thing I am allowed to look at, and I decided long ago that I would rather pay the bill than lose the past."
An Attention Weight That Refuses to Sum to More Than One
Chapter Overview
Chapter 10 ended on a bottleneck. The sequence-to-sequence model read an entire input sequence into one fixed context vector and asked the decoder to generate everything from that single summary, so the further back a piece of the input lay, the more it had faded by the time the decoder needed it. Attention is the idea that removes the bottleneck outright. Instead of compressing the past into one vector and reading from that, an attention layer lets each position issue a query, score it against a key at every other position, and read back a weighted blend of the values there. The access is content-based and data-dependent: what a position attends to is decided by the data at run time, not fixed by architecture, and crucially the whole input stays available, so reaching far back costs no more than reaching nearby. This is the deepest contrast in the temporal thread of Part III. Where recurrence carries the past forward step by step and lets it decay, and convolution reaches back through a fixed receptive field, attention connects any two timesteps directly with a learned, on-the-fly weight.
Self-attention, where a sequence attends to itself, is the engine of the Transformer, the architecture this chapter builds in full. A Transformer block stacks multi-head self-attention, which runs several attention patterns in parallel so different heads can track different relationships, with a position-wise feed-forward network, and wraps both in residual connections and layer normalization so a deep stack stays trainable. Because every position is computed independently and in parallel, the Transformer trains far faster on long sequences than the inherently sequential recurrence it replaced, which is the practical reason it took over. The chapter develops the query-key-value mechanism from scratch, assembles it into the encoder and decoder blocks, and shows how a causal mask preserves the no-peeking-at-the-future discipline that every temporal model owes its data.
That parallel, all-to-all design buys speed and reach at two prices the chapter treats honestly. The first is that pure attention is permutation invariant: shuffle the inputs and the output set is unchanged, because the mechanism has no built-in notion of order. For a temporal model that is a fatal flaw, and the fix is positional encoding, a signal added to each input that tells the model when each token occurred. The chapter develops the original sinusoidal encodings, learned positional embeddings, and the rotary and relative schemes (RoPE and Transformer-XL style) that have become standard, treating the choice of temporal encoding as a first-class modeling decision rather than a detail. The second price is cost. Comparing every position to every other makes attention scale as the square of the sequence length in both time and memory, which is exactly the regime long time-series forecasting lives in. That quadratic wall is what drives the final movement of the chapter and the next.
The closing sections survey the long-sequence problem and the efficient and sparse attention variants built to scale past it: sparse and local-window attention, low-rank and kernelized approximations that make attention linear in sequence length, memory-recurrent schemes that extend context, and the IO-aware exact-attention kernels that cut the memory cost without changing the math. These are the engineering that lets Transformers reach the long horizons forecasting demands, and they are also a signpost. The observation that attention can be made linear, and that a linear attention is a kind of recurrence in disguise, leads directly to the structured state-space models of Chapter 13, which chase the same long-range reach from the other side, starting from the linear recurrence and the Kalman filter rather than from all-to-all attention. Following the Part III convention, the chapter's worked examples and lab race the Transformer forecaster against the recurrent and convolutional baselines of Chapters 10 and 11 on tasks where the cost and the accuracy can both be measured rather than asserted.
Prerequisites
This chapter builds on two anchors in Part III. From Chapter 9: Neural Sequence Modeling Fundamentals you need the sequence-task taxonomy, the unrolled view of a sequence model, and the encoder-decoder framing, because attention is introduced as a new way to wire an encoder to a decoder and will be clearest against that backdrop. From Chapter 10: Recurrent Neural Networks you need the sequence-to-sequence model and, above all, its single-context-vector bottleneck, since attention is the precise cure for that bottleneck and every section here is motivated by the limit Chapter 10 left on the table. You also need a working command of PyTorch at the level the earlier Part III chapters assumed: defining modules, training with an optimizer, and trusting autograd to run the backward pass, plus comfort with the matrix products and the softmax that make up the attention operation itself. Readers who want a refresher on linear algebra and the softmax, on optimization and deep learning basics, or on PyTorch will find Appendices A, C, and D and every referenced chapter indexed in the Table of Contents.
If you keep one idea from this chapter, keep this: attention replaces the recurrence's fixed, decaying summary of the past with content-based, data-dependent access to the whole past at once, scoring a query against a key at every position and reading back a weighted blend of values, which dissolves the seq2seq bottleneck and parallelizes training, at the cost of needing positional encodings to recover a sense of time and of scaling as the square of the sequence length. Stack multi-head self-attention with a position-wise feed-forward network, residual connections, and layer normalization, and you have the Transformer block; add a causal mask and you have a forecaster that respects the arrow of time. The quadratic cost is the wall that drives the efficient and sparse variants of this chapter and motivates the structured state-space models of Chapter 13, which reach the same long horizons from the linear-recurrence side. Where recurrence carries the past and convolution reaches back through a fixed window, attention connects any two timesteps directly with a learned weight.
Chapter Roadmap
- 12.1 Self-Attention The query-key-value mechanism built from scratch: how each position issues a query, scores it against every key, and reads a softmax-weighted blend of values, why this is content-based data-dependent access to the whole sequence, and the explicit contrast with the carried, decaying memory of the recurrence it replaces.
- 12.2 The Transformer Architecture Assembling self-attention into the Transformer block: multi-head attention that tracks several relationships at once, the position-wise feed-forward network, residual connections and layer normalization that keep a deep stack trainable, the encoder and decoder, and the causal mask that preserves the arrow of time.
- 12.3 Positional and Temporal Encodings The price of permutation invariance and how to pay it: sinusoidal encodings, learned positional embeddings, and the relative and rotary schemes (Transformer-XL style and RoPE), plus the time-aware encodings that let a Transformer represent irregular sampling and calendar structure in real temporal data.
- 12.4 Long-Sequence Modeling and Memory Why full attention scales as the square of the sequence length in time and memory, why that quadratic wall bites hardest in long-horizon forecasting, and the memory and recurrence schemes (segment-level recurrence, recurrent memory) that extend a Transformer's effective context past a single window.
- 12.5 Efficient and Sparse Transformers The variants built to break the quadratic wall: sparse and local-window attention, low-rank and kernelized approximations that make attention linear in sequence length, and IO-aware exact kernels, with the linear-attention-as-recurrence observation that opens the door to the state-space models of Chapter 13.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single experiment. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have built the self-attention of Section 12.1 into the Transformer forecaster of Section 12.2, added the positional encoding of Section 12.3, measured the quadratic cost of Section 12.4, and swapped in the efficient attention of Section 12.5.
Hands-On Lab: Forecast With a Transformer
Objective
Build a small encoder-only Transformer forecaster end to end, race it against the recurrent and convolutional baselines you trained in Chapters 10 and 11, and feel both the promise and the price of attention directly. You will start by patching a univariate series into a sequence of tokens, a short window of consecutive values projected to a model dimension, so the Transformer sees a manageable number of tokens rather than one per timestep, following the self-attention construction of Section 12.1. You will add a positional encoding from Section 12.3 so the permutation-invariant attention recovers a sense of order, apply a causal mask so no token attends to its own future, and assemble the multi-head blocks of Section 12.2 into a forecaster with a linear forecast head. You will train it on the same forecasting task and split you used for the gated recurrent forecaster of Chapter 10 and the temporal convolutional forecaster of Chapter 11, and compare accuracy head to head. Then comes the cost lesson of Section 12.4: you will measure wall-clock time and peak memory as you grow the sequence length and watch the curve bend upward as the square of the length, the quadratic wall made concrete. Finally you will swap the full attention for a linear or efficient attention from Section 12.5, repeat the cost sweep, and confirm the curve flattens toward linear, at a measured and usually small change in accuracy, the engineering that lets attention reach long horizons.
What You'll Practice
- Patching a time series into tokens and implementing scaled dot-product self-attention from scratch, following Section 12.1.
- Assembling multi-head attention, the feed-forward network, residuals, and layer norm into a causal Transformer block, following Section 12.2.
- Adding a positional encoding so the permutation-invariant stack recovers temporal order, following Section 12.3.
- Measuring attention's time and memory cost against sequence length and seeing the quadratic curve, following Section 12.4.
- Swapping in a linear or efficient attention and confirming the cost curve flattens at a measured accuracy change, following Section 12.5.
Setup
You need PyTorch for the attention layers, the Transformer blocks, and the autograd that runs the backward pass, NumPy to build the forecasting windows and patch them into tokens, and Matplotlib to plot accuracy against the baselines and cost against sequence length. No new dataset download is required: reuse the exact forecasting series and the temporal split from the Chapter 10 and Chapter 11 labs, so the Transformer competes against the gated recurrent and temporal convolutional forecasters on identical data and the comparison is fair. Every model trains and evaluates on disjoint windows and respects the temporal split discipline of Chapter 2, so a strong score reflects genuine forecasting skill rather than a leak of the future, and the causal mask enforces the same no-peeking rule inside the attention itself. PyTorch supplies nn.TransformerEncoderLayer and nn.MultiheadAttention as drop-in blocks, so once your from-scratch attention matches their output you can let the library carry the larger runs.
import torch, torch.nn as nn
class PatchTransformer(nn.Module):
def __init__(self, patch=8, d_model=64, heads=4, layers=2, horizon=24):
super().__init__()
self.embed = nn.Linear(patch, d_model) # patch a window into a token
self.pos = nn.Parameter(torch.randn(1, 512, d_model) * 0.02) # learned positional encoding
block = nn.TransformerEncoderLayer(d_model, heads, batch_first=True)
self.enc = nn.TransformerEncoder(block, layers)
self.head = nn.Linear(d_model, horizon) # linear forecast head
def forward(self, patches): # patches: (batch, n_patch, patch)
z = self.embed(patches) + self.pos[:, :patches.size(1)] # tokens plus position
mask = nn.Transformer.generate_square_subsequent_mask(patches.size(1))
z = self.enc(z, mask=mask.to(z.device)) # causal self-attention
return self.head(z[:, -1]) # forecast from the last token
Steps
Step 1: Patch the series into tokens and build self-attention
Slice each input window into short non-overlapping patches and project each patch to the model dimension, turning a long series into a short sequence of tokens, the move that keeps attention affordable and that Section 12.1 motivates. Implement scaled dot-product self-attention from scratch on these tokens, then check it against PyTorch's nn.MultiheadAttention on the same input so you trust the library version for the rest of the lab.
Step 2: Add positional encoding and a causal mask
Attention is permutation invariant, so add the positional encoding of Section 12.3 to the tokens and confirm that shuffling the input now changes the output, the order information restored. Apply the causal mask of Section 12.2 so no token attends to a later one, and verify on a tiny example that the forecast for a position is unchanged when you alter values strictly after it.
Step 3: Train the Transformer against the Chapter 10 and 11 baselines
Assemble the multi-head blocks of Section 12.2 into the forecaster above and train it on the same task, split, and metric you used for the gated recurrent forecaster of Chapter 10 and the temporal convolutional forecaster of Chapter 11. Plot all three accuracies side by side. The Transformer should be competitive on the long-horizon task while training faster per epoch than the sequential recurrence, the parallel all-to-all design paying off.
Step 4: Measure attention cost against sequence length
Hold the model fixed and sweep the number of tokens, timing a forward and backward pass and recording peak memory at each length. Plot both against sequence length. The curves should bend upward as the square of the length, the quadratic cost of Section 12.4 made visible, and the bend tells you the length at which full attention stops being practical for your hardware.
import time, torch
def attention_cost(model, n_patch, patch=8, batch=16):
x = torch.randn(batch, n_patch, patch, device="cuda")
torch.cuda.reset_peak_memory_stats(); torch.cuda.synchronize(); t = time.time()
model(x).sum().backward() # one forward + backward
torch.cuda.synchronize()
return time.time() - t, torch.cuda.max_memory_allocated() / 1e6 # seconds, MB
Step 5: Swap in a linear or efficient attention
Replace the full attention with a linear or efficient variant from Section 12.5, a kernelized or low-rank approximation whose cost grows linearly with sequence length, and rerun both the accuracy comparison of Step 3 and the cost sweep of Step 4. The cost curve should flatten from quadratic toward linear, letting you push to sequence lengths the full model could not reach, at a change in accuracy you now measure rather than guess. This is the engineering that carries attention to long horizons, and the linear-attention-as-recurrence view that opens the door to Chapter 13.
Expected Output
The lab produces a clear, measured story about attention's promise and its bill. The accuracy plot of Step 3 places the Transformer beside the gated recurrent and temporal convolutional forecasters of Chapters 10 and 11 on identical data, showing it competitive on the long-horizon task and faster per epoch than the sequential recurrence, the parallel design earning its keep. The mask and shuffle checks of Step 2 confirm the model now respects order and never peeks at the future, the two corrections that turn raw attention into a valid temporal model. The cost sweep of Step 4 is the chapter's hard lesson in one figure: time and memory curve upward as the square of the sequence length, and you can read off the length where full attention becomes impractical on your hardware. The swap of Step 5 answers that lesson: the efficient attention's cost curve flattens toward linear, reaching lengths the full model could not, at a measured and usually small accuracy change. The reader finishes with a Transformer forecaster they built and understand, a felt sense of the quadratic wall, and a concrete bridge to the state-space models of Chapter 13 that scale long-range modeling from the linear-recurrence side.
Once you have built scaled dot-product attention from scratch in Section 12.1 so you know exactly what the query-key-value product computes, let PyTorch carry the experiments. nn.MultiheadAttention implements multi-head scaled dot-product attention with masking in one call, nn.TransformerEncoderLayer wraps attention, the feed-forward network, residuals, and layer normalization into a complete block, and nn.TransformerEncoder stacks them, so the whole encoder collapses from a careful hundred-plus-line implementation to a handful of lines while the library handles the fused attention kernel and the masking internally. nn.Transformer.generate_square_subsequent_mask builds the causal mask for you, and recent PyTorch routes attention through an IO-aware fused kernel automatically, so the exact-attention memory savings of Section 12.5 come for free. Build the mechanism once by hand, then let these blocks run every comparison in the lab; the line-count reduction is dramatic and the numerics are battle-tested.
Stretch Goals
- Replace the learned positional encoding with sinusoidal and rotary (RoPE) encodings from Section 12.3 and measure how each affects accuracy as you extrapolate to longer sequences than seen in training.
- Add a segment-level recurrence in the style of Transformer-XL from Section 12.4 and measure how much it extends the model's effective context beyond a single window.
- Inspect the attention weights of the trained forecaster and check whether any head specializes on the seasonal lag of the series, connecting the learned weights back to the spectral structure of Chapter 4.
What's Next?
This chapter gave every timestep direct, content-based access to every other, dissolving the recurrence's bottleneck and parallelizing training, but it ran straight into a wall: full attention costs the square of the sequence length, and even the efficient variants that flatten that curve trade away some of the exactness that made attention attractive. Chapter 13: State-Space and Continuous-Time Neural Models attacks the same long-range goal from the opposite side. Instead of starting from all-to-all attention and approximating it down toward linear, it starts from the linear recurrence and the Kalman filter of Chapter 7 and builds it up into a structured state-space model that processes a sequence in linear time, can be unrolled as a recurrence at inference and computed as a long convolution in training, and carries information across very long horizons without the quadratic bill. The observation that closed this chapter, that a linear attention is a recurrence in disguise, is exactly the door into it. The temporal thread comes full circle: the classical linear filter, made trainable in Chapter 10 and challenged by attention here, returns in Chapter 13 as a deep architecture engineered for the long sequences that defeated both.
Bibliography & Further Reading
Foundational Papers
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I. "Attention Is All You Need." NeurIPS, 2017. arXiv:1706.03762. arxiv.org/abs/1706.03762
The paper that introduced the Transformer, multi-head self-attention, and sinusoidal positional encoding, the architecture of Sections 12.2 and 12.3 and the spine of the whole chapter.
Bahdanau, D., Cho, K., Bengio, Y. "Neural Machine Translation by Jointly Learning to Align and Translate." ICLR, 2015. arXiv:1409.0473. arxiv.org/abs/1409.0473
The work that introduced attention as a cure for the encoder-decoder bottleneck of Chapter 10, the conceptual seed from which the self-attention of Section 12.1 grew.
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., Liu, Y. "RoFormer: Enhanced Transformer with Rotary Position Embedding." Neurocomputing, 2024. arXiv:2104.09864. arxiv.org/abs/2104.09864
The rotary position embedding (RoPE) that encodes relative position by rotating queries and keys, now a standard temporal encoding and a centerpiece of Section 12.3.
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., Salakhutdinov, R. "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context." ACL, 2019. arXiv:1901.02860. arxiv.org/abs/1901.02860
The segment-level recurrence and relative positional scheme that extend a Transformer's context past a single window, the memory mechanism of Section 12.4.
Efficient and Sparse Attention
Beltagy, I., Peters, M. E., Cohan, A. "Longformer: The Long-Document Transformer." 2020. arXiv:2004.05150. arxiv.org/abs/2004.05150
A sparse, local-window-plus-global attention that scales linearly with sequence length, a canonical sparse Transformer for the long-sequence problem of Sections 12.4 and 12.5.
Wang, S., Li, B. Z., Khabsa, M., Fang, H., Ma, H. "Linformer: Self-Attention with Linear Complexity." 2020. arXiv:2006.04768. arxiv.org/abs/2006.04768
A low-rank approximation that projects keys and values to a fixed dimension, making attention linear in sequence length, a core efficient variant of Section 12.5.
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., et al. "Rethinking Attention with Performers." ICLR, 2021. arXiv:2009.14794. arxiv.org/abs/2009.14794
The kernelized random-feature attention (FAVOR+) that approximates softmax attention in linear time and memory, a key building block of Section 12.5.
Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F. "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention." ICML, 2020. arXiv:2006.16236. arxiv.org/abs/2006.16236
The linear-attention formulation that recasts a Transformer as a recurrence, the precise observation that closes Section 12.5 and opens the door to the state-space models of Chapter 13.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., Re, C. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." NeurIPS, 2022. arXiv:2205.14135. arxiv.org/abs/2205.14135
The IO-aware exact-attention kernel that cuts memory cost by tiling and recomputation without changing the math, the practical exact-attention speedup behind Section 12.5 and the lab.
Transformers for Time Series
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W. "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting." AAAI, 2021. arXiv:2012.07436. arxiv.org/abs/2012.07436
A sparse-attention Transformer designed specifically for long-horizon forecasting, the direct time-series instance of the efficient-attention argument of Sections 12.4 and 12.5.
Nie, Y., Nguyen, N. H., Sinthong, P., Kalagnanam, J. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (PatchTST)." ICLR, 2023. arXiv:2211.14730. arxiv.org/abs/2211.14730
The patching strategy that turns a series into a short token sequence and a strong forecasting baseline, the design the lab's patch Transformer is built on.
Tools & Libraries
PyTorch Transformer layers: nn.MultiheadAttention, nn.TransformerEncoderLayer, nn.TransformerEncoder, and causal masking. Official documentation. pytorch.org/docs
The drop-in attention and Transformer blocks behind the lab; their masking and fused-kernel support make the Transformer forecaster and the cost comparison a few lines each.