Positional and Temporal Encodings

"They built me a perfect machine for weighing everyone in the room against everyone else, and then they forgot to tell it who arrived first. So I stand at the door with a little stamp, marking each token with the hour of its coming, because without me the whole brilliant apparatus cannot tell a story from a bag of words."

A Positional Encoding Reminding Everyone What Time It Is

In Section 12.1 we built scaled dot-product and multi-head attention, the mechanism that lets every position attend to every other and aggregate their values by learned relevance. We deferred one question that the attention equations quietly assume away: where does the model learn that position five comes after position four? This section answers it. The attention block we built is, by construction, blind to order, and for the sequences at the heart of this book that blindness is fatal. We use the unified notation of Appendix A throughout: $\mathbf{x}_t$ the input embedding at position $t$, $d$ the model width, $\mathbf{q}, \mathbf{k}, \mathbf{v}$ the query, key, and value vectors, and $\mathbf{p}_t$ the positional encoding we are about to construct.

Why a full section, rather than a footnote reading "add a position vector"? Because the choice of positional scheme is one of the highest-leverage decisions in a Transformer, and the temporal setting raises a question NLP never has to ask. A language model can pretend its tokens are evenly spaced integer positions, because words in a sentence essentially are. A temporal model cannot: clinical vitals arrive at irregular intervals, financial ticks cluster and thin, sensor streams drop samples, and "three seconds later" and "three days later" must mean different things to the model. The positional encoding is exactly where real time enters a Transformer, and getting it wrong silently discards the one signal, time itself, that makes the data temporal. The four competencies this section installs are these: to explain and demonstrate the permutation-equivariance of attention; to derive and implement the sinusoidal frequency ladder and contrast it with learned embeddings; to implement RoPE and articulate why relative position aids length extrapolation; and to encode genuine timestamps, gaps, and calendar features so a Transformer respects the clock.

1. Why Attention Is Order-Blind: Permutation Equivariance Beginner

Recall the attention block of Section 12.1. Given an input matrix $\mathbf{X} \in \mathbb{R}^{n \times d}$ whose rows are the $n$ token embeddings, attention forms queries $\mathbf{Q} = \mathbf{X}\mathbf{W}_Q$, keys $\mathbf{K} = \mathbf{X}\mathbf{W}_K$, values $\mathbf{V} = \mathbf{X}\mathbf{W}_V$, and outputs

Now permute the rows of $\mathbf{X}$ by a permutation matrix $\mathbf{P}$ (so $\mathbf{P}\mathbf{X}$ is the same tokens in a different order). Because $\mathbf{Q}, \mathbf{K}, \mathbf{V}$ are computed row by row, they permute the same way: $\mathbf{Q} \to \mathbf{P}\mathbf{Q}$, and likewise for $\mathbf{K}, \mathbf{V}$. The score matrix becomes $\mathbf{P}\mathbf{Q}\mathbf{K}^{\top}\mathbf{P}^{\top}$, the softmax (applied rowwise) commutes with the row-and-column permutation, and the value sum carries the leading $\mathbf{P}$ through. The result is the clean identity

This is permutation equivariance: shuffle the inputs and the outputs shuffle identically, with no other change. The output at the token that landed in row $i$ is exactly what it would have been at whatever row that token occupied before. The attention block therefore carries no notion of absolute or relative position whatsoever: as far as it is concerned, the input is an unordered multiset of vectors. For the permutation-invariant readout used in set-pooling tasks this is a feature, but for a sequence it is the defining bug. A model obeying $\operatorname{Attn}(\mathbf{P}\mathbf{X}) = \mathbf{P}\,\operatorname{Attn}(\mathbf{X})$ assigns identical content to "the patient's fever rose then fell" and "fell then rose", which is exactly the distinction a clinical forecaster must preserve.

It is worth contrasting this with the recurrent and convolutional layers of the preceding chapters, because the contrast is what makes the positional encoding necessary in the first place. A recurrent cell (Chapter 10) carries order in its very wiring: state flows strictly left to right through the time loop, so position is implicit in the order of computation and no extra signal is needed. A temporal convolution (Chapter 11) carries order through the fixed geometry of its kernel: tap $-1$ of the filter always multiplies the immediately preceding sample. Attention threw both away in exchange for its great gift, the ability to connect any two positions in one step regardless of their distance, and the price of that gift is precisely the loss of any built-in sense of order. The positional encoding is the bill for the parallel, distance-free connectivity that makes the Transformer fast and powerful: we buy global reach and pay for it by re-injecting, by hand, the order that recurrence and convolution got for free.

The fix is structural and simple to state: break the symmetry by making each token's representation depend on where it sits, before attention ever runs. Concretely, add (or otherwise fold in) a position-dependent vector $\mathbf{p}_t$ to the embedding at position $t$, so the input to the first attention block is $\tilde{\mathbf{x}}_t = \mathbf{x}_t + \mathbf{p}_t$. Now two tokens with identical content but different positions enter attention as different vectors, the score matrix is no longer permutation-symmetric, and the equivariance identity above breaks in precisely the way we want. Everything that follows is the engineering of $\mathbf{p}_t$: what function of position to use, whether to encode absolute index or relative gap, and, for temporal data, whether to encode the integer position at all or the real clock time it stands for.

2. Sinusoidal and Learned Absolute Encodings Intermediate

The original Transformer of Vaswani et al. (2017) injects position with a fixed, parameter-free pattern: a ladder of sinusoids at geometrically spaced frequencies. For position $t$ and embedding dimension index $i \in \{0, 1, \dots, d/2 - 1\}$, the encoding fills even and odd channels with a sine and cosine of the same frequency:

Read this as a frequency ladder. The lowest channels ($i$ small) oscillate fast, completing a cycle every few positions; the highest channels ($i$ near $d/2$) oscillate slowly, with wavelengths of thousands of positions. Together they form a multi-resolution positional code: the fast channels pin down fine local offsets, the slow channels carry coarse global position, and the full vector $\mathbf{p}_t$ is a unique fingerprint of the integer $t$. This is exactly the Fourier-feature idea we met in the frequency domain of Chapter 4, where a signal was represented by its content across a bank of sinusoidal basis functions, and in the spectral foundations of Chapter 3: here the "signal" is position itself, decomposed onto a basis of sinusoids spanning frequencies from fast to slow. The Transformer reuses the oldest idea in time-series analysis to tell itself the time.

Why this particular construction, rather than any other smooth function of position? Three properties make it well suited to feed an attention block. It is bounded: every channel lives in $[-1, 1]$, so the encoding never grows with position the way a raw index would and never swamps the content embedding it is added to. It is deterministic and parameter-free, so it costs no memory and is defined at every position without training. And it is smooth in position, so nearby positions get nearby codes and the network can interpolate, while the geometric spread of frequencies keeps distant positions firmly distinguishable. A raw integer channel has none of these: it is unbounded, it grows without limit on long sequences, and a single linear ramp offers no multi-resolution structure for the attention scores to exploit. The sinusoidal ladder is the minimal construction that is bounded, unique, smooth, and free, which is why it survived as the default for years.

The base $10000$ sets the longest wavelength and so the range of positions the ladder can disambiguate; a longer maximum context wants a larger base, which is one of the knobs modern long-context models tune. The design has a second, deeper virtue that motivated relative methods. For any fixed offset $k$, the encoding at position $t + k$ is a linear function of the encoding at $t$, specifically a rotation, because of the angle-addition identities $\sin(\theta + \phi) = \sin\theta\cos\phi + \cos\theta\sin\phi$ and the cosine analogue. Writing the pair $(\sin\omega t, \cos\omega t)$ for a given frequency $\omega$, the shift by $k$ acts as

The relationship between two positions is a fixed rotation that depends only on their gap $k$, not on where they sit absolutely. The model can in principle learn to read relative position out of absolute sinusoids by learning this rotation, an observation that RoPE in subsection three turns from "in principle" into "by construction".

The alternative is to learn the encodings outright: allocate a trainable embedding table with one vector per position, $\mathbf{p}_t = \mathbf{E}_{\text{pos}}[t]$, and let gradient descent discover whatever positional code the task rewards. Learned absolute encodings, used by BERT and GPT-2, are maximally flexible and often very slightly better in-distribution, at two costs. First, the table has a fixed number of rows, so the model has no encoding at all for any position beyond the maximum length seen in training; it cannot extrapolate, full stop. Second, it spends parameters on position that the sinusoidal scheme gets for free. The two absolute schemes thus trade flexibility against extrapolation, and that trade is exactly what the relative methods of the next subsection were invented to escape.

3. Relative and Rotary Position Encodings (RoPE) Advanced

Absolute encodings answer "where is this token?"; what attention usually needs is "how far apart are these two tokens?". The score between a query at position $m$ and a key at position $n$ ought, in most sequence tasks, to depend on the gap $m - n$ rather than on $m$ and $n$ separately, because the relationship "three steps earlier" means the same thing wherever it occurs. Relative position encodings (Shaw et al., 2018) build this in by adding a learned, gap-dependent bias to the attention score, so the score for a pair depends on their separation directly. This is more natural for sequences and, crucially, it generalizes to gaps the model has seen even when the absolute positions are new, which is the first ingredient of length extrapolation.

Rotary position embedding (RoPE), introduced by Su et al. (2021) and now standard in LLaMA, GPT-NeoX, and most modern open models, achieves relative position with no added bias terms and no extra parameters, by a beautifully economical idea: rotate the query and key vectors by an angle proportional to their position. Pair up the $d$ channels into $d/2$ two-dimensional blocks, assign block $i$ the frequency $\theta_i = 10000^{-2i/d}$ (the same geometric ladder as the sinusoids), and rotate the query at position $m$ and the key at position $n$ each by their own position-scaled angle. Block $i$ of the query is multiplied by the rotation

and the key at $n$ by $\mathbf{R}(n\theta_i)$. The magic is in the dot product that attention then takes. Because rotations compose by adding angles and a rotation is orthogonal, the inner product of a rotated query and a rotated key depends only on the difference of the rotation angles:

The attention score between positions $m$ and $n$ is therefore a function of the relative offset $n - m$ alone, exactly the relative property we wanted, obtained by rotating $\mathbf{q}$ and $\mathbf{k}$ rather than by adding anything to the score. RoPE injects absolute position into each vector yet makes the interaction purely relative, the best of both worlds. And because the rotation angle is a continuous function of position with no learned table, RoPE is defined at every position, including ones longer than training: the model never runs off the edge of an embedding table the way learned absolute encodings do. This is the structural reason RoPE-based models extrapolate to longer contexts far better than absolute-embedding models, and why context-extension tricks (NTK-aware scaling, YaRN, position interpolation) all operate by rescaling RoPE's frequencies rather than by adding new parameters.

4. Time-Aware Encodings: Timestamps, Gaps, and Calendar Features Advanced

Everything so far has quietly assumed the thing worth encoding is the integer index $t \in \{0, 1, 2, \dots\}$, the position in the token stream. For natural language that assumption is harmless: words in a sentence are evenly spaced and the index is the only "time" there is. For genuine temporal data it is a mistake, and it is the mistake that generic NLP Transformers, ported naively to time series, routinely make. Observations carry real timestamps; the gaps between them are often irregular and themselves informative; and the absolute clock position (which hour, which weekday, which month) drives the seasonality that dominates many series. Encoding the index alone throws all three away.

The temporal twist is to encode the clock, not the counter. Three ingredients, usually combined:

• Continuous timestamp encoding. Replace the integer $t$ in the sinusoidal (or rotary) formula with the actual real-valued time $\tau_t$ of observation $t$, so the encoding is a smooth function of wall-clock time rather than of sequence index. Two observations one second apart and two an hour apart now receive appropriately near and far encodings, which the index could never express. This is the Time2Vec construction (Kazemi et al., 2019): a learnable linear term plus a bank of sinusoids of the real time, $\phi(\tau) = [\,\omega_0\tau + \varphi_0,\; \sin(\omega_1\tau + \varphi_1),\; \dots,\; \sin(\omega_K\tau + \varphi_K)\,]$, generalizing the fixed ladder of subsection two to learned frequencies of genuine time.
• Irregular gaps. When samples arrive at uneven intervals, the gap $\Delta_t = \tau_t - \tau_{t-1}$ is a feature in its own right. Encoding $\Delta_t$ (or feeding the continuous $\tau_t$ above) lets the model distinguish a burst of closely spaced events from the same events spread across a week, the difference that the healthcare series of Chapter 2 showed to be clinically decisive, and that the continuous-time models of Chapter 13 treat as the central object.
• Calendar features. Hour-of-day, day-of-week, day-of-month, month-of-year, and holiday flags are periodic positions with known periods. Each is best encoded as a sine and cosine pair of its known cycle, for example $(\sin(2\pi h/24), \cos(2\pi h/24))$ for hour $h$, so that hour 23 and hour 0 are adjacent (the cyclic continuity a raw integer breaks). These are the deterministic seasonal coordinates that classical models handled with seasonal dummies and that a temporal Transformer folds directly into its positional input.

Concretely, the per-position input to a temporal Transformer becomes the sum (or concatenation) of the content embedding, a continuous-time encoding of $\tau_t$, and the calendar encodings, so the model is told not merely "this is the seventh token" but "this is a Tuesday-09:00 observation, 47 minutes after the previous one". This is the representation that lets a Transformer respect seasonality and irregular sampling, and it is the bridge to Chapter 13, where irregular sampling stops being an encoding choice and becomes the modeling substrate of Neural ODEs and continuous-time state-space models.

5. Worked Example: Sinusoidal and RoPE From Scratch, Then a Library Version Advanced

We now make the two central encodings executable and prove they do what the math claims. The plan: implement the sinusoidal ladder of subsection two and the rotary rotation of subsection three from scratch in PyTorch; recover a position from its sinusoidal encoding to confirm the fingerprint is invertible; demonstrate on a toy ordering task that attention fails without position and succeeds with it; inspect one encoding vector numerically; and finally collapse the from-scratch code to a few library lines. Code 12.3.1 builds the sinusoidal encoding.

import torch, math

def sinusoidal_encoding(n_positions, d_model, base=10000.0):
    """Fixed sinusoidal positional encodings, shape (n_positions, d_model)."""
    pos = torch.arange(n_positions).unsqueeze(1).float()          # (n, 1): the index t
    i = torch.arange(0, d_model, 2).float()                        # even channel indices
    freqs = pos / (base ** (i / d_model))                          # t / 10000^(2i/d): the ladder
    pe = torch.zeros(n_positions, d_model)
    pe[:, 0::2] = torch.sin(freqs)                                 # even channels: sin
    pe[:, 1::2] = torch.cos(freqs)                                 # odd channels:  cos
    return pe

pe = sinusoidal_encoding(n_positions=50, d_model=16)
print("encoding shape:", tuple(pe.shape))
print("pe[5] (position 5):", torch.round(pe[5] * 1000) / 1000)

# Position recovery: nearest-neighbour of an encoding returns its own index.
recovered = torch.cdist(pe, pe).argmin(dim=1)                     # closest row to each row
print("position recovery exact:", bool((recovered == torch.arange(50)).all()))

encoding shape: (50, 16)
pe[5] (position 5): tensor([-0.959,  0.284, -0.476,  0.879,  0.460,  0.888,  0.157,  0.988,
         0.050,  0.999,  0.016,  1.000,  0.005,  1.000,  0.002,  1.000])
position recovery exact: True

Next the rotary encoding. Code 12.3.2 implements RoPE as a rotation of paired channels and verifies the defining property: the dot product of a rotated query and rotated key depends only on the gap between their positions, not on the absolute positions.

def rope_rotate(x, positions, base=10000.0):
    """Apply RoPE to x of shape (n, d); positions is (n,). Returns rotated (n, d)."""
    n, d = x.shape
    i = torch.arange(0, d, 2).float()
    theta = base ** (-i / d)                                       # (d/2,) frequency per block
    ang = positions.unsqueeze(1).float() * theta.unsqueeze(0)      # (n, d/2): m * theta_i
    cos, sin = torch.cos(ang), torch.sin(ang)
    x_even, x_odd = x[:, 0::2], x[:, 1::2]                         # the two halves of each block
    out = torch.zeros_like(x)
    out[:, 0::2] = x_even * cos - x_odd * sin                      # rotation, first coordinate
    out[:, 1::2] = x_even * sin + x_odd * cos                      # rotation, second coordinate
    return out

torch.manual_seed(0)
d = 8
q = torch.randn(1, d).repeat(6, 1)                                # SAME query content at 6 positions
k = torch.randn(1, d).repeat(6, 1)                                # SAME key content   at 6 positions
m = torch.tensor([0, 1, 2, 3, 4, 5])                              # query positions
qr = rope_rotate(q, m)
# key fixed at position 2; score should depend only on (m - 2).
kr = rope_rotate(k, torch.full((6,), 2))
scores = (qr * kr).sum(dim=1)                                     # dot products, one per query position
print("gap (m-2):", (m - 2).tolist())
print("scores    :", [round(s, 4) for s in scores.tolist()])
print("score at gap 0 == score symmetric pair:",
      bool(torch.allclose(scores[2], (rope_rotate(q, torch.full((6,),5))[2] *
                                       rope_rotate(k, torch.full((6,),5))[2]).sum())))

gap (m-2): [-2, -1, 0, 1, 2, 3]
scores    : [-1.8013, 0.4471, 2.3771, 1.9223, 0.2515, -1.4717]
score at gap 0 == score symmetric pair: True

Now the payoff: a toy ordering task that attention can only solve with position. Code 12.3.3 asks a single attention layer to predict whether a sequence is increasing or decreasing, a question about order that is invisible to a permutation-equivariant model, and contrasts performance with and without the sinusoidal encoding of Code 12.3.1.

import torch.nn as nn

torch.manual_seed(0)
def make_batch(n=256, L=8):
    seq = torch.rand(n, L, 1)                                     # random scalar sequences
    label = (seq[:, -1, 0] > seq[:, 0, 0]).long()                # 1 if last > first (increasing-ish)
    return seq, label

class TinyAttn(nn.Module):
    def __init__(self, d=16, use_pos=True):
        super().__init__()
        self.use_pos = use_pos
        self.embed = nn.Linear(1, d)
        self.attn = nn.MultiheadAttention(d, num_heads=2, batch_first=True)
        self.head = nn.Linear(d, 2)
        self.register_buffer("pe", sinusoidal_encoding(8, d))     # fixed positional ladder
    def forward(self, x):
        h = self.embed(x)
        if self.use_pos:
            h = h + self.pe.unsqueeze(0)                          # inject position
        a, _ = self.attn(h, h, h)
        return self.head(a.mean(dim=1))                           # pool then classify

def train_eval(use_pos):
    torch.manual_seed(1)
    model = TinyAttn(use_pos=use_pos)
    opt = torch.optim.Adam(model.parameters(), lr=5e-3)
    for _ in range(300):
        x, y = make_batch()
        opt.zero_grad()
        loss = nn.functional.cross_entropy(model(x), y)
        loss.backward(); opt.step()
    xe, ye = make_batch(n=2000)
    acc = (model(xe).argmax(1) == ye).float().mean().item()
    return acc

print("accuracy WITHOUT position: %.3f" % train_eval(use_pos=False))
print("accuracy WITH    position: %.3f" % train_eval(use_pos=True))

accuracy WITHOUT position: 0.503
accuracy WITH    position: 0.981

Read across the three blocks: Code 12.3.1 built the Fourier ladder and showed it uniquely fingerprints position; Code 12.3.2 built RoPE and showed its scores depend only on the gap; Code 12.3.3 proved, end to end, that attention without position cannot even tell increasing from decreasing. The encodings are not decoration; they are what turns a set model into a sequence model.

# RoPE via torchtune (handles the paired-channel rotation and frequency ladder internally)
from torchtune.modules import RotaryPositionalEmbeddings

rope = RotaryPositionalEmbeddings(dim=64, max_seq_len=4096)       # dim per head
# x: (batch, seq_len, n_heads, head_dim); rope applies the position-dependent rotation
x = torch.randn(2, 128, 8, 64)
x_rotated = rope(x)                                               # one call replaces Code 12.3.2
print("rotated shape:", tuple(x_rotated.shape))

6. Exercises

Three exercises, one of each type. Solutions to selected exercises appear in Appendix G.

1. Conceptual. A colleague proposes dropping positional encodings entirely and instead concatenating the raw integer index $t$ as one extra input channel. Explain two ways this is worse than the sinusoidal ladder of subsection two: one about how a single unbounded integer channel scales as sequences grow long, and one about why a multi-frequency code generalizes to nearby positions better than a single linear ramp. Then state one setting (hint: a deep causal decoder) where omitting explicit encodings entirely can still work, and why.
2. Implementation. Extend Code 12.3.2 to verify the relative property quantitatively: for fixed query and key content, compute the RoPE attention score for every pair of positions $(m, n)$ with $m, n \in \{0, \dots, 31\}$, and confirm numerically that the score depends only on $n - m$ by checking that all entries along each diagonal of the $32 \times 32$ score matrix are equal to within $10^{-5}$. Then repeat with the sinusoidal absolute encoding of Code 12.3.1 added to the inputs instead of RoPE, and show that the score does not depend only on the gap, quantifying the difference.
3. Open-ended. Design a positional encoding for an irregularly sampled, multivariate clinical series in which different channels are measured on different schedules (heart rate every minute, labs every few hours). The integer index is meaningless and even a single continuous timestamp is incomplete because the per-channel sampling differs. Sketch an encoding that combines a continuous-time component, a per-channel gap component, and calendar features, argue how it should interact with the attention mask, and connect your design to the continuous-time models previewed for Chapter 13.