Robustness and Adversarial Attacks on Time Series

"I was shown a heartbeat trace I had classified ten thousand times as normal sinus rhythm. Someone added a ripple smaller than the sensor's own noise floor, a wiggle no cardiologist would ever see, and asked me again. I said ventricular tachycardia with ninety-nine percent confidence. The trace was the same trace. I had simply been convinced it was a different day entirely."

A Tiny Perturbation That Convinced the Model It Was a Different Day Entirely

In Section 33.1 we opened the model to make its temporal predictions interpretable: attribution over time, attention as explanation, and the caveats that come with both. Interpretability answers "why did the model say that". This section answers a sharper and more adversarial question: "can someone make the model say something false on purpose, and how do we stop them". The two are linked. The same gradient that a saliency map visualizes is the gradient an attacker ascends to build a perturbation, so the interpretability machinery of the previous section is, read backward, the attack machinery of this one. We will use the gradient of the loss with respect to the input, $\nabla_{\mathbf{x}}\mathcal{L}$, as the single workhorse object, exactly as Section 33.1 did, but now to break the model rather than to explain it.

Why does a textbook on temporal AI need a dedicated section on adversarial robustness, beyond the general treatment any deep-learning course gives? Because the systems Part VIII is about deploying are disproportionately attractive and disproportionately exposed. A forecaster sets a price; an anomaly detector triggers an alert or suppresses one; a control policy moves a physical actuator. Each is a place where a small, well-chosen lie in the input converts directly into money, into a missed alarm, or into a physical action, and each consumes a stream that an adversary may partly control. The natural-shift robustness of Chapter 20 and the adversarial robustness of this section are the two halves of what it takes to trust a temporal model in the world, and a deployed system needs both.

The competencies this section installs are four. To separate the two robustness notions cleanly and say which mechanism each calls for. To write FGSM and PGD as constrained gradient ascent on the loss and to state the extra constraints, smoothness, causality, imperceptibility, that make a time-series attack distinctive. To name the temporal-specific threat models, poisoning, forecasting evasion, and policy-observation attacks, and connect them to the rest of the book. And to run an attack and a defense end to end, measuring the budget-versus-success-rate curve that quantifies fragility. These are the load-bearing skills for Section 33.3 on fairness and for the deployment chapter that follows.

1. Two Kinds of Robustness: Natural Shift and Adversarial Perturbation Beginner

Robustness is an overloaded word, and the first discipline is to split it into two notions that look similar from a distance and demand opposite mindsets up close. Natural robustness is robustness to perturbations the world produces on its own and does not choose to hurt you: the slow distribution drift that Chapter 20 studies (the data-generating process changes, last year's regime is not this year's), additive measurement noise, missing samples from a dropped packet, a stuck or miscalibrated sensor. These perturbations are typically high-entropy and indifferent: noise points in a random direction, drift moves the whole distribution, and a model that generalizes well absorbs them because they look like the variation it already saw in training.

Adversarial robustness is robustness to perturbations a thinking opponent constructs to maximize your loss. The defining difference is not size but direction. Random noise of a given magnitude is almost always harmless because it spreads its energy across all input directions, most of which the model ignores; an adversarial perturbation of the same magnitude concentrates all of its energy in the single worst direction, the one in which the decision boundary is nearest. This is why a model can be provably stable to noise with standard deviation $0.1$ and catastrophically broken by an adversarial perturbation of norm $0.01$: the noise is averaged over a benign sphere, the attack is a spear aimed at the boundary. The two notions even pull defenses in opposite ways, as Figure 33.2.1 lays out.

Temporal, financial, and control systems are unusually attractive targets, and it is worth being concrete about why. They sit on the critical path of decisions with direct payoff: a forecaster that an attacker can nudge upward by a fraction of a percent is a money pump if it sets a quoted price or a trading signal; an anomaly detector an attacker can keep quiet is a way to hide an intrusion in the sensor stream of Chapter 8; a control policy whose observations can be perturbed can be steered into an unsafe action. They also consume streaming, partially observable inputs over channels an adversary may co-own (a public market feed, a shared sensor bus, a network link), so the attacker frequently has genuine write access to part of the input. And because the input is a signal a human occasionally audits, the attacker's incentive is to stay imperceptible in that signal, which is a sharper and more interesting constraint than staying imperceptible in a static image.

2. Adversarial Examples for Time Series: FGSM and PGD Intermediate

An adversarial example is an input $\mathbf{x}' = \mathbf{x} + \boldsymbol{\delta}$, close to a real input $\mathbf{x}$ under some norm, on which the model errs: a classifier flips its label, or a forecaster's output is dragged toward a target the attacker wants. "Close" is formalized by a budget $\epsilon$ and a norm, almost always the $\ell_\infty$ norm $\|\boldsymbol{\delta}\|_\infty \le \epsilon$ (every timestep may be nudged by at most $\epsilon$) or the $\ell_2$ norm $\|\boldsymbol{\delta}\|_2 \le \epsilon$ (the total energy of the perturbation is bounded). The attacker's problem is a constrained optimization: maximize the loss the defender minimizes, subject to staying within the budget,

where $f_\theta$ is the trained model, $y$ the true label, and $\mathcal{L}$ the same loss used in training. This is the adversary's mirror image of empirical risk minimization, and every attack below is a different way of approximately solving it.

Fast Gradient Sign Method (FGSM). The cheapest attack takes a single step. Linearize the loss around $\mathbf{x}$: to first order, $\mathcal{L}(\mathbf{x}+\boldsymbol{\delta}) \approx \mathcal{L}(\mathbf{x}) + \boldsymbol{\delta}^\top \nabla_{\mathbf{x}}\mathcal{L}$. Under the $\ell_\infty$ budget $\|\boldsymbol{\delta}\|_\infty\le\epsilon$, the $\boldsymbol{\delta}$ that maximizes this linear form is the one that points each coordinate in the direction of its gradient sign, at full budget. That gives the one-shot perturbation

One gradient, one sign, one step of size $\epsilon$. FGSM (Goodfellow, Shlens, and Szegedy, 2015) is fast and, on undefended models, alarmingly effective, but because it trusts the linearization globally it is a weak attack against any model whose loss surface curves.

Projected Gradient Descent (PGD). The strong attack iterates FGSM and projects back into the budget after each step. With step size $\alpha$, projection $\Pi_{\mathcal{B}_\epsilon}$ onto the $\epsilon$-ball around $\mathbf{x}$, and $N$ steps,

For the $\ell_\infty$ ball the projection is a coordinatewise clip to $[\mathbf{x}-\epsilon, \mathbf{x}+\epsilon]$. PGD (Madry et al., 2018) is the standard strong first-order attack: by re-linearizing at each step it follows the curved loss surface where FGSM's single linearization fails, and the random start escapes the trivial degeneracy at $\mathbf{x}$. PGD is both the benchmark attack and, run inside the training loop, the basis of the strongest empirical defense in subsection four.

So far this is the image-attack story unchanged. What makes a time-series adversarial example distinctive is the set of constraints the perturbation must satisfy to count as a real threat, and these have no clean analogue in the static-image setting:

• Smoothness and spectral plausibility. A raw $\operatorname{sign}$ perturbation is white noise: it injects energy at every frequency, including frequencies a real signal never contains, so it stands out to any spectral check (the frequency-domain tools of Chapter 4 are a cheap detector). A serious time-series attack low-pass-filters or otherwise band-limits $\boldsymbol{\delta}$ so the adversarial signal still looks like something a sensor could have produced.
• Causality. In a streaming or control setting the attacker can only edit the present and future, not rewrite history that the model has already consumed. A causal attack constrains $\boldsymbol{\delta}_t = 0$ for $t$ already past, and may only perturb the most recent window, which is a strictly harder optimization than the offline, full-sequence attack.
• Imperceptibility in the signal, not the pixel. "Small" means small to whoever audits the waveform. For an ECG that is a clinician, for a price tape an analyst, for a vibration trace a maintenance engineer. The right budget is set in the units and perceptual frame of the domain, and an $\ell_\infty$ ball is only a crude proxy for it. Some attacks instead bound a perceptual or dynamic-time-warping distance.
• Sparsity in time. An attacker who controls only a few timesteps (a brief window of write access) needs a perturbation supported on those steps, an $\ell_0$-style constraint that concentrates the attack rather than spreading it.

Adding these constraints turns the clean $\ell_p$ ball into a smaller, oddly shaped feasible set, and a competent time-series attack is PGD plus a projection onto that set at every step. The good news for the defender is that each constraint the attacker must respect also narrows the space of perturbations to defend against; the bad news is that the constrained attacks that survive are exactly the ones that look like plausible signals and slip past naive checks.

3. Attacks Specific to Temporal Systems Intermediate

FGSM and PGD are evasion attacks at test time on a classifier, the most studied case. But a deployed temporal system exposes attack surfaces a static classifier does not, because it trains on a stream, forecasts the future, and sometimes acts on the world. Three threat models matter.

Poisoning the training stream. A model that learns online or is periodically retrained on incoming data (the continual setting of Chapter 20) can be attacked at training time. The adversary injects crafted samples into the stream so that the updated model behaves as the attacker wants: a backdoor that misclassifies any input carrying a specific trigger pattern, or a slow drift that degrades the model while looking like ordinary nonstationarity. Streaming poisoning is especially dangerous because the usual defense against drift, "just keep learning from the new data", is precisely the mechanism the attacker exploits. The line between malicious poisoning and benign concept drift is genuinely thin, and telling them apart is an open problem that connects this section directly to the drift-detection machinery of Chapter 8.

Evasion in forecasting. Against a regressor the attacker's goal is not a flipped label but a steered prediction. The objective swaps the classification loss for a targeting loss that pulls the forecast $\hat{\mathbf{y}}$ toward an attacker-chosen target $\mathbf{y}^{\text{target}}$, for instance $\mathcal{L}_{\text{atk}} = \|f_\theta(\mathbf{x}+\boldsymbol{\delta}) - \mathbf{y}^{\text{target}}\|^2$, minimized over $\boldsymbol{\delta}$ within budget. The same FGSM and PGD machinery applies, only the sign flips (the attacker now descends a targeting loss). A perturbation to the recent history of a price or load series that nudges the next-step forecast up or down by a controlled amount is the canonical money-relevant attack, and because forecasts feed downstream decisions, a small steered error compounds.

Attacks on RL policies and observations. A control or reinforcement-learning agent (the policies of Part VI) reads an observation $\mathbf{o}_t$ and emits an action $\mathbf{a}_t = \pi_\theta(\mathbf{o}_t)$. An observation attack perturbs $\mathbf{o}_t$ within a budget to change the action, $\arg\max_{\|\boldsymbol{\delta}\|\le\epsilon}\,\big[\text{some measure of how bad } \pi_\theta(\mathbf{o}_t+\boldsymbol{\delta}) \text{ is}\big]$, where "bad" might mean far from the optimal action, or minimizing the agent's expected return, or steering toward a specific unsafe action. Because the agent acts in a feedback loop, a perturbation at one step changes the state distribution at the next, so temporal attacks on policies compound through the dynamics in a way a one-shot classifier attack does not: a small early nudge can cascade into a large trajectory deviation. This is the adversarial face of the robustness concerns that the optimal-control and imitation methods of Chapter 27 must contend with.

4. Defenses: Adversarial Training, Smoothing, Certificates, Detection Advanced

Defenses fall into four families, and the honest framing is that only some of them come with guarantees while the rest are empirical arms-race moves. We take them in order of how much they promise.

Adversarial training is the strongest empirical defense and the most expensive. The idea (Madry et al., 2018) is to redefine the training objective as a saddle point: minimize the loss against the worst-case perturbation within the budget, rather than against the clean input. Formally the defender solves

where the inner maximization is approximated by running PGD on each batch and the outer minimization is ordinary SGD on the resulting adversarial examples. Each training step now contains an attack, so adversarial training costs roughly $N+1$ times a normal step for $N$-step PGD, but it produces models whose loss surface is genuinely flatter near the data, which is the only known way to get broad empirical robustness. We demonstrate exactly this in subsection five.

Input smoothing and denoising sit in front of the model and try to remove the perturbation before it reaches the classifier. For time series the natural smoothers are the ones the book already taught: a low-pass or moving-average filter, a wavelet shrinkage, or the denoising autoencoders and reconstruction-based cleaners of Section 4.3 and the anomaly-aware reconstruction of Section 8.2. Because a white-noise $\operatorname{sign}$ perturbation lives at high frequencies and a real signal does not, a low-pass filter strips much of an unconstrained attack for almost no cost. The weakness is the obvious one: an attacker who knows the smoother simply optimizes through it (a band-limited PGD attack, subsection two), so smoothing raises the bar without certifying anything.

Certified robustness by randomized smoothing is the one defense in this list that comes with a provable guarantee. Instead of classifying $\mathbf{x}$ directly, build a smoothed classifier $g(\mathbf{x})$ that returns the class the base classifier $f$ outputs most often under Gaussian noise added to the input, $g(\mathbf{x}) = \arg\max_c\, \Pr_{\boldsymbol{\eta}\sim\mathcal{N}(0,\sigma^2 I)}\big[f(\mathbf{x}+\boldsymbol{\eta}) = c\big]$. Cohen, Rosenfeld, and Kolter (2019) proved that $g$ is certifiably robust: if the top class wins by a large enough margin under the noise, then no $\ell_2$ perturbation of radius below a computable $R$ can change $g$'s prediction. The certified radius grows with the noise level $\sigma$ and with the margin, which is the formal statement of the trade we keep meeting: more smoothing buys a larger guarantee at the cost of clean accuracy. For time series the Gaussian noise must respect the signal's structure to keep the base accuracy usable, an active research wrinkle, but the guarantee is real and that makes randomized smoothing qualitatively different from the empirical defenses around it.

Detection declines to classify suspicious inputs at all. A detector learns to tell adversarial inputs from clean ones, by their statistics (an adversarial perturbation often leaves a spectral or kurtosis fingerprint), by disagreement among an ensemble, or by anomalous activations, and routes flagged inputs to a human or a safe default. Detection is attractive when abstaining is cheap (refusing one trade is fine) and weak when it is not, and like smoothing it can be evaded by an attacker who adds "look clean to the detector" to the objective. Detection composes well with the others: a detector as a cheap first line, an adversarially trained model behind it.

5. Worked Example: Attack a Classifier From Scratch, Then Defend It Advanced

We now make the whole story executable on a small synthetic two-class problem: sequences that are either a low-frequency sine (class 0) or the same sine with a faint higher-frequency component (class 1), classified by a small 1D-convolutional network. The plan mirrors the section. First train the classifier and record its clean accuracy. Then implement FGSM and PGD from scratch and measure how accuracy collapses as the budget $\epsilon$ grows, the budget-versus-success-rate curve. Then reproduce the same attacks in three lines of torchattacks to show the library equivalence. Finally adversarially train the model and watch the robust accuracy recover. Code 33.2.1 sets up the data and the trained victim.

import torch, torch.nn as nn, torch.nn.functional as F

torch.manual_seed(0)
T = 64                                          # sequence length
def make_batch(n):
    t = torch.linspace(0, 6.28, T)
    y = torch.randint(0, 2, (n,))               # class label
    base = torch.sin(t).repeat(n, 1)            # shared low-freq carrier
    # class 1 adds a faint high-frequency component; class 0 does not
    hi = 0.3 * torch.sin(5 * t).repeat(n, 1) * y.unsqueeze(1)
    x = (base + hi + 0.1 * torch.randn(n, T)).unsqueeze(1)   # (n, 1, T)
    return x, y

class TinyCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.c1 = nn.Conv1d(1, 8, 5, padding=2); self.c2 = nn.Conv1d(8, 16, 5, padding=2)
        self.fc = nn.Linear(16, 2)
    def forward(self, x):
        x = F.relu(self.c1(x)); x = F.relu(self.c2(x))
        return self.fc(x.mean(-1))              # global average pool over time, then classify

model = TinyCNN()
opt = torch.optim.Adam(model.parameters(), 1e-2)
for _ in range(300):                            # train the victim to high clean accuracy
    x, y = make_batch(128)
    opt.zero_grad(); loss = F.cross_entropy(model(x), y); loss.backward(); opt.step()

xte, yte = make_batch(2000)                     # fixed test set for all attacks below
clean_acc = (model(xte).argmax(1) == yte).float().mean().item()
print("clean accuracy = %.3f" % clean_acc)

clean accuracy = 0.991

Now the attacks, written from the equations of subsection two. Code 33.2.2 implements FGSM and PGD by hand: take the input gradient, step along its sign, and for PGD iterate with projection back into the $\ell_\infty$ ball. We then sweep the budget $\epsilon$ and record accuracy under attack, which is one minus the attack success rate.

def fgsm(model, x, y, eps):
    x = x.clone().detach().requires_grad_(True)
    loss = F.cross_entropy(model(x), y)
    loss.backward()                             # nabla_x L
    return (x + eps * x.grad.sign()).detach()   # one signed step of size eps

def pgd(model, x, y, eps, alpha, steps=10):
    x0 = x.clone().detach()
    xadv = x0 + 0.001 * torch.randn_like(x0)    # small random start (subsection 2)
    for _ in range(steps):
        xadv = xadv.clone().detach().requires_grad_(True)
        loss = F.cross_entropy(model(xadv), y)
        loss.backward()
        xadv = xadv + alpha * xadv.grad.sign()  # FGSM step
        xadv = x0 + (xadv - x0).clamp(-eps, eps)  # project onto the L_inf eps-ball
    return xadv.detach()

def acc_under(attack, eps):
    xadv = attack(model, xte, yte, eps) if attack is fgsm \
           else attack(model, xte, yte, eps, alpha=eps/4)
    return (model(xadv).argmax(1) == yte).float().mean().item()

print("eps    FGSM_acc  PGD_acc")
for eps in [0.0, 0.02, 0.05, 0.10, 0.20]:
    print("%.2f   %.3f     %.3f" % (eps, acc_under(fgsm, eps), acc_under(pgd, eps)))

eps    FGSM_acc  PGD_acc
0.00   0.991     0.991
0.02   0.873     0.812
0.05   0.642     0.418
0.10   0.391     0.122
0.20   0.218     0.039

The from-scratch fgsm and pgd above are about 18 lines of gradient and projection bookkeeping. A library collapses them to a constructor call. Code 33.2.3 reproduces the identical attacks with torchattacks, which handles the gradient, the sign step, the projection, and the random restart internally and exposes them as one object each.

import torchattacks                              # pip install torchattacks

atk_fgsm = torchattacks.FGSM(model, eps=0.05)    # the whole of fgsm(), one line
atk_pgd  = torchattacks.PGD(model, eps=0.05, alpha=0.05/4, steps=10)  # all of pgd(), one line

for name, atk in [("FGSM", atk_fgsm), ("PGD", atk_pgd)]:
    xadv = atk(xte, yte)                          # craft adversarial batch
    acc = (model(xadv).argmax(1) == yte).float().mean().item()
    print("torchattacks %-4s acc @ eps=0.05 = %.3f" % (name, acc))

torchattacks FGSM acc @ eps=0.05 = 0.641
torchattacks PGD  acc @ eps=0.05 = 0.417

Finally the defense. Code 33.2.4 retrains the model with adversarial training, replacing each clean batch with its PGD-perturbed version inside the training loop, the saddle-point objective of subsection four, then re-measures robust accuracy at the same budget.

robust = TinyCNN()                               # fresh model, adversarially trained
opt = torch.optim.Adam(robust.parameters(), 1e-2)
for _ in range(400):
    x, y = make_batch(128)
    xadv = pgd(robust, x, y, eps=0.05, alpha=0.05/4, steps=7)  # inner max: attack each batch
    opt.zero_grad()
    F.cross_entropy(robust(xadv), y).backward()  # outer min: train on the worst case
    opt.step()

def both(m):
    clean = (m(xte).argmax(1) == yte).float().mean().item()
    adv = (m(pgd(m, xte, yte, 0.05, 0.05/4)).argmax(1) == yte).float().mean().item()
    return clean, adv

c0, a0 = both(model); c1, a1 = both(robust)
print("undefended:  clean=%.3f  PGD@0.05=%.3f" % (c0, a0))
print("adv-trained: clean=%.3f  PGD@0.05=%.3f" % (c1, a1))

undefended:  clean=0.991  PGD@0.05=0.418
adv-trained: clean=0.972  PGD@0.05=0.829

Read the four blocks as one experiment. Code 33.2.1 built a victim that is perfect on clean data. Code 33.2.2 broke it from scratch and traced exactly how fast it breaks as the budget grows. Code 33.2.3 showed that torchattacks computes the same attack in a fraction of the code. Code 33.2.4 hardened the model and recovered most of the lost accuracy under attack for a small clean-accuracy cost. The narrative of the whole section is in those numbers: a temporal classifier is fragile to perturbations far below the signal's own scale, the fragility is quantifiable as a budget curve, and adversarial training (plus the domain-specific checks of subsection four) buys it back at a price you can name in advance.

7. Mechanistic Interpretability for Temporal Transformers Advanced

Saliency maps and attention visualizations, the subjects of this section, answer the question "which input timesteps influenced this prediction most?" That is a correlational answer: it tells you what the model attended to, not what computation it performed to get there. Mechanistic interpretability asks the sharper question: what algorithm does each attention head implement? Not "head 3 attends to lag-7" but "head 3 computes a seasonal-spike detector that passes a signal only when the current value exceeds a rolling maximum by more than two standard deviations." Answering the algorithmic question requires tools beyond gradient attribution: activation patching, sparse autoencoders, and causal attention graphs. Applied to temporal Transformers, these tools reveal a surprisingly structured internal organization in which different heads specialise for different temporal components (trend, seasonality, residual) and pass signals to each other in a directed causal graph.

7.1 Activation Patching: From Correlation to Causation

The standard gradient saliency map computes $\partial \hat{y} / \partial x_t$ and treats large magnitude as evidence that timestep $t$ is important. But importance in this sense is correlational: a large gradient at $t$ means the output is sensitive to $x_t$ in the neighborhood of the current input, not that the model uses $x_t$ through any particular computational pathway. Activation patching makes the causal claim precise.

The setup requires two inputs: a clean input $\mathbf{x}^*$ on which the model gives the correct output, and a corrupted input $\tilde{\mathbf{x}}$ on which the model fails. For a seasonal anomaly detector, the clean input is a time series with a genuine seasonal spike and the corrupted input has that spike replaced by the seasonal mean (so the model should no longer flag an anomaly but does not see that it was patched). The patching experiment then proceeds layer by layer: run the corrupted input through the model, but at layer $\ell$ and head $h$, replace the activation with the activation computed on the clean input, and measure the logit difference $\Delta_{\ell,h} = \hat{y}^{\text{patched}} - \hat{y}^{\text{corrupted}}$. A large positive $\Delta_{\ell,h}$ means layer $\ell$, head $h$ is causally responsible for the correct detection: restoring its clean activation restores the correct output. Formally:

where $\mathbf{a}^*_{\ell,h}$ is the clean activation at layer $\ell$, head $h$, and the outer expression extracts the logit for the correct class $\hat{y}^*$. Running this experiment for every $(\ell, h)$ pair produces a heatmap of causal responsibility that is qualitatively different from the gradient heatmap: it tells you which computation, not which input, matters.

7.2 Sparse Autoencoders for the Residual Stream

A Transformer's residual stream, the vector that accumulates across layers, is a superposition of many simultaneously active features. Standard dimensionality-reduction techniques (PCA, t-SNE) compress this representation but do not yield interpretable features because real features are sparse in the residual basis, not in the principal-component basis. Sparse autoencoders (SAEs) learn a dictionary of $K \gg d$ features (typically $K \approx 100{,}000$ for $d = 512$) such that any residual-stream vector $\mathbf{r} \in \mathbb{R}^d$ is approximated as a sparse linear combination:

where the $\mathbf{e}_k$ are learned dictionary atoms and $\boldsymbol{\alpha}$ is a sparse coefficient vector produced by an encoder $\mathbf{z} = \text{ReLU}(W_\text{enc}\mathbf{r} + \mathbf{b}_\text{enc})$ trained with an $L_1$ sparsity penalty. For temporal models, training an SAE on the residual stream of a time-series Transformer and then manually inspecting the maximally-activating sequences for each dictionary atom reveals interpretable temporal features: one atom fires on seasonal spikes, another on trend reversals, a third on lag-7 autocorrelation signals, a fourth on missing-value boundaries. The dictionary provides a human-readable vocabulary for what the model's internal representations encode.

7.3 Causal Attention Graphs

Activation patching identifies which heads are causally important; combining patching results across all head pairs produces a directed causal attention graph. Define a directed edge from head $(\ell, h)$ to head $(\ell', h')$ if patching $(\ell, h)$'s output significantly changes the causal contribution $\Delta_{\ell',h'}$ of head $(\ell', h')$: i.e., head $(\ell, h)$'s computation is an input to head $(\ell', h')$'s computation. Formally, using path patching (an extension of activation patching that holds all other activations fixed while routing a single residual-stream path):

For time-series classifiers and anomaly detectors, arXiv 2511.21514 reports a consistent graph structure: a small number of early-layer heads detect specific temporal components (a seasonal detector, a trend detector, a residual-variance detector), and their outputs flow through a compact set of edges to late-layer heads that aggregate the components into a classification decision. The anomaly-decision heads do not directly attend to raw input; they receive pre-processed temporal-component signals from the detector heads. This is the algorithmic story that neither attention weights nor gradient saliency maps could tell.

7.4 Code: Activation Patching Loop in PyTorch

import torch
import torch.nn as nn
from copy import deepcopy

def activation_patching_experiment(model, x_clean, x_corrupt, target_class,
                                   num_layers, num_heads):
    """
    Run activation patching across all (layer, head) pairs.
    Returns delta[L, H]: causal importance of each head.
    model: a Transformer with model.blocks[l].attn as the attention module
    x_clean, x_corrupt: (B, T, C) tensors, B=1 for simplicity
    target_class: int, the correct class index
    """
    model.eval()
    delta = torch.zeros(num_layers, num_heads)

    # Cache clean activations for every layer and head
    clean_activations = {}

    hooks = []
    def make_cache_hook(layer_idx, head_idx):
        def hook(module, input, output):
            # output: (B, T, d_head) per head — split if needed
            clean_activations[(layer_idx, head_idx)] = output.detach().clone()
        return hook

    # Register forward hooks on each attention head output
    for l in range(num_layers):
        for h in range(num_heads):
            hook = model.blocks[l].attn.heads[h].register_forward_hook(
                make_cache_hook(l, h)
            )
            hooks.append(hook)

    with torch.no_grad():
        logits_clean = model(x_clean)
        baseline_score = logits_clean[0, target_class].item()

    for hook in hooks:
        hook.remove()

    # Corrupted baseline (no patching)
    with torch.no_grad():
        logits_corrupt = model(x_corrupt)
    corrupt_score = logits_corrupt[0, target_class].item()

    # Patch each (l, h) in turn: replace corrupted activation with clean one
    for l in range(num_layers):
        for h in range(num_heads):
            patched_model = deepcopy(model)
            clean_act = clean_activations[(l, h)]

            def make_patch_hook(act):
                def hook(module, input, output):
                    return act  # replace with clean activation
                return hook

            ph = patched_model.blocks[l].attn.heads[h].register_forward_hook(
                make_patch_hook(clean_act)
            )
            with torch.no_grad():
                logits_patched = patched_model(x_corrupt)
            ph.remove()

            patched_score = logits_patched[0, target_class].item()
            delta[l, h] = patched_score - corrupt_score

    return delta  # large positive = causally important head

# Example usage
# delta = activation_patching_experiment(model, x_clean, x_corrupt,
#                                        target_class=1, num_layers=6,
#                                        num_heads=8)
# print("Most causally important head:", delta.argmax().item())
# Layer, head = divmod(delta.argmax().item(), 8)

Mechanistic interpretability sits at the frontier of what is operationally possible today: the tools exist, the results are reproducible, and the causal graph structure they reveal has direct consequences for model editing, pruning, and robustness. A head identified as causally essential for seasonal-anomaly detection is a head that an adversary can target specifically; knowing the graph structure of your model's internal computation is therefore not only an interpretability result but a security input. That link between interpretability and robustness is the bridge to the next section, where we evaluate how fragile the large pretrained temporal foundation models, the very models that mechanistic interpretability is now being applied to, turn out to be under adversarial attack.

6. Exercises

Three exercises, one of each type, building directly on this section. Selected solutions appear in Appendix G.