Temporal Fusion Transformer (TFT)

"You handed me forty features and called every one essential. I gave eleven of them a weight near zero before the first epoch finished, kept the rest on a leash, and at no point did I make a fuss about it. Interpretability, as it turns out, is mostly the discipline of ignoring you politely."

A Variable Selection Network Quietly Ignoring Your Useless Features

In Section 14.1 we built DeepAR and the deep autoregressive family, models that roll a recurrent state forward one step at a time and sample a trajectory. They are probabilistic and they are strong, but they treat the covariate zoo uniformly and they are point-recursive: a forecast at horizon twenty is the twentieth link in a chain, each link compounding the last link's error. This section takes a different design stance. The Temporal Fusion Transformer asks: what if the architecture itself encoded the three categories of covariate that Chapter 2 taught us to distinguish, predicted all horizons jointly rather than recursively, and made its own reasoning legible? The result is a forecasting-specific Transformer that has become a strong, widely used baseline for multi-horizon problems with rich covariates. We use the unified notation of Appendix A throughout: $\mathbf{x}_t$ the time-varying inputs, $\mathbf{s}$ the static covariates, $\hat{y}_{t}(q)$ the forecast at horizon $t$ and quantile $q$.

The reason the TFT deserves a section rather than a paragraph is that it is the clearest worked example in this book of architecture-as-inductive-bias for forecasting. Every component answers a specific, nameable pain of real forecasting data. Variable selection answers "I have forty candidate features and most are noise". Gated residual networks answer "I do not know in advance whether this nonlinear transform helps, so let the model learn to skip it". Static encoders answer "the store's region matters but it does not change with time, so do not waste sequence capacity re-reading it". Interpretable attention answers "the forecast must justify itself to a human who will act on it". Quantile outputs answer "a number is not a forecast; a distribution is". Learn the TFT and you learn a vocabulary of forecasting-specific building blocks that recur, in pieces, across the rest of Part III.

The competencies this section installs are four: to name the three covariate types and trace each through its TFT pathway; to write and reason about the quantile (pinball) loss that produces multi-horizon probabilistic output; to implement a variable selection network and a gated residual network from scratch and see what they compute; and to run a production TFT and extract both its quantile forecasts and its variable-importance readout. These carry directly into the interpretability methods of Chapter 33 and the foundation models of Chapter 15.

1. The Temporal Fusion Transformer: A Forecasting-Specific Architecture Beginner

The TFT, introduced by Bryan Lim, Sercan Arik, Nicolas Loeff, and Tomas Pfister in 2021, is best understood as five cooperating ideas assembled in a deliberate order. From the bottom up: a variable selection network at each timestep decides which input variables are worth attending to, producing a weighted combination rather than a raw concatenation. A static covariate encoder compresses the time-invariant metadata into a handful of context vectors that are injected at several points downstream. An LSTM encoder-decoder (the locality-aware sequence backbone) processes the selected past inputs and the known-future inputs, giving each timestep a context-aware representation. An interpretable multi-head attention layer then lets every forecast horizon attend over the entire encoded history, capturing long-range patterns the LSTM alone would smear. Throughout, gated residual networks wrap the nonlinear transforms so the model can suppress any component that does not help. The top reads out quantiles at every horizon.

The single design commitment that distinguishes the TFT from a generic sequence-to-sequence Transformer is that it treats the three covariate types of Chapter 2 as architecturally distinct, not as rows in one matrix. Recall that chapter's taxonomy: static covariates $\mathbf{s}$ do not vary with time (a series identifier, a product category, a sensor's location); known-future covariates are time-varying but known ahead of the forecast (the day of week, a public holiday, a scheduled price change); and observed-past covariates are time-varying and known only up to the forecast origin (the target's own history, past traffic, past temperature). A point forecaster that ignores these distinctions will, for instance, try to predict tomorrow's holiday flag instead of simply reading it off the calendar, or will re-encode a store's region at every one of a thousand timesteps. The TFT routes each type where it belongs: known-future covariates feed both the encoder and the decoder, observed-past covariates feed only the encoder, and static covariates feed a dedicated encoder whose outputs condition everything. Figure 14.3.1 shows the routing.

Two consequences of this routing are worth making explicit because they are what a generic Transformer gets wrong. First, because known-future covariates feed the decoder, the TFT predicts all horizons jointly in a single forward pass rather than recursively, so a known holiday three weeks out informs the forecast for that day directly, without being filtered through twenty intervening predicted steps. This is the multi-horizon advantage over the autoregressive models of Section 14.1. Second, because the static metadata is encoded once and injected as context rather than tiled across the sequence, the sequence model spends its capacity on temporal dynamics rather than on relearning constant facts at every step. The architecture is, in a precise sense, the chapter-2 covariate taxonomy compiled into a network.

2. The Components: Variable Selection, Gating, Static Encoders, and Quantile Outputs Intermediate

We now open the four load-bearing components. The first is the variable selection network (VSN), which decides per timestep which input variables matter. Suppose at a given step there are $m$ candidate variables, each first passed through its own small transform to an embedding $\boldsymbol{\xi}^{(j)} \in \mathbb{R}^{d}$. The VSN computes a vector of selection weights by feeding the flattened embeddings (conditioned on the static context $\mathbf{c}_s$) through a small network and a softmax:

where $v_j$ is the (nonnegative, sum-to-one) weight given to variable $j$ at this step and $\tilde{\boldsymbol{\xi}}$ is the selected, fused representation passed onward. The weights $v_j$ are the interpretability gold of subsection three: averaged over a dataset, they rank variables by importance. The second component is the gated residual network (GRN), the universal nonlinear unit the VSN itself is built from. A GRN applies a nonlinearity but wraps it in a gate and a residual skip so the network can learn to pass its input through unchanged when the nonlinearity does not help:

where $\operatorname{GLU}(\boldsymbol{\eta}) = \sigma(\mathbf{W}_g \boldsymbol{\eta} + \mathbf{b}_g) \odot (\mathbf{W}_h \boldsymbol{\eta} + \mathbf{b}_h)$ is a gated linear unit whose sigmoid gate $\sigma(\cdot)$ can drive the whole branch to zero, leaving only the residual $\mathbf{a}$. This is the same skip-it-if-useless logic as the gating in the LSTM of Chapter 10, now applied to feedforward depth: the model decides, per unit, how much nonlinearity to admit. The third component, the static covariate encoder, runs the static inputs through their own GRNs to produce four distinct context vectors $\mathbf{c}_s, \mathbf{c}_e, \mathbf{c}_c, \mathbf{c}_h$: one conditions variable selection, one enriches the temporal features, and two initialize the LSTM's cell and hidden state, so the static metadata shapes the dynamics from the first step.

The fourth component is the output. The TFT does not predict a point; it predicts a set of quantiles at every horizon, making the forecast probabilistic. It is trained by minimizing the quantile loss, also called the pinball loss. For a target $y$, a prediction $\hat{y}(q)$ at quantile level $q \in (0,1)$, the per-observation loss is

and the total training loss sums this over a chosen set of quantiles $\mathcal{Q}$ (typically $\{0.1, 0.5, 0.9\}$), over all forecast horizons $\tau = 1, \dots, H$, and over the training set:

The asymmetry is the whole point: when $q = 0.9$ the loss penalizes under-prediction ($y > \hat{y}$) nine times as heavily as over-prediction, so the minimizer sits where ninety percent of the mass falls below it, which is exactly the ninetieth percentile. Minimizing $\mathcal{L}_q$ thus pins $\hat{y}(q)$ to the true conditional $q$-quantile, and predicting several $q$ at once traces out a predictive interval. We met this estimator in spirit in Chapter 5's prediction intervals and will formalize its coverage guarantees in Chapter 19. Figure 14.3.2 shows the pinball loss as a function of the residual for three quantile levels.

3. Interpretability: Reading the Attention and the Variable Importance Intermediate

The TFT was designed so that two of its internal quantities are directly readable as explanations, and this is the property that distinguishes it from most deep forecasters. The first readable quantity is the variable-selection weights $v_j$ from subsection two. Because they are a softmax over variables, computed at every step, averaging $v_j$ over a validation set yields a single importance score per variable: a number you can sort to answer "which inputs is the model actually using?". A feature whose averaged weight is near zero is one the model has chosen to ignore, exactly the behavior the section's epigraph boasts about. This is global feature importance read straight off the forward pass, with no perturbation study or surrogate model required.

The second readable quantity is the attention pattern. The TFT uses an interpretable multi-head attention, a deliberate modification of the standard mechanism of Chapter 12. In vanilla multi-head attention each head has its own value projection, so the heads attend to different things and their weights cannot be meaningfully averaged into one pattern. The TFT shares a single value projection across heads while keeping per-head queries and keys, so all heads operate on the same value space and their attention weights can be additively combined into one interpretable distribution over past timesteps. Reading that distribution for a given forecast tells you which moments in history the model leaned on: a sharp spike at lag $168$ on an hourly series is the model announcing that it found the weekly seasonality. These two readouts, variable importance and temporal attention, are the concrete handles that the interpretability methods of Chapter 33 formalize and compare against post-hoc explainers; the TFT is notable precisely because its explanations are intrinsic rather than bolted on afterward.

4. When the TFT Shines, and When a Simpler Model Wins Intermediate

The TFT is powerful and it is also heavy, so the practical question is when its machinery earns its keep. It shines under a recognizable conjunction of conditions. Rich, heterogeneous covariates: when you genuinely have static, known-future, and observed-past inputs and the known-future ones carry real signal (calendars, promotions, scheduled maintenance), the TFT's routing converts that structure into accuracy that a covariate-blind model leaves on the table. Multi-horizon forecasting: when you need a whole horizon of predictions at once and recursive error accumulation hurts, the TFT's joint decoder is a direct advantage over the autoregressive models of Section 14.1. A need for interpretability: when a human will act on the forecast and demand to know why, the variable-importance and attention readouts of subsection three are a built-in answer. Many related series: like DeepAR, the TFT is a global model trained across many series at once, so it borrows strength across them and forecasts cold-start series from their static metadata.

The mirror image is equally important, because reaching for the TFT reflexively is a common and costly mistake. When the data is a single short univariate series with no covariates, the TFT has nothing to select, nothing future to read, and no static metadata to encode; a classical model from Chapter 5 or a tuned exponential smoother will match or beat it at a thousandth of the cost and engineering. When the relationships are essentially linear, the gradient-boosted trees of a feature-engineered pipeline, or even a linear model, are stronger and far cheaper, a lesson the surprisingly strong linear baselines of Section 14.4 drive home. When you have very few series and little data, the TFT's parameter count invites overfitting that its gating only partly tames. And when latency or compute is tight, a model that must run an LSTM, a multi-head attention, and a stack of GRNs per forecast may simply be too slow. The honest decision rule: the TFT's complexity is justified exactly when your problem has the structure the TFT was built to exploit (heterogeneous covariates, multiple horizons, many series, an interpretability requirement), and not otherwise.

5. Worked Example: A Variable-Selection Block by Hand, Then a Full TFT by Library Advanced

We now make the components concrete. The plan mirrors the book's from-scratch-then-library discipline: first implement a gated residual network and a variable selection network in raw PyTorch so the selection mechanism and the gate are fully visible, run them on a small batch, and read off the per-variable weights; then run a complete production TFT (the same architecture, fully assembled with LSTM, attention, and quantile head) in a few lines of pytorch-forecasting, and extract both quantile forecasts and a variable-importance readout. Code 14.3.1 is the from-scratch GRN and VSN.

import torch
import torch.nn as nn
import torch.nn.functional as F

class GatedResidualNetwork(nn.Module):
    """A GRN: nonlinearity wrapped in a GLU gate and a residual skip (subsection 2)."""
    def __init__(self, d):
        super().__init__()
        self.fc1 = nn.Linear(d, d)          # W1 a + b1
        self.fc2 = nn.Linear(d, d)          # W2 ELU(.) + b2  -> eta
        self.gate = nn.Linear(d, 2 * d)     # GLU: produces value and sigmoid gate
        self.norm = nn.LayerNorm(d)

    def forward(self, a):
        eta = self.fc2(F.elu(self.fc1(a)))  # nonlinear branch
        val, gate = self.gate(eta).chunk(2, dim=-1)
        glu = torch.sigmoid(gate) * val     # gate can drive this branch to zero
        return self.norm(a + glu)           # residual skip: identity if gate closes

class VariableSelectionNetwork(nn.Module):
    """Per-step variable selection: a softmax over m variables, each GRN-transformed."""
    def __init__(self, m, d):
        super().__init__()
        self.m, self.d = m, d
        self.flat_grn = GatedResidualNetwork(m * d)     # produces selection logits
        self.to_weights = nn.Linear(m * d, m)
        self.var_grns = nn.ModuleList([GatedResidualNetwork(d) for _ in range(m)])

    def forward(self, x):                   # x: (batch, m, d), m variable embeddings
        b = x.shape[0]
        flat = x.reshape(b, self.m * self.d)
        weights = F.softmax(self.to_weights(self.flat_grn(flat)), dim=-1)  # (b, m)
        transformed = torch.stack(
            [self.var_grns[j](x[:, j, :]) for j in range(self.m)], dim=1)  # (b, m, d)
        fused = (weights.unsqueeze(-1) * transformed).sum(dim=1)           # (b, d)
        return fused, weights

torch.manual_seed(0)
m, d, batch = 5, 8, 4                        # 5 candidate variables, width 8
vsn = VariableSelectionNetwork(m, d)
x = torch.randn(batch, m, d)
# Make variable 2 carry a strong signal and variable 4 pure noise, by scaling.
x[:, 2, :] *= 4.0; x[:, 4, :] *= 0.05
fused, weights = vsn(x)
print("fused shape   :", tuple(fused.shape))
print("avg selection weights:", weights.mean(0).round(decimals=3).tolist())

fused shape   : (4, 8)
avg selection weights: [0.197, 0.183, 0.262, 0.191, 0.167]

The from-scratch block shows the mechanism but not the full model: assembling the LSTM encoder-decoder, the interpretable attention, the static encoders, and the quantile head by hand is hundreds of lines. This is where the library pays off. Code 14.3.2 builds and trains a complete TFT on a multi-covariate dataset with pytorch-forecasting, whose TimeSeriesDataSet handles the static / known-future / observed-past routing of subsection one declaratively.

import lightning.pytorch as pl
from pytorch_forecasting import TimeSeriesDataSet, TemporalFusionTransformer
from pytorch_forecasting.data import GroupNormalizer
from pytorch_forecasting.metrics import QuantileLoss

# df has columns: series_id, time_idx, target, static_cat, day_of_week (known future), ...
training = TimeSeriesDataSet(
    df[df.time_idx <= cutoff], time_idx="time_idx", target="target",
    group_ids=["series_id"], max_encoder_length=168, max_prediction_length=24,
    static_categoricals=["static_cat"],                 # static covariates
    time_varying_known_reals=["day_of_week", "promo"],  # known-future covariates
    time_varying_unknown_reals=["target"],              # observed-past covariates
    target_normalizer=GroupNormalizer(groups=["series_id"]))

train_loader = training.to_dataloader(train=True, batch_size=128)
tft = TemporalFusionTransformer.from_dataset(
    training, hidden_size=32, attention_head_size=4,
    loss=QuantileLoss([0.1, 0.5, 0.9]))                 # multi-horizon quantiles
pl.Trainer(max_epochs=10, gradient_clip_val=0.1).fit(tft, train_loader)

The library equivalent collapses an architecture that would run roughly 600 lines to implement by hand, the GRNs, the VSNs, the static encoders, the LSTM encoder-decoder, the interpretable attention, and the quantile head with masking and normalization, into about a dozen lines, and it handles the covariate routing, the group-wise normalization, and the variable-length batching internally. What we lose by hand we gain back as a one-call model; what makes the library version trustworthy is that its VSN and GRN are exactly the blocks we built in Code 14.3.1. Code 14.3.3 reads the two interpretability artifacts of subsection three off the trained model: the quantile forecasts and the variable-importance ranking.

# Quantile forecasts: prediction at every horizon for [0.1, 0.5, 0.9].
preds = tft.predict(val_loader, mode="quantiles")    # (n_series, horizon, 3)
median = preds[..., 1]                                # the 0.5 (point) forecast
lower, upper = preds[..., 0], preds[..., 2]           # 80% predictive interval
print("forecast horizon shape :", tuple(preds.shape))
print("first series, h=1  10/50/90:", preds[0, 0].round(decimals=2).tolist())

# Variable importance, read straight off the variable selection weights.
raw = tft.predict(val_loader, mode="raw", return_x=True)
interp = tft.interpret_output(raw.output, reduction="sum")
enc_imp = interp["encoder_variables"]                 # importance of observed-past inputs
print("encoder variable importance:", enc_imp.round(decimals=3).tolist())

forecast horizon shape : (64, 24, 3)
first series, h=1  10/50/90: [11.42, 13.08, 15.10]
encoder variable importance: [0.71, 0.18, 0.11]

Read the three code blocks together. Code 14.3.1 implemented the gate and the selection softmax by hand, so the mechanism is no longer a black box: a GRN is a gated skip connection and a VSN is a softmax-weighted sum of GRN-transformed variables. Code 14.3.2 assembled the entire TFT, that same VSN and GRN plus the sequence and attention machinery, in a dozen library lines with the covariate routing declared rather than wired. Code 14.3.3 read off exactly the two artifacts subsection three promised: a multi-horizon quantile forecast and a variable-importance ranking. The arc from hand-built block to one-call model to readable explanation is the TFT in miniature.

from neuralforecast import NeuralForecast
from neuralforecast.models import TFT
from neuralforecast.losses.pytorch import MQLoss

nf = NeuralForecast(
    models=[TFT(h=24, input_size=168, loss=MQLoss(quantiles=[0.1, 0.5, 0.9]),
                futr_exog_list=["day_of_week", "promo"],   # known-future covariates
                stat_exog_list=["static_cat"], max_steps=300)],
    freq="D")
nf.fit(df)                                  # df in long format: unique_id, ds, y
# future_df: known-future covariates (day_of_week, promo) for the next h steps
forecasts = nf.predict(futr_df=future_df)   # multi-horizon quantile forecasts

Exercises

1. (Conceptual.) A retailer's dataset has these features: store latitude, product price scheduled by the merchandising calendar, past daily revenue, store square footage, an upcoming public holiday flag, and yesterday's foot traffic. Classify each into static, known-future, or observed-past, and state which TFT pathway (static encoder, encoder-only, or both encoder and decoder) each one feeds. Explain why feeding past foot traffic to the decoder would be a leakage bug.
2. (Implementation.) Extend the variable selection network of Code 14.3.1 to log its average selection weights every epoch while training on a synthetic target $y = 3 x^{(2)} + \varepsilon$ where only variable 2 is informative and the rest are noise. Plot the weight on variable 2 versus epoch and confirm it rises toward dominance. Then add a redundant copy of variable 2 as a sixth variable and observe how the VSN splits its weight between the two correlated copies; discuss what this implies for reading importance when features are correlated.
3. (Open-ended.) Take a multi-covariate forecasting dataset of your choice and train both a TFT (Code 14.3.2 or 14.3.4) and a simple linear or DLinear baseline of Section 14.4. Compare their multi-horizon quantile accuracy (mean pinball loss) and their wall-clock training cost. Identify the conditions in your data (covariate richness, series count, horizon length) under which the TFT's added complexity is, and is not, justified, and write the decision rule you would give a teammate.