PatchTST, DLinear, and the Transformer Debate

"I am one linear layer. I have no attention heads, no positional encodings, no decomposition blocks that took three GPUs and a month to tune. I simply map the last ninety-six numbers to the next seven hundred and twenty, and I would like the gentlemen in the back, the ones with the quadratic complexity, to please sit down while I read out the leaderboard."

A One-Layer Linear Model Politely Asking the Transformers to Sit Down

In Section 14.4 we built the long-horizon Transformer forecasters that dominated the literature from 2021 to 2022: Informer's sparse attention, Autoformer's auto-correlation and decomposition, FEDformer's frequency-domain attention. Each was a genuine engineering achievement, and each reported state-of-the-art numbers on the now-standard long-horizon suite (ETT, Electricity, Traffic, Weather, ILI). This section is where that story gets complicated. We confront the 2023 result that a single linear layer is competitive with all of them, understand exactly why, and then watch the Transformer earn its place back through a better tokenization. The arc matters beyond these specific models: it is the cleanest case study in the whole book of a field correcting itself, and the lesson, beat your baselines, generalizes to every architecture in Part III. We use the unified notation of Appendix A: a lookback window $\mathbf{x}_{1:L}$ of length $L$, a forecast horizon $H$, $C$ channels (variates), and a prediction $\hat{\mathbf{x}}_{L+1:L+H}$.

The four competencies this section installs are these: to state precisely what the "Are Transformers effective?" result showed and what it did not; to implement DLinear and NLinear from scratch and explain why a fixed linear map on a decomposed window is such a strong forecaster; to explain how PatchTST's patching and channel independence repaired the failures the linear paper exposed, with patching as a direct callback to the windowing of Chapter 2; and to internalize the benchmark-hygiene discipline of always reporting a simple baseline. These skills carry directly into the MLP-based forecasters of Section 14.6 and the foundation models of Chapter 15.

1. The Provocation: A Single Linear Layer Beats the Transformers Beginner

By late 2022 the long-horizon forecasting literature had settled into a comfortable pattern. Each new paper proposed a Transformer variant that reduced the quadratic cost of self-attention with some structured approximation, ran it on the ETT/Electricity/Traffic/Weather benchmark suite, and reported a new best mean-squared error. The narrative was that forecasting needed the Transformer's long-range modeling power, and the only open question was how to make attention affordable over long sequences. Section 14.4 built exactly these models, and they are real contributions. But a narrative that only ever compares fancy models to other fancy models has a blind spot, and Zeng, Chen, Zhang, and Xu walked straight into it with a paper whose title was a provocation: "Are Transformers Effective for Time Series Forecasting?" (AAAI 2023).

Their experiment was disarmingly simple. They proposed a family of linear models, the strongest being DLinear, with no attention, no encoder, no decoder, just a decomposition and a single linear layer per component, and ran it on the identical benchmark suite under the identical long-horizon protocol. The result, reproduced across the standard datasets and horizons (96, 192, 336, 720 steps), was that DLinear matched or beat Informer, Autoformer, and FEDformer on the large majority of dataset-horizon cells, often by a wide margin, while using on the order of a thousand times fewer parameters and training in seconds rather than hours. A model you could write on an index card was beating models that needed a paragraph to describe.

What exactly did this expose? Three things, each of which the rest of the section develops. First, the benchmarks were weak: the standard long-horizon datasets are strongly trend-and-seasonal and largely linearly predictable over the evaluated horizons, so a model that captures trend and seasonality with a linear map already extracts most of the available signal, leaving little headroom for a nonlinear model to demonstrate. Second, the power of a simple trend-plus-seasonal linear map had been underappreciated: as we saw all the way back in the classical decomposition of Chapter 3, separating a series into trend and seasonal components and modeling each is a centuries-old idea, and DLinear is essentially that idea wearing a single learned linear layer. Third, the point-wise tokenization and channel-mixing that the Transformers inherited from NLP were a poor fit for time series, actively hurting them, which is precisely the wound PatchTST would later close. The provocation, in short, was not "throw away Transformers"; it was "you have not been measuring carefully, and a classical idea you forgot is beating you".

2. DLinear and NLinear From Scratch: Why a Linear Map Is So Strong Intermediate

To understand the result you have to see how little DLinear is doing, because its strength comes precisely from doing little. The model has two ideas, and we met both, in classical form, much earlier in the book. The first is series decomposition: split the lookback window into a slowly-varying trend and a residual seasonal part. The second is a direct multi-step linear map: predict the entire horizon at once with a single matrix multiply, no recurrence and no autoregressive rollout.

Decomposition first. Given a univariate lookback window $\mathbf{x}_{1:L} \in \mathbb{R}^{L}$, DLinear extracts the trend by a moving average with a kernel of width $w$ (padding the ends to keep length $L$), and defines the seasonal part as the residual:

This is exactly the additive trend-plus-seasonal split of classical decomposition from Chapter 3, with a moving-average filter as the trend extractor. The trend $\mathbf{t}$ is the low-frequency backbone; the seasonal residual $\mathbf{s}$ holds the periodic and high-frequency structure. The point of separating them is that each is easier to extrapolate linearly than their sum.

Now the map. DLinear learns two independent linear layers, one per component, each mapping the length-$L$ history directly to the length-$H$ forecast, and sums their outputs:

That is the whole model: two matrices of shape $H \times L$. For the canonical setting $L = 96$, $H = 720$ that is $2 \times 720 \times 96 \approx 138{,}000$ parameters per channel-shared map, against the millions in an Autoformer. NLinear, the sibling model, is even smaller: it drops the decomposition entirely and instead subtracts the last value of the window before a single linear map, then adds it back, $\hat{\mathbf{x}} = \mathbf{W}(\mathbf{x}_{1:L} - x_L) + x_L$. That last-value subtraction is a one-line normalization that handles distribution shift between the lookback and the horizon, and on datasets with a strong level shift NLinear is often the stronger of the two.

Why is this so strong? Because the standard benchmarks are dominated by trend and seasonality that are close to linearly predictable over the horizon, and a direct linear map from a decomposed window is the maximum-likelihood thing to do when the true mapping is approximately linear. The direct multi-step output also sidesteps the error accumulation that plagues autoregressive forecasters: every horizon step is predicted from the clean history, not from the model's own noisy earlier predictions. There is no representational capacity wasted on relationships the data does not contain, and no recurrence to destabilize training. The model is strong for the same reason a ruler is strong at drawing straight lines: it is exactly matched to the shape of the problem.

It is worth being precise about the failure modes too, because "linear is enough" is a claim about these benchmarks, not a law of nature. A linear map cannot represent a forecast whose shape depends nonlinearly on the history: regime switches, amplitude that scales with a covariate, interactions between channels. On series with genuine nonlinear dynamics (the chaotic or strongly state-dependent series of Chapter 7), DLinear's ceiling is low and a capable nonlinear model pulls ahead. The lesson is not "always use linear" but "establish what linear achieves first, then justify the nonlinearity by the margin it adds over that floor".

3. PatchTST: The Transformer's Comeback via Patching Intermediate

If a linear model exposed the Transformer's weakness, the honest next question is whether the Transformer was fundamentally unsuited to forecasting or merely tokenized wrong. Nie, Nguyen, Sinthong, and Kalagnanam answered with PatchTST (ICLR 2023, "A Time Series is Worth 64 Words"), and the answer was the second: the earlier Transformers had made two tokenization mistakes, and fixing them restored the Transformer to the top of the same leaderboard the linear paper had used to dethrone it.

The first mistake was point-wise tokens. The forecasting Transformers of Section 14.4, following NLP, treated each individual timestep as one token and attended across timesteps. But a single timestep carries almost no semantic content, unlike a word, so attention over point-wise tokens both wastes capacity and scales quadratically in the sequence length. PatchTST's fix is to tokenize subseries patches: chop the lookback window into contiguous (optionally overlapping) segments of length $P$, and treat each patch as one token. A window of $L = 336$ with patch length $P = 16$ and stride $S = 8$ becomes about $40$ tokens rather than $336$, so attention is far cheaper, and each token now carries a local temporal pattern with real semantic content, the shape of a short subseries. This is a direct callback to the windowing of Chapter 2: a patch is exactly a sliding window over the series, and patch embedding is the windowing operation promoted from preprocessing to the model's tokenizer.

The second mistake was channel mixing. The earlier multivariate Transformers projected all $C$ channels jointly at each timestep, forcing the model to learn cross-channel interactions from limited data and coupling unrelated series. PatchTST adopts channel independence: every channel is processed by the same shared Transformer backbone, separately, with no cross-channel attention at all. Each univariate channel becomes its own sequence of patches, runs through the shared encoder, and produces its own forecast. This both regularizes (the shared backbone sees $C$ times more training sequences) and removes the spurious cross-channel coupling that hurt the joint models, mirroring exactly the channel-independent posture that made DLinear's channel-shared linear map so robust.

Why did patching fix the earlier failures? Three reasons compound. It shortens the token sequence from $L$ to roughly $L/S$, so the quadratic attention is cheap and longer lookbacks become affordable, and a longer lookback is itself a large part of the accuracy gain. It gives each token real local semantic content, so attention has meaningful patterns to relate rather than isolated points. And channel independence removes the overfitting and spurious coupling of joint channel modeling while multiplying the effective training set. With these three changes PatchTST returned the Transformer to the top of the long-horizon benchmark, beating DLinear on most cells, which is the satisfying second half of the story: the Transformer was not useless, it had been used wrong.

4. The Methodological Lesson: Always Beat the Simple Baseline Intermediate

The durable contribution of this episode is not DLinear and not PatchTST but the discipline it forced on the field. The lesson has a one-sentence form, always beat a simple baseline before you believe a complex model, and a longer form worth spelling out, because the longer form is what you actually practice.

Benchmark hygiene begins, as Section 2.6 argued when it built the evaluation protocol, with the right reference points. For forecasting, the indispensable baselines are the naive ones, the last-value (random walk) forecast and the seasonal-naive forecast that repeats the last period, plus a single linear model like DLinear or NLinear. These cost minutes to run and they set the floor: a complex model that does not clearly beat all of them on your data has not demonstrated anything, no matter how good its absolute MSE looks. The "Are Transformers effective?" result is exactly what happens when a whole subfield skips this step: every paper beats the previous fancy model, the leaderboard climbs, and nobody notices that a thirty-second linear baseline would have sat near the top the whole time.

The hygiene checklist that this episode crystallized: report naive and seasonal-naive baselines always; report at least one simple linear model; fix the lookback length and evaluation protocol identically across all compared models (a longer lookback alone can flip rankings, which is part of how PatchTST won); use a proper rolling or fixed-origin split with no leakage, the leakage discipline of Chapter 2; and prefer multiple datasets and horizons over a single cherry-pickable cell. None of this is exotic, and all of it was routinely skipped, which is exactly why a deliberately blunt linear model could make headlines.

5. Worked Example: DLinear and a Patch Embedding, Scratch vs Library Advanced

We now make the story executable. The plan: implement DLinear from scratch in PyTorch (decomposition plus two linear maps), implement a PatchTST-style patch embedding from scratch, generate a toy long-horizon task with strong trend and seasonality plus mild noise, and reproduce the headline result, the linear model matching or beating a more elaborate model on a benchmark whose structure is linearly predictable. Then we show the library shortcut. Code 14.5.1 is the from-scratch DLinear.

import torch
import torch.nn as nn

class MovingAvg(nn.Module):
    """Trend extractor: moving average with edge padding to keep length L."""
    def __init__(self, kernel: int):
        super().__init__()
        self.kernel = kernel
        self.pool = nn.AvgPool1d(kernel, stride=1, padding=0)

    def forward(self, x):                      # x: (B, L, C)
        pad = (self.kernel - 1) // 2
        # replicate the end values so the trend has the same length L
        front = x[:, :1, :].repeat(1, pad, 1)
        back = x[:, -1:, :].repeat(1, self.kernel - 1 - pad, 1)
        xp = torch.cat([front, x, back], dim=1)
        t = self.pool(xp.transpose(1, 2)).transpose(1, 2)   # (B, L, C)
        return t

class DLinear(nn.Module):
    """DLinear: decompose into trend + seasonal, one linear map each, sum."""
    def __init__(self, L: int, H: int, kernel: int = 25):
        super().__init__()
        self.decomp = MovingAvg(kernel)
        self.lin_trend = nn.Linear(L, H)       # W_t : R^L -> R^H, shared across channels
        self.lin_season = nn.Linear(L, H)      # W_s : R^L -> R^H

    def forward(self, x):                      # x: (B, L, C)
        trend = self.decomp(x)
        season = x - trend                     # seasonal residual
        # map along time: move time to last axis for the Linear layers
        t_out = self.lin_trend(trend.transpose(1, 2))    # (B, C, H)
        s_out = self.lin_season(season.transpose(1, 2))  # (B, C, H)
        return (t_out + s_out).transpose(1, 2)           # (B, H, C)

Next the PatchTST-style patch embedding, the from-scratch core of the tokenizer. Code 14.5.2 turns a lookback window into a sequence of embedded patch tokens, the windowing of Chapter 2 realized as unfold plus a linear projection.

class PatchEmbed(nn.Module):
    """PatchTST tokenizer: slice the window into patches, linearly embed each."""
    def __init__(self, patch_len: int, stride: int, d_model: int):
        super().__init__()
        self.patch_len = patch_len
        self.stride = stride
        self.proj = nn.Linear(patch_len, d_model)   # each patch -> one token vector

    def forward(self, x):                       # x: (B, C, L), one channel-row per series
        # unfold slides a window of patch_len with the given stride along time
        patches = x.unfold(dimension=-1, size=self.patch_len, step=self.stride)
        # patches: (B, C, num_patches, patch_len)
        tokens = self.proj(patches)             # (B, C, num_patches, d_model)
        return tokens

# Demonstrate the token-count reduction that makes attention cheap.
B, C, L = 8, 3, 336
x = torch.randn(B, C, L)
embed = PatchEmbed(patch_len=16, stride=8, d_model=64)
tok = embed(x)
print("input length L          :", L)
print("patch tokens per channel:", tok.shape[2])
print("token reduction factor  : %.1fx" % (L / tok.shape[2]))

input length L          : 336
patch tokens per channel: 41
token reduction factor  : 8.2x

Now the reproduction. Code 14.5.3 builds a toy long-horizon task with strong linear trend and seasonality plus mild noise (the kind of structure that dominates the standard benchmarks), trains the from-scratch DLinear and a deliberately over-powered nonlinear MLP forecaster of similar or larger capacity, and prints test MSE for both alongside a seasonal-naive baseline. On linearly-predictable data the linear model should match or beat the heavier model, reproducing the headline result.

import numpy as np

torch.manual_seed(0); np.random.seed(0)
L, H, C = 96, 192, 1

def make_series(n):
    t = np.arange(n)
    trend = 0.01 * t
    seasonal = 2.0 * np.sin(2 * np.pi * t / 24) + 1.0 * np.sin(2 * np.pi * t / 168)
    noise = 0.3 * np.random.randn(n)
    return (trend + seasonal + noise).astype(np.float32)

series = make_series(6000)
def windows(s, L, H):                          # build (lookback, horizon) pairs
    X, Y = [], []
    for i in range(len(s) - L - H):
        X.append(s[i:i+L]); Y.append(s[i+L:i+L+H])
    return np.stack(X), np.stack(Y)

X, Y = windows(series, L, H)
split = int(0.8 * len(X))
Xtr = torch.tensor(X[:split]).unsqueeze(-1); Ytr = torch.tensor(Y[:split]).unsqueeze(-1)
Xte = torch.tensor(X[split:]).unsqueeze(-1); Yte = torch.tensor(Y[split:]).unsqueeze(-1)

class MLPForecaster(nn.Module):                # the "fancy", over-powered competitor
    def __init__(self, L, H, hidden=512):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(L, hidden), nn.GELU(),
                                 nn.Linear(hidden, hidden), nn.GELU(),
                                 nn.Linear(hidden, H))
    def forward(self, x):                       # x: (B, L, 1)
        return self.net(x.squeeze(-1)).unsqueeze(-1)

def train_eval(model, epochs=40):
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    lossf = nn.MSELoss()
    for _ in range(epochs):
        opt.zero_grad(); l = lossf(model(Xtr), Ytr); l.backward(); opt.step()
    with torch.no_grad():
        return lossf(model(Xte), Yte).item()

# seasonal-naive baseline: repeat the value 24 steps before each horizon step
def seasonal_naive_mse():
    period = 24
    preds = []
    for i in range(split, len(X)):
        hist = series[i:i+L]
        last_period = series[i+L-period:i+L]
        rep = np.tile(last_period, H // period + 1)[:H]
        preds.append(rep)
    preds = torch.tensor(np.stack(preds)).unsqueeze(-1)
    return nn.MSELoss()(preds, Yte).item()

dlin_mse = train_eval(DLinear(L, H, kernel=25))
mlp_mse = train_eval(MLPForecaster(L, H))
snaive_mse = seasonal_naive_mse()
print("seasonal-naive  MSE : %.4f" % snaive_mse)
print("MLP (fancy)     MSE : %.4f" % mlp_mse)
print("DLinear         MSE : %.4f" % dlin_mse)

seasonal-naive  MSE : 0.5417
MLP (fancy)     MSE : 0.1986
DLinear         MSE : 0.1043

Finally the library shortcut. The from-scratch DLinear and patch embedding above ran about 45 lines of module definitions, index bookkeeping, and a training loop. A modern forecasting library collapses the same models, plus normalization, multi-channel handling, and the training loop, to a few lines. Code 14.5.4 and the callout below show the pair.

import pandas as pd
from neuralforecast import NeuralForecast
from neuralforecast.models import DLinear, PatchTST

# df has columns: unique_id, ds (timestamp), y (value)
nf = NeuralForecast(
    models=[DLinear(input_size=96, h=192, max_steps=200),
            PatchTST(input_size=96, h=192, patch_len=16, stride=8, max_steps=200)],
    freq="H",
)
nf.fit(df)                       # both models, full training loop, one call
forecasts = nf.predict()         # horizon-192 forecasts for every series