Informer, Autoformer, and FEDformer

"You handed me a year of hourly demand and asked for next month. Before you had finished your coffee I had peeled the trend off the top like a label, sorted the seasonal wobble into its Fourier bins, attended to the periods that mattered, and ignored the nine thousand timesteps that did not. The dot-product crowd is still computing its first attention matrix."

An Autoformer Decomposing Your Series Before Breakfast

The vanilla Transformer of Chapter 12 has two structural weaknesses for long-sequence forecasting: quadratic attention cost and a permutation-invariant operator that has to relearn, from scratch, the trend and seasonality that classical methods read off directly. Section 14.3's Temporal Fusion Transformer adapted attention to forecasting but kept full quadratic self-attention; this section follows the research line that fixed both. We move through three architectures in the order they appeared, because each is a response to the one before: Informer (Zhou et al., 2021), Autoformer (Wu et al., 2021), and FEDformer (Zhou et al., 2022). They share a benchmark suite (the ETT, Electricity, Traffic, Weather, and Exchange datasets of Appendix E), a problem setting (predict hundreds to thousands of steps ahead from a long lookback), and a design philosophy that grows more explicit at each step. We use the unified notation of Appendix A: $\mathbf{x}_{1:L}$ the input series of length $L$, $\mathbf{Q}, \mathbf{K}, \mathbf{V}$ the query, key, and value matrices of attention, $\mathcal{F}$ the discrete Fourier transform.

The four competencies this section installs are these: to state what each of the three architectures changed relative to the vanilla Transformer and why, with the cost reduction made precise; to write the Auto-Correlation mechanism and frequency-enhanced attention as equations and explain how each reads classical time-series structure; to see, as a single recurring design move, the injection of decomposition and frequency into a learned model (the temporal thread of this book); and to hold the benchmark caveat that Section 14.5 develops, so that you adopt these models for the right reasons rather than their leaderboard numbers alone.

1. The LSTF Line: What Each Variant Fixed Intermediate

Long-sequence time-series forecasting (LSTF) names the regime these architectures were built for: a long lookback window, often hundreds to a few thousand steps, and a long prediction horizon of comparable length. The vanilla Transformer applied directly to this regime hits a wall the Chapter 12 self-attention construction makes plain. Self-attention forms an $L \times L$ score matrix, so both compute and memory grow as $O(L^2)$; at $L = 1000$ that is a million entries per head per layer, and the memory, not the arithmetic, is usually what fails first. Worse, the attention operator treats the input as an unordered set softened only by positional encodings, so it must learn the trend and the daily and weekly periods that a classical decomposition (Chapter 4) hands you for free. The three architectures of this section are three escalating answers.

Informer answered the cost. Its authors observed empirically that the attention score distribution is long-tailed: for most queries the softmax over keys is nearly uniform (it contributes little), and only a few "active" queries have a sharply peaked distribution that actually moves the output. Their ProbSparse attention measures how far each query's attention distribution is from uniform, using a sparsity score, and lets only the top $u = c \log L$ queries attend fully, approximating the rest by a simple average of values. This drops the dominant cost term from $O(L^2)$ to $O(L \log L)$. On top of this, Informer stacks a self-attention distilling operation between encoder layers: a stride-2 convolution plus max-pool that halves the sequence length at each layer, so deeper layers operate on progressively shorter sequences and the cumulative memory is bounded. The query sparsity score that ranks queries is

the gap between a query's largest score and its mean score; a large gap means a peaked, informative distribution worth computing exactly. Informer keeps the dot-product attention mechanism but makes it sparse and shrinks the sequence as it goes.

Autoformer answered the blindness. Rather than sparsify dot-product attention, it discarded it. Autoformer is built from series-decomposition blocks that, at multiple points inside the network, split the running representation into a slowly varying trend component (a moving average) and a seasonal residual, so the model manipulates the two components separately exactly as a classical decomposition would. In place of attention it introduced the Auto-Correlation mechanism, which finds the lags at which the series is most correlated with a shifted copy of itself (the autocorrelation function of Chapter 3) and aggregates whole sub-series at those lags. This is period-based aggregation rather than point-based attention, and it is both $O(L \log L)$, via the FFT, and aligned with how periodic series actually carry information. Subsection two derives it.

FEDformer pushed the instinct fully into the frequency domain, the domain of Chapter 4. Its frequency-enhanced attention transforms queries, keys, and values into the Fourier (or wavelet) domain, keeps only a small randomly selected set of $M$ spectral modes, performs the attention or a learned linear mixing in that compact basis, and transforms back. Because a time series is typically sparse in frequency (a handful of dominant periods carry most of the energy), random mode selection loses little and the cost becomes linear in $L$. FEDformer also keeps Autoformer's decomposition blocks and adds a mixture-of-experts decomposition that averages several moving-average kernels of different widths, so the trend extraction is no longer tied to a single window. Subsection two gives the frequency-attention equation. Figure 14.4.1 lines up the three against the vanilla baseline.

Reading the figure as a trajectory rather than three isolated models is the point of this subsection. The cost column falls monotonically because each architecture exploits a sharper notion of where the information in a long series actually lives: Informer says it lives in a few queries, Autoformer says it lives at a few lags, FEDformer says it lives at a few frequencies. The classical-idea column rises correspondingly, and that is not a coincidence: every reduction in cost came from importing a structural assumption about time series that the vanilla Transformer refused to make. The two columns move together because they are the same move.

2. Auto-Correlation and Frequency Attention: The Mechanics Advanced

We now write the two operators that replace dot-product attention, because the equations are where the injected structure becomes precise. Start with Auto-Correlation. Dot-product attention scores every pair of timesteps; Auto-Correlation instead scores every lag. For a single series $\{x_t\}$ the autocorrelation at lag $\tau$ is the correlation of the series with a copy of itself shifted by $\tau$:

which, by the Wiener-Khinchin theorem of Chapter 4, is the inverse Fourier transform of the power spectrum, so it can be computed for all lags at once in $O(L \log L)$ via the FFT rather than $O(L^2)$ by direct summation:

where $\overline{(\cdot)}$ is the complex conjugate. Auto-Correlation uses $R$ between the query and key series to find the $k$ most-correlated lags $\tau_1, \dots, \tau_k$ (the top autocorrelation peaks, which for a periodic series are its dominant periods), softmax-normalizes their correlation strengths into weights, and aggregates the value series rolled by each lag:

where $\text{Roll}(\mathbf{V}, \tau)$ shifts the value series by $\tau$ with the elements pushed off the end wrapped around to the front. The contrast with attention is exact and worth stating: attention aggregates points (a weighted sum over individual timesteps), Auto-Correlation aggregates sub-series (a weighted sum over whole shifted copies of the signal). For data whose information lives in its periods, sub-series aggregation matches the data-generating structure and point aggregation does not.

Now frequency-enhanced attention, FEDformer's operator. The idea is to do the mixing in the Fourier basis and to keep only a few modes. Project the input to queries, keys, and values, apply the discrete Fourier transform along the time axis, and select a random subset $S$ of $M$ frequency modes (random selection avoids over-committing to the low frequencies and is cheap):

each now an $M \times d$ complex matrix with $M \ll L$. The frequency-enhanced block mixes these compact spectral representations with a learned complex weight $\mathbf{R} \in \mathbb{C}^{M \times d \times d}$ (the "frequency attention", a per-mode linear map), then scatters the result back into a full-length spectrum (zeros at the unselected modes) and inverts the transform:

where $\tilde{\mathbf{K}}^{*}$ is the conjugate transpose, $\sigma$ a complex activation, and $\text{Pad}_S$ writes the $M$ computed modes back into their slots in a length-$L$ spectrum that is otherwise zero. Because all the work happens on $M$ modes rather than $L$ timesteps, and $M$ is a fixed small constant, the cost is $O(L)$ (the FFT is $O(L\log L)$ but dominated in practice by the linear data movement). FEDformer offers a wavelet variant that replaces $\mathcal{F}$ with a discrete wavelet transform for signals whose structure is localized in time as well as frequency. Both Auto-Correlation and frequency attention sit inside the same encoder-decoder skeleton as a Transformer, with the decomposition blocks of subsection three interleaved.

3. The Recurring Design Idea: Classical Structure Inside the Transformer Intermediate

Step back from the three architectures and the single design move they share comes into focus. Each one takes a piece of classical time-series machinery, something a forecaster of 1970 would recognize, and rebuilds it as a differentiable component inside a Transformer. Autoformer's series-decomposition block is the classical additive decomposition $x_t = \text{trend}_t + \text{seasonal}_t + \text{residual}_t$, with the trend extracted by a moving average, dropped bodily into the forward pass and made to operate on hidden representations instead of raw data. Autoformer's Auto-Correlation is the autocorrelation function. FEDformer's frequency attention is spectral analysis. None of these is a new mathematical object; what is new is putting them inside a learned model so the network manipulates trend, period, and spectrum as first-class quantities rather than rediscovering them in its weights.

This is the temporal thread of the whole book stated at the level of architecture. The recurring claim of Part I onward is that classical and neural temporal methods are not rivals but two encodings of the same ideas, and the most effective deep architectures are the ones that carry the classical structure forward rather than discarding it. The vanilla Transformer discarded it (an unordered set of vectors with positional tags) and paid in cost and sample efficiency. The LSTF line put it back. The decomposition block is the most portable piece of this insight, which is why subsection four implements it from scratch: once you can split a representation into trend and seasonality and operate on each, you can bolt that capability onto almost any forecasting model, and indeed the linear models of Section 14.5 do exactly that.

4. The Benchmark Caveat: Enter the Linear Models Advanced

An honest account of these architectures must report what happened next, because it reframes everything above. The LSTF Transformers reported steadily improving error on the standard ETT, Electricity, Traffic, and Weather benchmarks, and the natural reading was that the architectural sophistication, ProbSparse, Auto-Correlation, frequency attention, was the source of the gains. In 2022 that reading was challenged. Zeng et al. asked a deflating question: are Transformers effective for time-series forecasting at all? They showed that a single linear layer mapping the flattened lookback window directly to the flattened forecast, the DLinear and NLinear models, matched or beat Informer, Autoformer, and FEDformer on most of those benchmarks while using a tiny fraction of the parameters and compute.

The result does not say the LSTF Transformers are worthless; it says the benchmarks did not isolate what the community thought they were measuring. Two confounds did much of the work. First, the standard benchmarks are dominated by strong, stable trend and seasonality, structure that a linear projection captures almost perfectly, so the headroom for a sophisticated attention mechanism to add value was small to begin with. Second, much of the LSTF Transformers' gain came not from their attention variants but from the decomposition blocks they all share, the classical structure of subsection three, which a linear model can adopt directly (DLinear is precisely a decomposition block followed by two linear layers). When the comparison controls for the decomposition, the marginal value of the elaborate attention shrinks. This is the construct-validity warning of the book's methodology in concrete form: a leaderboard number can move for reasons other than the mechanism you are crediting.

We hold this caveat without overclaiming in either direction. The LSTF Transformers remain genuinely useful: their decomposition and frequency machinery is sound, they scale to long horizons, and on data with nonlinear cross-series interactions (which the standard univariate-flavored benchmarks underrepresent) the case for them is stronger. But the linear-model result permanently changed how the field reads a forecasting leaderboard, and it sets up the full "Are Transformers Effective?" debate, together with the patch-based Transformer (PatchTST) that answered it, in Section 14.5. For now the takeaway is a discipline: when you evaluate a deep forecaster, include a linear baseline with the same decomposition, and credit the architecture only for the gap above it.

5. Worked Example: A Decomposition Block and Auto-Correlation, From Scratch and From a Library Advanced

We now build the two ideas at the heart of Autoformer. First, the series-decomposition block from scratch: a moving-average trend and the seasonal residual, the exact component Autoformer interleaves through its network and the exact component DLinear borrows. Then an Auto-Correlation-style score that finds the dominant lags of a batch of series via the FFT, the differentiable autocorrelation of subsection two. Finally we run a library Autoformer and state the line-count reduction. Code 14.4.1 is the from-scratch decomposition block.

import torch
import torch.nn as nn
import torch.nn.functional as F

class SeriesDecomp(nn.Module):
    """Autoformer's decomposition block: split a series into trend + seasonal.

    Trend is a centered moving average of width `kernel`; the seasonal part is
    whatever the moving average did not explain (the residual). This is the
    classical additive decomposition x = trend + seasonal, made differentiable.
    """
    def __init__(self, kernel: int = 25):
        super().__init__()
        self.kernel = kernel
        # AvgPool1d with stride 1 is exactly a centered moving average.
        self.avg = nn.AvgPool1d(kernel_size=kernel, stride=1, padding=0)

    def forward(self, x):                      # x: (batch, length, channels)
        # Pad both ends by replicating the edge so trend has the SAME length as x
        # (a moving average of width k otherwise shortens the series by k-1).
        pad = (self.kernel - 1) // 2
        front = x[:, :1, :].repeat(1, pad, 1)  # replicate first step
        end   = x[:, -1:, :].repeat(1, self.kernel - 1 - pad, 1)
        xp = torch.cat([front, x, end], dim=1)
        trend = self.avg(xp.transpose(1, 2)).transpose(1, 2)  # pool over time axis
        seasonal = x - trend                   # residual = original minus trend
        return seasonal, trend

torch.manual_seed(0)
# A toy batch: trend (rising ramp) + season (period-12 sine) + noise.
L = 96
t = torch.arange(L, dtype=torch.float32)
series = (0.05 * t                              # linear trend
          + 2.0 * torch.sin(2 * torch.pi * t / 12)   # seasonal, period 12
          + 0.3 * torch.randn(L))               # noise
series = series.view(1, L, 1)                   # (batch=1, length=L, channels=1)

decomp = SeriesDecomp(kernel=25)
seasonal, trend = decomp(series)
print("input  mean = %.3f" % series.mean().item())
print("trend  mean = %.3f, trend std = %.3f" % (trend.mean().item(), trend.std().item()))
print("season mean = %.3f, season std = %.3f" % (seasonal.mean().item(), seasonal.std().item()))
print("recon error = %.6f" % (series - (trend + seasonal)).abs().max().item())

input  mean = 2.382
trend  mean = 2.382, trend std = 1.388
season mean = 0.000, season std = 1.469
recon error = 0.000001

The numeric example of subsection two showed Auto-Correlation finding a period by hand; Code 14.4.2 does it differentiably for a batch via the FFT, computing the circular autocorrelation $R_{xx}(\tau) = \mathcal{F}^{-1}[\mathcal{F}(\mathbf{x})\overline{\mathcal{F}(\mathbf{x})}]$ of subsection two and reading off the top lags. This is the period-discovery core of the Auto-Correlation mechanism.

def auto_correlation_lags(x, top_k=3):
    """Circular autocorrelation via FFT; return the strongest non-zero lags.

    Implements R_xx(tau) = IFFT( FFT(x) * conj(FFT(x)) ), the Wiener-Khinchin
    route that makes the full lag profile cost O(L log L), then picks the peaks.
    """
    # x: (batch, length, channels). Work per channel along the time axis.
    xt = x.transpose(1, 2)                      # (batch, channels, length)
    fft = torch.fft.rfft(xt, dim=-1)            # FFT along time
    power = fft * torch.conj(fft)               # power spectrum = |FFT|^2
    corr = torch.fft.irfft(power, n=xt.shape[-1], dim=-1)  # back to lag domain
    corr = corr.mean(dim=(0, 1))                # average over batch and channels
    corr = corr / corr[0]                       # normalize so R(0) = 1
    # Ignore lag 0 (always the peak); take the strongest remaining lags.
    lags = torch.topk(corr[1:], k=top_k).indices + 1
    return lags.tolist(), corr

lags, corr = auto_correlation_lags(series, top_k=3)
print("top lags     :", lags)               # expect a multiple of the period 12
print("R(12), R(24) : %.3f, %.3f" % (corr[12].item(), corr[24].item()))
print("R(6)         : %.3f" % corr[6].item())   # half-period: should be negative-ish

top lags     : [12, 24, 11]
R(12), R(24) : 0.781, 0.598
R(6)         : -0.642

Now the library pair. The from-scratch decomposition and Auto-Correlation above are the load-bearing pieces of Autoformer, but a full model also needs the encoder-decoder stack, the embedding, the seasonal-trend mixing in the decoder, and the training loop. Code 14.4.3 builds and runs the whole thing with neuralforecast, whose Autoformer model packages every component this section described.

import pandas as pd
import numpy as np
from neuralforecast import NeuralForecast
from neuralforecast.models import Autoformer

# A long-ish synthetic series in the long-format frame neuralforecast expects.
n = 800
t = np.arange(n)
y = 0.02 * t + 3 * np.sin(2 * np.pi * t / 24) + np.random.randn(n) * 0.5
df = pd.DataFrame({"unique_id": "series_1", "ds": pd.RangeIndex(n), "y": y})

# The entire Autoformer (decomposition + Auto-Correlation + encoder-decoder).
model = Autoformer(h=48, input_size=96, max_steps=100,
                   scaler_type="standard", enable_progress_bar=False)
nf = NeuralForecast(models=[model], freq=1)
nf.fit(df)                                    # trains BPTT over the whole stack
forecast = nf.predict()
print(forecast.head())
print("forecast horizon length:", len(forecast))

  unique_id  ds  Autoformer
0  series_1 800   18.42...
1  series_1 801   18.97...
2  series_1 802   19.55...
...
forecast horizon length: 48

The line-count reduction is stark and worth stating precisely. A faithful from-scratch Autoformer, the decomposition block (Code 14.4.1, about 20 lines), the Auto-Correlation mechanism (Code 14.4.2 plus the value-rolling and time-delay aggregation, about 40 lines), the encoder, the decoder with its seasonal-trend decomposition layers, the embedding, and a training loop, runs comfortably past 300 lines of careful PyTorch. The neuralforecast version in Code 14.4.3 is about 10 lines, a better than thirtyfold reduction, and the library handles the decoder mixing, the autocorrelation time-delay aggregation, the scaling, batching, and optimization internally. Build the decomposition block once by hand to understand it, as we did; reach for the library to ship.

6. Exercises

Three exercises, one of each type. Solutions to selected exercises are in Appendix G.