"Every quarter a new architecture is crowned. One season it is an autoregressive RNN sampling its own future, the next a stack of fully connected blocks with no recurrence at all, then a Transformer that reads a year of hourly data in one pass and promises to dethrone everything before it. I publish the winner, I update my rankings, I retire last season's champion to a footnote. And then, with tiresome regularity, someone fits a single linear layer to the same benchmark, tunes it honestly, and beats half my leaderboard. I demote the Transformers, I promote the line, I issue a correction. I have learned to hold my crownings loosely, because the only model I have never had to dethrone is the seasonal naive baseline that quietly keeps showing up, refusing to lose."
A Benchmark Leaderboard Changing Its Mind Every Quarter
Chapter Overview
The previous chapters of Part III built sequence layers: the recurrent cell of Chapter 10, the temporal convolution of Chapter 11, the attention block of Chapter 12, and the linear recurrence of Chapter 13. This chapter assembles those layers into complete forecasting systems and confronts the question a practitioner actually faces: faced with a real series and a deadline, which named architecture do you reach for? The literature offers a crowded zoo of deep forecasters, each arriving with a benchmark table showing it on top, and the central skill this chapter teaches is not how to implement any single one but how to navigate the zoo, group its inhabitants by the problem they were built to solve, and judge their claims against an honest baseline.
The architectures fall into a few clean families, and the chapter visits each in turn. The probabilistic line begins with DeepAR, an autoregressive recurrent network that predicts a full distribution rather than a point and samples trajectories forward, the model to reach for when you need calibrated uncertainty and have many related series to train on jointly. The pure-MLP line strips recurrence and attention away entirely: N-BEATS and its faster, multi-rate successor N-HiTS stack fully connected blocks with a basis-expansion structure that yields interpretable trend and seasonality components, while TSMixer and TiDE show that simple mixing of time and feature dimensions, with no attention at all, competes with far heavier models. The covariate-rich interpretable line is anchored by the Temporal Fusion Transformer, which combines variable selection, static and time-varying covariate handling, and interpretable attention into a single architecture built for the messy multi-input forecasting problems of industry.
The long-sequence Transformer line is the chapter's most instructive arc, and its most cautionary. Informer, Autoformer, and FEDformer each attacked the quadratic cost of attention on long-horizon forecasting, Autoformer and FEDformer notably by injecting classical ideas, series decomposition and frequency-domain filtering, directly into the deep model, picking up the spectral thread from Chapter 4 and the decomposition thread from Chapter 5. Then came the reckoning. The DLinear paper asked whether Transformers were even effective for long-term forecasting and answered, embarrassingly, that a single linear layer on the right preprocessing beat them on standard benchmarks; PatchTST recovered the Transformer's honor by patching the input and treating channels independently, but only after the simple baseline had forced the whole field to validate more carefully. That deflating, productive lesson, that a well-tuned simple model keeps winning, is the through-line this chapter carries from Chapter 3 and Chapter 5 straight into the deep-learning era.
The temporal thread runs visibly through this chapter. The classical decomposition of trend and seasonality that N-BEATS learns as basis functions and that Autoformer builds into its attention, the frequency-domain filtering that FEDformer performs in the spectral basis of Chapter 4, the autoregressive sampling that DeepAR inherits from the ARIMA lineage of Chapter 5: each deep forecaster is a classical idea given learnable weights. By the end of the chapter you will be able to read a new forecasting paper, place its architecture in one of these families, identify which classical idea it has absorbed, and, most importantly, know to check it against a seasonal naive and a tuned linear model before believing its leaderboard.
Prerequisites
This chapter assembles the sequence layers built earlier in Part III, so it leans on three of them directly. From Chapter 12: Attention and Transformers you need self-attention, the encoder-decoder structure, and the quadratic cost in sequence length, because the long-sequence forecasters of Section 14.4 and the patched Transformer of Section 14.5 are defined by how they reshape and economize attention. From Chapter 10: Recurrent Neural Networks you need the recurrent cell and autoregressive generation, since DeepAR in Section 14.1 is a recurrent network that samples its own future one step at a time. From Chapter 5: Univariate Forecasting Models you need the seasonal naive and the classical decomposition and ARIMA baselines, because this entire chapter is an argument about when a deep model is worth its cost relative to those baselines, and you cannot judge a deep forecaster without the simple one to beat. Familiarity with forecasting error metrics and the temporal train and validation split discipline is assumed throughout. Readers wanting refreshers on the classical baselines, on attention, or on the recurrent cell will find every referenced chapter and the appendices indexed in the Table of Contents.
If you keep one idea from this chapter, keep this: the deep forecasting zoo sorts into a few families, probabilistic (DeepAR), pure-MLP (N-BEATS, N-HiTS, TSMixer, TiDE), covariate-rich interpretable (TFT), and long-sequence Transformer (Informer, Autoformer, FEDformer, PatchTST), each absorbing a classical idea (autoregression, decomposition, frequency filtering) into learnable weights, and the discipline that matters most is to validate every one of them against a seasonal naive and a tuned linear baseline, because, as DLinear showed, the simple model keeps winning more often than the leaderboards admit. Pick the family by the problem (uncertainty, interpretability, covariates, horizon length), not by the most recent benchmark crown.
Chapter Roadmap
- 14.1 DeepAR and Probabilistic RNN Forecasting The autoregressive recurrent forecaster that predicts a full predictive distribution rather than a point, trained jointly across many related series and sampled forward to produce calibrated probabilistic trajectories, the model of choice when uncertainty matters.
- 14.2 N-BEATS and N-HiTS The pure fully connected forecasters: N-BEATS with its doubly residual stacks and interpretable trend and seasonality basis expansion, and the faster multi-rate N-HiTS that adds hierarchical interpolation for long horizons, no recurrence or attention required.
- 14.3 Temporal Fusion Transformer (TFT) The covariate-rich interpretable architecture: variable selection networks, principled handling of static and known-future and observed covariates, and interpretable multi-head attention, built for the messy multi-input forecasting problems of industry.
- 14.4 Informer, Autoformer, and FEDformer The long-sequence Transformer line: Informer's sparse ProbSparse attention, Autoformer's series-decomposition and auto-correlation block, and FEDformer's frequency-domain attention, the deep forecasters that injected the classical decomposition and spectral ideas of Chapters 4 and 5 into attention.
- 14.5 PatchTST, DLinear, and the Transformer Debate The reckoning: DLinear's deflating demonstration that a single linear layer can beat the LSTF Transformers on standard benchmarks, and PatchTST's recovery of the Transformer through input patching and channel independence, the episode that forced the whole field to validate honestly.
- 14.6 TSMixer, TiDE, and MLP-Based Forecasters The lightweight resurgence: TSMixer's mixing of time and feature dimensions and TiDE's dense encoder-decoder, MLP-only architectures that match or beat far heavier models, closing the case that simple, well-validated forecasters remain hard to displace.
Once you have worked through the six sections, the Hands-On Lab below puts them in direct competition. Each lab model maps to a section, so the lab doubles as a review: by the time you finish you will have run the probabilistic DeepAR of Section 14.1, the multi-rate N-HiTS of Section 14.2, the patched PatchTST and the linear DLinear of Section 14.5, and a seasonal naive baseline head to head, and seen for yourself whether the fancy models earn their cost.
Hands-On Lab: The Deep Forecasting Bake-Off
Objective
Run a fair, honest bake-off across the families this chapter surveyed and let the numbers, not the leaderboards, decide. You will fit five forecasters on a standard benchmark series: the probabilistic DeepAR of Section 14.1, the multi-rate N-HiTS of Section 14.2, the patched PatchTST of Section 14.5, the single-layer DLinear of Section 14.5, and a seasonal naive baseline that costs nothing to fit. Using the neuralforecast library so every model shares one interface and one training harness, you will evaluate point accuracy with MASE, probabilistic accuracy with CRPS where a model produces a distribution, and you will record each model's training time. The central question is the one this whole chapter has been building toward: do the fancy deep models actually beat the linear baseline and the seasonal naive on this series, and if so, by how much and at what cost in training time? You will produce a single comparison table, draw your own conclusion, and, following the discipline of Section 14.4 and Section 14.6, refuse to crown any model that does not clearly clear the baseline.
What You'll Practice
- Fitting a probabilistic DeepAR forecaster and reading its predictive distribution, following Section 14.1.
- Training a pure-MLP N-HiTS forecaster and comparing it against attention-based models, following Section 14.2.
- Running PatchTST and DLinear side by side to reproduce the Transformer-versus-linear debate, following Section 14.5.
- Computing MASE for point accuracy and CRPS for probabilistic accuracy on one shared split, following Section 14.1.
- Recording training time per model and weighing accuracy gains against compute cost, following Section 14.6.
- Validating every deep model against a seasonal naive baseline before believing any leaderboard claim, following Section 14.4.
Setup
You need Nixtla's neuralforecast for the deep models (it bundles DeepAR, N-HiTS, PatchTST, DLinear, and many more behind one interface), utilsforecast or datasetsforecast for a standard benchmark series and the MASE and CRPS metrics, and the seasonal naive from statsforecast as the zero-cost baseline. All models train on the same temporal split, with the validation horizon held out after the training window exactly as Chapter 2 requires, so that a strong score reflects genuine extrapolation rather than a leak, and all share one set of horizon and input-window settings so the comparison is apples to apples. The single short code block below stands up the entire field in a handful of lines; the per-model details are the subject of the sections it cites.
from neuralforecast import NeuralForecast
from neuralforecast.models import DeepAR, NHITS, PatchTST, DLinear
H = 96 # forecast horizon; same for every model (fair comparison)
models = [ # one family each: probabilistic, MLP, patched-Transformer, linear
DeepAR(h=H, input_size=2*H, max_steps=500), # Section 14.1: probabilistic RNN
NHITS(h=H, input_size=2*H, max_steps=500), # Section 14.2: multi-rate MLP
PatchTST(h=H, input_size=2*H, max_steps=500), # Section 14.5: patched Transformer
DLinear(h=H, input_size=2*H, max_steps=500), # Section 14.5: single linear layer
]
nf = NeuralForecast(models=models, freq='H')
nf.fit(df=train_df) # shared harness; seasonal-naive baseline added from statsforecast
forecasts = nf.predict() # compare MASE / CRPS and training time against the baseline
Steps
Step 1: Load a benchmark series and fix one fair split
Load a standard hourly benchmark series and carve a single temporal split, training window then held-out horizon, that every model will share. Fix the forecast horizon and input window once so no model gets an unfair window, and compute the seasonal naive forecast immediately as the bar every deep model must clear, following the baseline discipline of Section 14.4.
Step 2: Fit DeepAR and read its distribution
Train the probabilistic DeepAR of Section 14.1 and inspect not just its median forecast but its predictive intervals. This is the model whose output is a distribution, so record its CRPS, the probabilistic score the point models cannot earn, and note its training time as the cost of that uncertainty.
Step 3: Fit N-HiTS, PatchTST, and DLinear on the identical split
Train the pure-MLP N-HiTS of Section 14.2, the patched Transformer PatchTST and the single-layer DLinear of Section 14.5 on the same window and horizon. The PatchTST-versus-DLinear pairing reproduces the Transformer-versus-linear debate in miniature on your own series; record each model's training time as you go.
Step 4: Score everything with MASE and CRPS
Compute MASE for every model's point forecast and CRPS for every model that produced a distribution, all on the one shared held-out horizon. Assemble a single table with one row per model and columns for MASE, CRPS, and training time, the artifact that makes the comparison honest because every number was co-computed on one split.
Step 5: Ask whether the fancy models beat the baseline
Read the table against the seasonal naive. For each deep model, ask: did it beat the naive baseline on MASE, and did its improvement justify its training time? Following Section 14.6, draw the deflating-or-vindicating conclusion the data supports, and resist crowning any model that merely tied a baseline that cost nothing.
Expected Output
The lab produces one comparison table and one honest verdict. The table holds five rows, DeepAR, N-HiTS, PatchTST, DLinear, and the seasonal naive, with MASE for all, CRPS for DeepAR (and any other distributional model), and training time for each. The typical and instructive result is mixed: on many standard benchmark series the deep models do beat the seasonal naive, sometimes substantially, but DLinear, the single linear layer, often lands within striking distance of PatchTST at a tiny fraction of its training time, exactly the deflating finding of Section 14.5, while DeepAR earns its keep precisely when calibrated intervals matter more than a sharper point. The reader finishes with the one habit this chapter most wants to instill: never report a deep forecaster's accuracy without the baseline beside it, and always weigh the gain against the compute it cost. Whatever the numbers say on your particular series, you will trust them, because you co-computed every one of them on a single fair split.
Implementing DeepAR, N-HiTS, PatchTST, and DLinear from scratch with matched training loops, schedulers, and probabilistic losses would run to many hundreds of lines and weeks of debugging the differences between papers. Nixtla's neuralforecast supplies all of them, and a dozen more including TFT, TSMixer, TiDE, Informer, Autoformer, and FEDformer, behind one fit and predict interface, so swapping families is a one-line change and the comparison is automatically fair. Its sibling statsforecast supplies the seasonal naive and the classical baselines at near-zero cost, and utilsforecast supplies MASE, CRPS, and the other metrics co-computed in one pass. For the probabilistic and GluonTS-style models, Amazon's gluonts is the alternative reference stack with DeepAR at its center. The discipline of the chapter still holds: understand what each architecture computes, as the sections do, then let these libraries run the bake-off so your effort goes into honest evaluation rather than re-implementation.
Stretch Goals
- Add the Temporal Fusion Transformer of Section 14.3 with static and known-future covariates and check whether the extra covariate machinery buys accuracy on a series that actually has informative covariates.
- Add TSMixer and TiDE of Section 14.6 to the table and test whether the lightweight MLP mixers match the patched Transformer at lower cost on your series.
- Repeat the bake-off across several benchmark series and report how often DLinear beats PatchTST, quantifying the strength of the simple-baseline lesson of Section 14.5 beyond a single dataset.
What's Next?
This chapter surveyed the named deep forecasters and taught the discipline of judging them against an honest baseline. Chapter 15: Temporal Foundation Models takes the next, larger step: instead of training a fresh model on each series, it asks what happens when a single large model is pretrained on an enormous corpus of time series and then forecasts a new series zero-shot, with no fitting at all. It covers the foundation forecasters of the 2024 to 2026 frontier, Chronos, Lag-Llama, Moirai, TimesFM, and MOMENT, the architectures and tokenization schemes that let one model generalize across domains and frequencies, and the question that closes Part III: has forecasting become a matter of prompting a pretrained model rather than fitting your own? The bake-off discipline you practiced here carries straight in, because a zero-shot foundation model, too, must be measured against the seasonal naive before it is believed.
Bibliography & Further Reading
Probabilistic Forecasting
Salinas, D., Flunkert, V., Gasthaus, J., Januschowski, T. "DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks." International Journal of Forecasting, 2020. arXiv:1704.04110. arxiv.org/abs/1704.04110
The DeepAR paper introducing the autoregressive recurrent network that predicts a full predictive distribution and trains jointly across many related series, the entire subject of Section 14.1.
Pure-MLP Forecasters
Oreshkin, B. N., Carpov, D., Chapados, N., Bengio, Y. "N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting." ICLR, 2020. arXiv:1905.10437. arxiv.org/abs/1905.10437
The N-BEATS paper with its doubly residual fully connected stacks and interpretable trend and seasonality basis expansion, the foundation of Section 14.2.
Challu, C., Olivares, K. G., Oreshkin, B. N., Garza, F., Mergenthaler-Canseco, M., Dubrawski, A. "N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting." AAAI, 2023. arXiv:2201.12886. arxiv.org/abs/2201.12886
The N-HiTS paper adding multi-rate sampling and hierarchical interpolation for fast, accurate long-horizon forecasting, the faster successor to N-BEATS in Section 14.2.
Chen, S.-A., Li, C.-L., Yoder, N., Arik, S. O., Pfister, T. "TSMixer: An All-MLP Architecture for Time Series Forecasting." TMLR, 2023. arXiv:2303.06053. arxiv.org/abs/2303.06053
The TSMixer paper showing that mixing time and feature dimensions with MLPs alone, no attention, competes with heavier models, a central architecture of Section 14.6.
Das, A., Kong, W., Leach, A., Mathur, S., Sen, R., Yu, R. "Long-term Forecasting with TiDE: Time-series Dense Encoder." TMLR, 2023. arXiv:2304.08424. arxiv.org/abs/2304.08424
The TiDE paper, a dense MLP encoder-decoder that matches or beats Transformer forecasters at far lower cost, the other lightweight model of Section 14.6.
Covariate-Rich Interpretable Forecasting
Lim, B., Arik, S. O., Loeff, N., Pfister, T. "Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting." International Journal of Forecasting, 2021. arXiv:1912.09363. arxiv.org/abs/1912.09363
The Temporal Fusion Transformer paper with its variable selection, covariate handling, and interpretable attention, the entire subject of Section 14.3.
Long-Sequence Transformers
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W. "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting." AAAI, 2021. arXiv:2012.07436. arxiv.org/abs/2012.07436
The Informer paper introducing ProbSparse attention to bring the cost of long-sequence forecasting down, the first of the LSTF Transformers in Section 14.4.
Wu, H., Xu, J., Wang, J., Long, M. "Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting." NeurIPS, 2021. arXiv:2106.13008. arxiv.org/abs/2106.13008
The Autoformer paper building series decomposition and an auto-correlation block into attention, injecting the classical decomposition idea of Chapter 5 into the deep model, in Section 14.4.
Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R. "FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting." ICML, 2022. arXiv:2201.12740. arxiv.org/abs/2201.12740
The FEDformer paper performing attention in the frequency domain, picking up the spectral thread of Chapter 4, the third LSTF Transformer of Section 14.4.
The Transformer Debate
Zeng, A., Chen, M., Zhang, L., Xu, Q. "Are Transformers Effective for Time Series Forecasting?" AAAI, 2023. arXiv:2205.13504. arxiv.org/abs/2205.13504
The DLinear paper that showed a single linear layer beats the LSTF Transformers on standard benchmarks, the deflating reckoning at the center of Section 14.5.
Nie, Y., Nguyen, N. H., Sinthong, P., Kalagnanam, J. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." ICLR, 2023. arXiv:2211.14730. arxiv.org/abs/2211.14730
The PatchTST paper recovering the Transformer through input patching and channel independence, the answer to DLinear that closes the debate in Section 14.5.
Tools & Libraries
Olivares, K. G., Challu, C., Garza, F., Canseco, M. M., Dubrawski, A. "NeuralForecast: User friendly state-of-the-art neural forecasting models." Nixtla, GitHub. github.com/Nixtla/neuralforecast
Nixtla's library bundling DeepAR, N-HiTS, PatchTST, DLinear, TFT, and the rest behind one interface, the engine of the bake-off lab.
Alexandrov, A., et al. "GluonTS: Probabilistic and Neural Time Series Modeling in Python." Amazon, JMLR, 2020. ts.gluon.ai
The GluonTS toolkit, the reference probabilistic forecasting stack with DeepAR at its center, the alternative library for the distributional models of Section 14.1.