"They handed me a series I had never seen. No history with it, no chance to fit, no tuning budget, just a window of numbers and the instruction to say what comes next. The old me would have asked for the training data first. But I was pretrained on a billion series drawn from energy and traffic and retail and weather and finance, and somewhere in all that I learned the grammar of a trend, the shape of a seasonality, the way a level shifts and a spike decays. So I looked at the strange new series, recognized its dialect, and forecast it cold. Sometimes I beat the carefully tuned model that spent an afternoon learning only this one series. Sometimes I lost to a seasonal naive that a child could write. I never know in advance which it will be, and that, more than my billion-series education, is the honest part of my job."
A Pretrained Model Forecasting a Series It Has Never Met
Chapter Overview
Every chapter of Part III so far has assumed a workflow that the rest of machine learning quietly abandoned years ago: you have a series, you train a model on that series, you forecast that series. The recurrent network of Chapter 10, the temporal convolution of Chapter 11, the Transformer of Chapter 12, the state-space layer of Chapter 13, and the deep forecasters of Chapter 14 all fit fresh on the target data. Language and vision left that world behind when pretrained foundation models arrived: one large model, trained once on an enormous and diverse corpus, then applied to new tasks with no training at all or with a light finetune. Between 2023 and 2026, time series got its own foundation-model moment, and this chapter is about what arrived, how it was built, and the uncomfortable question of whether any of it actually beats the simple baselines this book taught you in Part II.
The new models fall into a few families that the chapter takes in turn. The first paradigm tokenizes a real-valued series so that a language-model architecture can be trained on it directly: Amazon's Chronos quantizes values into a vocabulary and trains a T5-style sequence model on hundreds of thousands of series, while Lag-Llama builds a decoder-only probabilistic forecaster from lagged features, and MOMENT pretrains a masked-reconstruction encoder usable across forecasting, classification, and anomaly detection. A second paradigm builds purpose-designed forecasting transformers at scale: Nixtla's TimeGPT was the first commercial forecasting API, Google's TimesFM is a decoder-only patched model pretrained on a corpus spanning Google Trends and Wikipedia traffic, and Salesforce's Moirai uses an any-variate masked encoder trained on the open LOTSA corpus to handle arbitrary numbers of channels and frequencies. A third strand goes the other way, toward small: the Tiny Time Mixer (TTM) shows a model with one to five million parameters can be competitive zero-shot, and TabPFN-TS adapts a pretrained tabular in-context learner to forecasting by framing it as a regression over time features. A fourth strand presses general-purpose large language models into forecasting directly, either by encoding the numbers as text (LLMTime) or by reprogramming the series into the model's embedding space (Time-LLM).
Through all of it the chapter holds one question in front of you, because the field too often does not: does zero-shot forecasting from a pretrained model actually beat a well-tuned classical baseline on your data? The honest answer from the 2024 to 2026 benchmark record is that it depends, and it depends in ways worth understanding. On many datasets a foundation model forecasts a never-seen series competitively with, and occasionally better than, a model fit specifically to it, which is genuinely remarkable and saves the entire fitting step. On other datasets, especially short, idiosyncratic, or strongly domain-specific ones, a seasonal naive or an AutoARIMA from Chapter 5 still wins, and a foundation model that confidently forecasts the wrong shape is worse than useless. Rigorous benchmarks like GiftEval now exist precisely to keep these claims honest across many series and horizons, and you should trust them over any single vendor chart.
A warning particular to this chapter: it sits on the currency frontier of the entire book, and it will date faster than anything else here. The model names, parameter counts, and leaderboard positions are a snapshot of a field releasing a major model every few months. The durable content is not which model leads today but the paradigms, how a series is tokenized or patched, what a model is pretrained to predict, how zero-shot and few-shot transfer are evaluated, and the discipline of always comparing against a tuned baseline. Learn the paradigms and the evaluation discipline, treat the specific leaderboard as perishable, and re-check the benchmarks before you commit a model to production.
Prerequisites
This chapter assumes the architectural vocabulary of the two chapters just before it and the baseline discipline of Part II. From Chapter 12: Attention and Transformers you need self-attention, the encoder-decoder and decoder-only forms, tokenization, and the pretraining-then-transfer paradigm in general, because every foundation model here is a Transformer pretrained at scale and the whole chapter is that paradigm carried into time series. From Chapter 14: Deep Forecasting Architectures you need patching, channel-independence, and probabilistic forecasting heads, since the foundation models reuse exactly those design choices, and you need the chapter's sobering lesson that a tuned linear or classical model is a serious competitor, because this chapter puts that lesson to the test against pretrained giants. From Chapter 5: Univariate Forecasting Models you need the baselines that are the measuring stick throughout: the seasonal naive and AutoARIMA against which every zero-shot claim must be checked, and the scale-free error metrics, MASE in particular, that make comparison across series meaningful. Readers wanting a refresher on transformers, on probabilistic forecasting and proper scoring rules, or on the classical baselines and error metrics will find each referenced chapter and the appendices indexed in the Table of Contents.
If you keep one idea from this chapter, keep this: a temporal foundation model is a Transformer pretrained once on a large, diverse corpus of time series, then applied to a never-seen series zero-shot or with a light finetune, and whether it beats a tuned classical baseline is an empirical question you must answer on your own data, not assume. The families differ in how they convert a series into tokens the architecture can pretrain on: Chronos quantizes values into a vocabulary, Lag-Llama uses lagged features, MOMENT masks and reconstructs patches, TimesFM and Moirai patch and forecast directly, TTM stays tiny, TabPFN-TS borrows a tabular in-context learner, and LLMTime and Time-LLM repurpose a language model. The paradigms and the evaluation discipline, especially comparing MASE and CRPS against a seasonal naive and AutoARIMA on a held-out series, are the durable lesson; the specific leaderboard is perishable and will date fastest in the book.
Chapter Roadmap
- 15.1 Pretraining Paradigms for Time Series What it means to pretrain on time series: how a real-valued series becomes tokens a Transformer can learn from, by value quantization, by lagged features, or by patching; the pretraining objectives of next-step prediction and masked reconstruction; the corpus problem of assembling enough diverse series; and why this paradigm followed language and vision into temporal data.
- 15.2 Chronos, Lag-Llama, and MOMENT The tokenize-and-pretrain family in detail: Chronos quantizing values into a vocabulary and training a T5-style model for probabilistic forecasts, Lag-Llama building a decoder-only forecaster on lagged features, and MOMENT pretraining a masked-reconstruction encoder reusable across forecasting, classification, and anomaly detection.
- 15.3 TimeGPT, TimesFM, and Moirai The purpose-built forecasting transformers at scale: Nixtla's TimeGPT as the first commercial forecasting API, Google's decoder-only patched TimesFM pretrained on web-scale traffic series, and Salesforce's any-variate masked Moirai trained on the open LOTSA corpus to handle arbitrary channels and frequencies in one model.
- 15.4 Tiny and Tabular Time-Series Models (TTM, TabPFN-TS) The counter-current toward small: the Tiny Time Mixer showing a one-to-five-million-parameter model can be competitive zero-shot at a fraction of the cost, and TabPFN-TS adapting a pretrained tabular in-context learner to forecasting by casting it as regression over time features, the case for not always reaching for the largest model.
- 15.5 LLMs for Time Series (LLMTime, Time-LLM, prompting) Pressing general-purpose language models into forecasting: LLMTime encoding the numbers as text so a frozen LLM continues the sequence, Time-LLM reprogramming a series into the model's embedding space with a textual prompt, the prompting patterns that make it work, and the active debate over whether the language model contributes anything beyond a competent backbone.
- 15.6 Zero-Shot, Few-Shot, and Transfer The evaluation chapter the hype skips: how to measure zero-shot and few-shot forecasting honestly with MASE and CRPS against tuned baselines, when a light finetune is worth it and when it is not, the leakage trap of pretraining corpora overlapping the test set, and reading GiftEval and similar benchmarks instead of vendor charts.
- 15.X Future Directions Where the frontier is heading: multimodal and covariate-aware foundation models, longer context and native irregular sampling, agentic and tool-using forecasting, and the open question of a single model that forecasts, classifies, and detects anomalies across every temporal task.
Once you have worked through the six sections, the Hands-On Lab below chains them into a single experiment. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have loaded a pretrained model from Section 15.2 or Section 15.3, forecast a batch of held-out series zero-shot as Section 15.1 describes, scored it against the baselines of Chapter 5, and run the honest comparison that Section 15.6 insists upon.
Hands-On Lab: Zero-Shot Forecasting Shootout
Objective
Put a temporal foundation model on trial against the simplest baselines in the book and find out, on your own data, whether the pretrained giant earns its place. You will load a pretrained model, Chronos or TimesFM of Section 15.2 and Section 15.3, and use it to forecast several held-out series it has never seen, entirely zero-shot, exactly the pretrain-once-forecast-anything paradigm of Section 15.1. You will then forecast the same series with two baselines from Chapter 5, a seasonal naive and an AutoARIMA, and score all three on the scale-free MASE and the probabilistic CRPS so the comparison is fair across series of different magnitudes. The first headline of the lab is the answer to the chapter's question on these series: how often, and by how much, does the foundation model beat the baselines, and on which series does the seasonal naive still win. Then you will do a light few-shot finetune of the foundation model on a small slice of each target series and measure the gain over its own zero-shot score, learning when the finetune is worth its cost and when zero-shot was already as good as it gets, the central practical judgment of Section 15.6.
What You'll Practice
- Loading a pretrained temporal foundation model and forecasting never-seen series zero-shot, following Section 15.1 and Section 15.2.
- Forecasting the same series with a seasonal naive and an AutoARIMA baseline from Chapter 5.
- Scoring point and probabilistic forecasts with MASE and CRPS so the comparison is scale-free and fair, following Section 15.6.
- Reading a head-to-head table to judge when the foundation model beats the baselines and when it does not, following Section 15.6.
- Running a light few-shot finetune and measuring its gain over the zero-shot score, following Section 15.4 and Section 15.6.
- Relating the result to the LLM-based forecasters of Section 15.5 and the tiny models of Section 15.4 as alternative backbones.
Setup
You need the chronos-forecasting package (which pulls the pretrained Chronos weights from HuggingFace) or the timesfm package for the foundation model, statsforecast from Nixtla for the AutoARIMA and seasonal-naive baselines, and a small batch of held-out univariate series; the GiftEval or Monash archive subsets of Section 15.6 work well and ship many series at once. Critically, you must confirm your test series are not in the foundation model's pretraining corpus, the leakage trap of Section 15.6, because a model that was pretrained on your test set is not forecasting zero-shot, it is recalling, and the split discipline of Chapter 2 applies to the pretraining corpus too. The few-shot step uses only a short prefix of each target series so the finetune is genuinely light. The code below loads a pretrained Chronos pipeline and produces a probabilistic zero-shot forecast for one series in a few lines.
import torch
from chronos import ChronosPipeline # pretrained tokenize-and-forecast model (Section 15.2)
pipeline = ChronosPipeline.from_pretrained( # downloads pretrained weights; no fitting on your data
"amazon/chronos-t5-small", device_map="cpu", torch_dtype=torch.float32)
context = torch.tensor(series_history) # a held-out series the model has never seen
forecast = pipeline.predict(context, prediction_length=24, num_samples=100) # zero-shot, probabilistic
median = forecast.median(dim=1).values # point forecast for MASE; full samples give CRPS
Steps
Step 1: Forecast a batch of held-out series zero-shot
Load the pretrained model and run it across every held-out series with no fitting, producing a probabilistic forecast over the test horizon for each, the pretrain-once-forecast-anything move of Section 15.1. Confirm first that none of these series appear in the model's pretraining corpus, so the result is genuine zero-shot transfer and not recall.
Step 2: Forecast the same series with classical baselines
Fit a seasonal naive and an AutoARIMA from Chapter 5 to each series and forecast the same horizon. These are the measuring stick: a foundation model that cannot beat a seasonal naive on a seasonal series has not earned its compute, and the seasonal naive costs nothing to run.
Step 3: Score everything on MASE and CRPS
Compute MASE for the point forecasts and CRPS for the probabilistic ones, the scale-free metrics of Section 15.6, so that a series measured in dollars and a series measured in counts contribute comparably. Aggregate across the batch and build the head-to-head table that is the spine of the lab.
Step 4: Read the table and answer the chapter's question
Read off how often the foundation model wins, by how much, and on which series it loses to the seasonal naive. This is the honest answer of Section 15.6 on your data: not a single number but a distribution of wins and losses, with the losses concentrated on the short, idiosyncratic, or strongly domain-specific series the model never saw the like of.
Step 5: Light few-shot finetune and measure the gain
Finetune the foundation model briefly on a short prefix of each target series and re-score it against its own zero-shot numbers. On some series the finetune buys a clear MASE drop; on others it changes almost nothing because zero-shot was already near the achievable error. Recording which is which is the practical judgment of Section 15.6, whether the finetune is worth its cost.
Expected Output
The lab produces one honest table and a clear verdict. The head-to-head of Steps 3 and 4 shows the foundation model winning on a substantial share of series, often by a meaningful MASE and CRPS margin, while losing on a stubborn minority where the seasonal naive or AutoARIMA holds, the wins-and-losses distribution that is the truthful answer to the chapter's question rather than the single triumphant number a vendor chart would show. The zero-shot result is itself the headline: a model that fit nothing on these series still forecasts many of them competitively with a model fit specifically to each. Step 5 adds the finetune verdict, a per-series record of where a light few-shot pass bought a real improvement and where zero-shot was already as good as it gets, which is exactly the cost-benefit call you will face in practice. The reader finishes able to put any new foundation model on trial the same way, with MASE, CRPS, a seasonal naive, and an AutoARIMA as the standing jury, and able to read its leaderboard claims with the skepticism this fast-moving frontier demands.
The entire pretrain-and-deploy pipeline that took the research teams behind these models months of compute is, for the consumer, a few library lines. The chronos-forecasting package downloads the pretrained weights and produces a probabilistic forecast for any series in three calls, with the value tokenization, the T5 backbone, and the sampling all handled internally; timesfm and the Moirai weights on HuggingFace are equally short, and statsforecast supplies the AutoARIMA and seasonal-naive baselines and the MASE scorer in a matching few lines so the whole shootout is well under a hundred lines of code. The discipline of the chapter still holds: the three-line forecast is trivial to produce and dangerously easy to trust, so the work that matters is not loading the model but the comparison, confirming no leakage from the pretraining corpus, scoring against a real baseline on a scale-free metric, and reading the wins and losses honestly. The library makes the model free to run; it does not make the model right.
Stretch Goals
- Add a tiny model, the TTM of Section 15.4, to the shootout and compare its zero-shot accuracy and its inference cost against the larger foundation model, testing whether the small model is competitive at a fraction of the compute.
- Add an LLM-based forecaster, LLMTime of Section 15.5, by encoding the series as text and prompting a frozen language model, and see where a general-purpose LLM lands against the purpose-built forecasters.
- Re-run the whole shootout on a standardized benchmark subset such as GiftEval of Section 15.6 and compare your numbers to the published leaderboard, learning to read those tables and to distrust any result you cannot reproduce.
What's Next?
This chapter closes Part III: Temporal Deep Learning. Over seven chapters the part built the full neural toolkit for sequences, from the recurrence and the convolution through attention, structured state-space models, deep forecasting architectures, and now the pretrained foundation models that forecast a series cold. Every one of those models, however, treats its job as prediction: map a past window to a future one. Part IV: Temporal Representation Learning opens with Chapter 16: Learning Temporal Representations, which asks a different question, not what comes next but what a stretch of series means, learning embeddings of temporal data without forecasting labels through self-supervised and contrastive objectives. Those learned representations turn out to be exactly what powers transfer in the foundation models you just met, so Part IV explains from underneath why pretraining on time series works at all, and it sets up the generative and event models that follow. The foundation models of this chapter are the destination of Part III and the doorway to the representation learning of Part IV.
Bibliography & Further Reading
Tokenize-and-Pretrain Models
Ansari, A. F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., et al. "Chronos: Learning the Language of Time Series." TMLR, 2024. arXiv:2403.07815. arxiv.org/abs/2403.07815
The Chronos paper that quantizes series values into a vocabulary and trains a T5-style model for probabilistic forecasting, the central tokenize-and-pretrain example of Section 15.2 and the model used in the lab.
Rasul, K., Ashok, A., Williams, A. R., Khorasani, A., Adamopoulos, G., Bhagwatkar, R., Bilos, M., Ghonia, H., et al. "Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting." 2023. arXiv:2310.08278. arxiv.org/abs/2310.08278
Lag-Llama, a decoder-only probabilistic forecaster built on lagged features, an early open foundation model for forecasting and a key example of the lagged-feature paradigm in Section 15.2.
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., Dubrawski, A. "MOMENT: A Family of Open Time-Series Foundation Models." ICML, 2024. arXiv:2402.03885. arxiv.org/abs/2402.03885
MOMENT, a masked-reconstruction encoder pretrained for forecasting, classification, and anomaly detection alike, the multi-task foundation model of Section 15.2.
Purpose-Built Forecasting Transformers
Garza, A., Challu, C., Mergenthaler-Canseco, M. "TimeGPT-1." 2023. arXiv:2310.03589. arxiv.org/abs/2310.03589
TimeGPT from Nixtla, the first commercial foundation-model forecasting API, the opening case of the purpose-built scale-up in Section 15.3.
Das, A., Kong, W., Sen, R., Zhou, Y. "A Decoder-Only Foundation Model for Time-Series Forecasting." ICML, 2024. arXiv:2310.10688. arxiv.org/abs/2310.10688
Google's TimesFM, a decoder-only patched model pretrained on web-scale traffic series, the second pillar of Section 15.3 and an alternative contestant in the lab.
Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., Sahoo, D. "Unified Training of Universal Time Series Forecasting Transformers." ICML, 2024. arXiv:2402.02592. arxiv.org/abs/2402.02592
Salesforce's Moirai, an any-variate masked encoder trained on the open LOTSA corpus to handle arbitrary channels and frequencies in one model, the third pillar of Section 15.3.
Tiny and Tabular Models
Ekambaram, V., Jati, A., Nguyen, N. H., Dayama, P., Reddy, C., Gifford, W. M., Kalagnanam, J. "Tiny Time Mixers (TTM): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series." NeurIPS, 2024. arXiv:2401.03955. arxiv.org/abs/2401.03955
The Tiny Time Mixer, showing a one-to-five-million-parameter model can be competitive zero-shot at a fraction of the cost, the central case for small in Section 15.4.
Hoo, S. B., Muller, S., Salinas, D., Hutter, F. "The Tabular Foundation Model TabPFN Outperforms Specialized Time Series Forecasting Models Based on Simple Features." 2025. arXiv:2501.02945. arxiv.org/abs/2501.02945
TabPFN-TS, adapting a pretrained tabular in-context learner to forecasting by casting it as regression over time features, the tabular route of Section 15.4.
Hollmann, N., Muller, S., Purucker, L., Krishnakumar, A., Korfer, M., Hoo, S. B., Schirrmeister, R. T., Hutter, F. "Accurate Predictions on Small Data with a Tabular Foundation Model." Nature, 2025. nature.com/articles/s41586-024-08328-6
The Nature paper introducing TabPFN, the in-context tabular foundation model that TabPFN-TS adapts to forecasting, the backbone behind the tabular approach of Section 15.4.
LLMs for Time Series
Gruver, N., Finzi, M., Qiu, S., Wilson, A. G. "Large Language Models Are Zero-Shot Time Series Forecasters." NeurIPS, 2023. arXiv:2310.07820. arxiv.org/abs/2310.07820
LLMTime, encoding a series as text so a frozen language model continues it, the surprising zero-shot result that opens Section 15.5.
Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., Wen, Q. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." ICLR, 2024. arXiv:2310.01728. arxiv.org/abs/2310.01728
Time-LLM, reprogramming a series into a language model's embedding space with a textual prompt, the reprogramming route of Section 15.5.
Benchmarks & Tools
Aksu, T., Woo, G., Liu, J., Liu, X., Liu, C., Savarese, S., Xiong, C., Sahoo, D. "GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation." 2024. arXiv:2410.10393. arxiv.org/abs/2410.10393
GiftEval, a standardized benchmark across many series, frequencies, and horizons for evaluating foundation forecasters honestly, the leaderboard to trust over vendor charts in Section 15.6 and the lab's stretch goal.
Salesforce AI Research. "uni2ts: Unified Training of Universal Time Series Forecasting Transformers (Moirai and the LOTSA corpus)." GitHub. github.com/SalesforceAIResearch/uni2ts
The reference library for Moirai and the open LOTSA pretraining corpus, the toolkit and data behind the any-variate model of Section 15.3 and a source of leakage-checked test series for the lab.
Amazon Science. "chronos-forecasting: Pretrained probabilistic time-series models." GitHub. github.com/amazon-science/chronos-forecasting
The reference package that downloads the pretrained Chronos weights and produces probabilistic forecasts in a few lines, the engine behind the zero-shot contestant in the lab.