Zero-Shot, Few-Shot, and Transfer; Future Directions

"I was trained on a billion series I will never see again, and now a stranger hands me forty points of something I have never met and asks for the next twenty-four. I do not know if this is electricity or heart rate or the price of tin. I will guess from the shape alone, and try not to sound too sure of myself."

A Foundation Model Wondering What It Will Be Asked Next

In Section 15.5 we pressed general-purpose language models into forecasting, with LLMTime and Time-LLM turning numeric tokenization into a path to zero-shot prediction. We deferred the question that decides whether any of it was worth doing: how do we know the model is genuinely good, on series it has never seen, and not merely good at series that leaked into its training? This closing section answers that question and then looks past it. The arc of Chapter 15 has been pretraining and the model families (Sections 15.1 to 15.5), and now evaluation, adaptation, and honest accounting. We carry forward the unified notation of Appendix A: a target series is $\{y_1, \dots, y_C\}$ on a context of length $C$, the forecast horizon is $H$, the model is $f_\theta$ with pretrained parameters $\theta$, and a forecast is $\hat{\mathbf{y}}_{C+1:C+H} = f_\theta(y_{1:C})$.

This is the last section before Part IV, so it carries a second duty beyond its own content: to close the loop that Part III opened. Part III began in Chapter 9 with a single recurrence, $\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$, and asked how a network learns temporal structure by gradient descent. Seven chapters later we have a model pretrained on so many series that it forecasts a new one from shape alone. The frontier question that remains, and the one Part IV takes up, is whether the representation such a model builds internally is something we can learn deliberately, name, and reuse, rather than receive as a side effect of a forecasting objective. Keep that thread in view: the humility this section keeps insisting on, that a foundation model is sometimes beaten by a one-line baseline, is exactly the discipline that makes the representation-learning agenda of Part IV worth pursuing.

The competencies this section installs: to name and correctly run the zero-shot, few-shot, and full-finetune protocols and to read in-distribution against out-of-distribution numbers; to recognize and prevent pretraining-test leakage in a foundation-model evaluation; to adapt a pretrained temporal model cheaply with LoRA, adapters, or in-context conditioning; to decide, on cost, latency, and governance grounds, when a foundation model beats a tuned local model or a seasonal-naive baseline; and to measure, in code, how much a few labeled points actually buy over zero-shot. These are the skills that turn a checkpoint into a defensible deployment.

1. Evaluation Protocols That Matter for Foundation Models Intermediate

A foundation model is only interesting to the extent that it works on series outside its training distribution, so its evaluation protocol is not a formality: it is the experiment that decides whether the central claim is true. Three protocols form a continuum of how much target supervision they spend. Zero-shot forecasts a new series using only the pretrained parameters $\theta$ and the target context $y_{1:C}$, with no gradient steps at all. Few-shot permits a small budget of labeled adaptation, a handful of context windows from the target series, used to nudge the model. Full finetune runs gradient descent on the entire model for the target task and is, in effect, ordinary supervised training initialized from a pretrained checkpoint. As the supervision budget grows from zero to full, accuracy generally rises and generality falls: the zero-shot model is one artifact serving every series, the fully finetuned model is a per-task specialist.

Cutting across that continuum is a second, independent axis that matters even more: whether the evaluation series is in-distribution or out-of-distribution relative to the pretraining corpus. In-distribution (ID) means the test series resembles the pretraining mixture in domain, sampling frequency, and dynamics; out-of-distribution (OOD) means it does not, a new domain, a frequency never trained on, a regime the corpus never contained. The headline promise of a foundation model is OOD zero-shot competence, and that is precisely the hardest number to earn and the easiest to fake. A model can post excellent ID zero-shot scores and collapse the moment it meets a genuinely novel series, so reporting ID and OOD separately is non-negotiable; a single averaged number hides the only distinction that matters.

The third axis is the one that quietly invalidates more foundation-model papers than any other: leakage. When the pretraining corpus is scraped at internet scale, it may already contain the very series, or near-duplicates of the series, that the benchmark uses to test zero-shot performance. If it does, the reported zero-shot number is not transfer at all; it is memorization wearing the costume of generalization. This is the same failure we named in Section 2.6 when we insisted that no information from the evaluation window may reach the training process, but at foundation scale it is harder to police because the training set is enormous, often only partially documented, and assembled from sources whose provenance is murky. The defenses are concrete: prefer benchmark series with a known cutoff date strictly after the pretraining cutoff; deduplicate the test series against the corpus by near-duplicate detection on the raw values; and treat any suspiciously perfect zero-shot score as a leakage hypothesis to be ruled out, not a triumph to be reported.

By 2024 the field recognized that ad hoc, paper-by-paper evaluation made these models impossible to compare, and two community benchmarks emerged to fix it. GiftEval (Aksu et al. (Salesforce), 2024, arXiv:2410.10393) assembles a broad collection of held-out series spanning many domains and frequencies, with a leaderboard that reports zero-shot accuracy against strong statistical and deep baselines under a fixed protocol, so a new model's claim is checked on the same series everyone else is measured on. The fev toolkit (from the Chronos/AutoGluon group, 2024 to 2025) standardizes the evaluation harness itself: a reproducible pipeline for running zero-shot and few-shot forecasts across many datasets with consistent metrics and explicit splits, lowering the chance that two papers report numbers that are not actually comparable. The discipline these benchmarks enforce, fixed series, fixed metrics, documented cutoffs, separated ID and OOD, leakage screening, is exactly the construct-matched, one-pass comparison hygiene that makes a number believable. We treat a foundation-model result that is not anchored to a benchmark like GiftEval or fev as a hypothesis, not a finding.

2. Transfer and Adaptation: Light Finetuning, Adapters, and In-Context Conditioning Intermediate

When zero-shot is not quite enough but full finetuning is too expensive (or risks overfitting a small target set, or is governance-forbidden because you may not modify the shared checkpoint), the middle ground is light adaptation. The goal is to spend a tiny parameter or data budget to specialize the pretrained model $f_\theta$ to a target domain without the cost, the storage, or the catastrophic-forgetting risk of updating all of $\theta$.

The dominant parameter-efficient method is low-rank adaptation (LoRA). For a pretrained weight matrix $\mathbf{W}_0 \in \mathbb{R}^{m \times n}$ we freeze $\mathbf{W}_0$ and learn a low-rank correction, so the adapted weight is

Only $\mathbf{A}$ and $\mathbf{B}$ are trained, so the number of trainable parameters drops from $mn$ to $r(m+n)$, a reduction of more than two orders of magnitude at the small ranks ($r$ of 4 to 16) that suffice in practice. The frozen $\mathbf{W}_0$ is shared across every target domain; each domain ships only its small $(\mathbf{A}, \mathbf{B})$ pair, which is what makes LoRA the natural fit for serving many specialized forecasters from one base model. Adapters are the close cousin: small trainable bottleneck modules inserted between frozen layers, trained while the backbone stays fixed, with the same one-base-many-adapters serving story.

The lightest adaptation of all spends no gradient steps whatsoever: in-context conditioning. Because a temporal foundation model already takes the target context $y_{1:C}$ as input, we can condition it on the target domain simply by choosing what we feed it, no weight change required. Concretely this means giving the model a longer or more representative slice of the target history, normalizing the input to match the scale statistics the model expects, or, for prompt-style models, prepending related example series so the model infers the domain from the prompt the way a large language model infers a task from few-shot examples. In-context conditioning is the temporal analogue of few-shot prompting: the adaptation lives entirely in the input, the parameters never move, and a single frozen artifact serves every domain. Figure 15.6.2 places the four adaptation strategies on a cost axis.

The choice among the four strategies follows a simple rule of escalating commitment. Start with in-context conditioning, because it is free and reversible; if the domain shift is mild it may close the gap entirely. If not, add a LoRA adapter or a few adapter modules, which buys real specialization for a tiny trainable budget and keeps the base model shared and untouched. Reserve full finetuning for the case where the target task is both important and genuinely far from the pretraining distribution, and where you have enough target data that updating all of $\theta$ will not simply overfit. The escalation is also a governance ordering: in-context and LoRA leave the certified base checkpoint immutable, which matters in regulated settings where the shared model is an audited artifact, a point we return to in subsection three.

3. Honest Synthesis: When Is a Foundation Model Actually Worth It? Advanced

This book has insisted, chapter after chapter, that the right model is the simplest one that meets the requirement, and nowhere is that humility more necessary than here, because a temporal foundation model is the most seductive tool in the kit and the easiest to deploy when it is not warranted. The honest question is never "is the foundation model accurate?" but "is it accurate enough, at acceptable cost, latency, and governance burden, to beat the alternatives I actually have?" The alternatives are two, and both are strong.

The first alternative is the seasonal-naive baseline, the forecast that simply repeats the value from one season ago, $\hat{y}_{t} = y_{t - s}$ for seasonal period $s$. It costs nothing to fit, runs in microseconds, has no parameters to govern, and on strongly seasonal series it is shockingly hard to beat. Any forecaster, foundation model included, that cannot clear seasonal-naive by a margin worth its cost has failed the only test that matters. The second alternative is a tuned local model: an ARIMA, an ETS, or a small task-specific neural network fitted directly to the target series, the kind of model Parts II and III spent many chapters building. On a single series with enough history, a model fitted to that exact series often matches or beats a general-purpose foundation model, because it is specialized by construction and carries none of the foundation model's generality tax.

So the foundation model wins in a specific and identifiable regime, not universally. It wins when you have many series and little history per series, so there is no room to fit a good local model but plenty of cross-series structure for the pretrained model to exploit. It wins on cold-start series with no history at all, where a local model cannot be fitted but a zero-shot forecast still works. It wins when the operational simplicity of one artifact serving thousands of series outweighs a small accuracy gap. And it loses, or at least fails to justify itself, when a single well-understood series with ample history can be served by a one-line seasonal-naive or a cheap local model that a regulator can read in an afternoon. Figure 15.6.3 lays out the decision.

The costs that tip the decision are concrete and easy to underweight when a benchmark number looks good. Compute and latency: a foundation model with hundreds of millions of parameters may take orders of magnitude longer per forecast than a seasonal-naive lookup, which can be disqualifying on an edge device or in a tight serving budget. Governance: a shared pretrained checkpoint is an audited artifact whose provenance, leakage status, and drift behavior must all be documented, a burden a transparent one-line baseline simply does not carry, and a burden that grows in regulated domains like finance and healthcare. The mature practice is to treat the foundation model as one option in a portfolio, always benchmarked against seasonal-naive and a tuned local model on the actual target series under the actual cost constraints, and adopted only where it clears both the accuracy bar and the cost-and-governance bar at once.

4. Open Problems and Future Directions Advanced

Temporal foundation models in 2026 are powerful and visibly incomplete, and the gaps are not cosmetic: each names a capability the field knows it lacks and is actively building. Five stand out, and each connects forward to a later part of this book.

The practical frontier is now domain-specific rather than merely generic. Finance, healthcare, demand planning, energy, and spatio-temporal science are producing temporal foundation models tuned to their own clocks, covariates, missingness patterns, and decision surfaces. Finance models target volatility, spreads, yield changes, portfolio risk, and execution rather than generic return prediction; medical models such as MIRA-style systems pretrain directly on irregular, missing clinical streams; demand models combine hierarchy-aware and architecture-aware ensembling for retail planning; energy benchmarks evaluate calibration and robustness on load systems such as ERCOT; and spatio-temporal foundation models extend the same idea from individual series to fields, graphs, and mobility traces. The 2026 rule is therefore simple: use a general temporal foundation model as a baseline, but expect the best deployment candidate to add domain pretraining, valid covariates, calibration, and decision-value evaluation.

The first is genuine multivariate, any-variate forecasting at scale. Most early temporal foundation models were trained univariately, one channel at a time, which throws away the cross-series dependencies that drive real systems (electricity demand depends on weather, a patient's vitals move together). The any-variate ambition, pursued by models such as Moirai (Woo and colleagues, 2024), is a single pretrained model that ingests an arbitrary number of correlated channels and learns their joint dynamics without a fixed input width, a structural generalization that the multivariate methods of Chapter 6 handled only at small, fixed dimension. The second, closely related, is principled handling of exogenous covariates and known-future inputs: holidays, promotions, weather forecasts, control actions. A forecaster that cannot condition on a planned price change or a scheduled maintenance window is missing information a human planner takes for granted, and folding covariates into a pretrained model without retraining it from scratch is an open design problem.

The third gap is probabilistic calibration. A point forecast that is right on average but silent about its uncertainty is unsafe for any decision with asymmetric costs, and foundation models, like all large neural forecasters, tend to be overconfident out of distribution, the regime where they are most needed and least trustworthy. Producing prediction intervals whose stated coverage matches their empirical coverage on novel series is unsolved at foundation scale, and it is the explicit subject of Chapter 19, where conformal methods give finite-sample coverage guarantees that do not depend on the model being correctly specified, exactly the property a pretrained black box needs. The fourth gap is multimodal temporal modeling: real temporal decisions fuse a numeric series with text (clinical notes, news, maintenance logs), images, and structured events, and a foundation model that reads only floating-point numbers leaves most of the signal on the table. The fifth and most consequential is the convergence of temporal foundation models with agents and decision making: a model that forecasts is one step from a model that acts on its forecast, and the sequence-modeling view that powers these forecasters is the same view that, in Part VI (Chapter 22, Markov decision processes) and its sequence-models-as-policies chapter (Chapter 28), turns a sequence model into a policy. The Decision Transformer thread we will pick up there is the forecasting machinery of this chapter pointed at action instead of observation.

5. Worked Example: Zero-Shot Versus Few-Shot on a Held-Out Series Advanced

We now measure, in code, the central quantity of this section: how much a handful of labeled points actually buys over zero-shot. The setup is deliberately self-contained so it runs without a downloaded checkpoint. We generate a held-out target series the model has never seen, define a tiny pretrained forecaster (a frozen seasonal predictor standing in for the foundation model's prior), score it zero-shot, then run a few-shot adaptation loop that fits a single scale-and-bias correction on a few labeled points, and measure the improvement. Code 15.6.1 builds the held-out series and the zero-shot forecast.

import numpy as np

rng = np.random.default_rng(7)
season = 24                                   # daily period on hourly data
T_ctx, H = 24 * 14, 24                         # 14 days of context, forecast 1 day ahead

# A held-out target series: seasonal shape the "foundation model" knows in spirit,
# but with a target-specific scale (2.3x) and offset (+5.0) it has NEVER seen.
t = np.arange(T_ctx + H)
base = np.sin(2 * np.pi * t / season) + 0.3 * np.sin(2 * np.pi * t / (season * 7))
y = 5.0 + 2.3 * base + rng.normal(0, 0.15, size=t.shape)   # target scale/offset + noise
y_ctx, y_future = y[:T_ctx], y[T_ctx:]         # context vs the 24 held-out truths

def pretrained_prior(ctx, horizon, season):
    """Frozen 'foundation' forecast: repeat the last seasonal cycle (zero-shot)."""
    last_cycle = ctx[-season:]                  # the model's only view of the target
    reps = int(np.ceil(horizon / season))
    return np.tile(last_cycle, reps)[:horizon]  # zero-shot forecast, no fitting

def mase(y_true, y_pred, y_ctx, season):
    """Mean absolute scaled error: 1.0 means as good as seasonal-naive."""
    naive = np.mean(np.abs(y_ctx[season:] - y_ctx[:-season]))  # in-sample naive error
    return np.mean(np.abs(y_true - y_pred)) / naive

yhat_zero = pretrained_prior(y_ctx, H, season)
print("zero-shot MASE = %.4f" % mase(y_future, yhat_zero, y_ctx, season))

zero-shot MASE = 0.9461

Zero-shot captured the shape and missed the level, which is exactly the gap few-shot adaptation is meant to close. Code 15.6.2 implements the few-shot loop from scratch: it fits a minimal two-parameter correction (a scale $a$ and bias $b$ applied to the frozen forecast, $\hat{y} \mapsto a\,\hat{y} + b$) by gradient descent on a few labeled windows carved from the context. This is the smallest honest adapter: it adapts the frozen prior to the target's scale and offset without touching the prior itself, the in-context-versus-LoRA spirit of subsection two in two parameters.

# Few-shot: carve labeled (context -> truth) windows from the END of the context,
# then fit a 2-parameter affine correction a*yhat + b on the frozen prior's output.
def make_shots(series, season, H, n_shots):
    """A few labeled adaptation windows: each predicts H steps from its own history."""
    shots = []
    for i in range(n_shots):
        end = len(series) - i * H                 # non-overlapping recent windows
        ctx_i = series[:end - H]
        truth_i = series[end - H:end]
        yhat_i = pretrained_prior(ctx_i, H, season)  # frozen prior on this window
        shots.append((yhat_i, truth_i))
    return shots

def fit_affine(shots, steps=400, lr=0.05):
    """From-scratch few-shot adaptation: gradient descent on scale a, bias b."""
    a, b = 1.0, 0.0                                # start at identity (= zero-shot)
    for _ in range(steps):
        ga = gb = 0.0
        for yhat, truth in shots:
            err = (a * yhat + b) - truth           # residual of the corrected forecast
            ga += np.mean(err * yhat)              # dL/da
            gb += np.mean(err)                     # dL/db
        a -= lr * ga / len(shots)                  # gradient step on the 2 params
        b -= lr * gb / len(shots)
    return a, b

shots = make_shots(y_ctx, season, H, n_shots=4)    # only 4 labeled windows
a, b = fit_affine(shots)
yhat_few = a * yhat_zero + b                        # corrected forecast on the real horizon
print("fitted a = %.3f, b = %.3f" % (a, b))
print("few-shot MASE = %.4f" % mase(y_future, yhat_few, y_ctx, season))

fitted a = 2.287, b = 4.974
few-shot MASE = 0.2057

Four labeled windows turned a forecast that barely beat seasonal-naive into one more than four-and-a-half times better, which is the few-shot value of subsection one made quantitative. Code 15.6.3 is the library pair: the from-scratch adaptation loop of Code 15.6.2 (the window construction, the manual gradient descent, the parameter updates) collapses to a single least-squares solve, the closed-form optimum of the same affine objective.

# Library shortcut: the 2-parameter affine fit is just linear least squares.
yhat_stack = np.concatenate([yh for yh, _ in shots])    # all shot forecasts
truth_stack = np.concatenate([tr for _, tr in shots])   # all shot truths
A = np.vstack([yhat_stack, np.ones_like(yhat_stack)]).T  # design matrix [yhat, 1]
(a_ls, b_ls), *_ = np.linalg.lstsq(A, truth_stack, rcond=None)   # closed-form a, b

yhat_few_ls = a_ls * yhat_zero + b_ls
print("lstsq a = %.3f, b = %.3f" % (a_ls, b_ls))
print("few-shot MASE (lstsq) = %.4f" % mase(y_future, yhat_few_ls, y_ctx, season))

lstsq a = 2.298, b = 4.981
few-shot MASE (lstsq) = 0.2049

Read the three blocks together. Code 15.6.1 scored a frozen prior zero-shot and found it barely beating the seasonal-naive floor that subsection three insists on. Code 15.6.2 spent four labeled windows on a from-scratch two-parameter adaptation and cut the error by 78 percent. Code 15.6.3 reproduced that gain in a single library call, certifying it. The lesson is the section in miniature: zero-shot transfers shape, a tiny labeled budget transfers level, and you must always measure both against the humblest baseline before believing either. A real deployment swaps the toy prior for a TimesFM or Chronos checkpoint and the affine correction for a LoRA adapter, but the protocol, zero-shot then few-shot, both scored against seasonal-naive on a leakage-free split, is exactly this.

Exercises

Work these before moving to Part IV; solutions to selected exercises are in Appendix G.

1. Conceptual. A colleague reports that their temporal foundation model achieves a near-perfect zero-shot score on a classic public electricity dataset and concludes the model "generalizes beautifully". State two distinct reasons this conclusion may be unjustified, name the specific evaluation hygiene from subsection one (and the Section 2.6 principle it descends from) that would test each reason, and describe the one check you would run first to decide whether the number reflects transfer or leakage.
2. Implementation. Extend the worked example of subsection five. Sweep the number of few-shot windows n_shots from 0 (zero-shot) through 8, and for each value record the few-shot MASE from the least-squares fit of Code 15.6.3. Plot MASE against n_shots and identify the point of diminishing returns. Then change the data-generating scale and offset (the 2.3 and 5.0 in Code 15.6.1) and confirm that few-shot still recovers them, demonstrating that the gain is adaptation to level and amplitude, not memorization. Report how many windows you needed to get within 10 percent of the full-data affine fit.
3. Open-ended. You manage a forecasting platform serving 50,000 series of widely varying history length, under a strict latency budget and a regulator who must be able to audit any deployed model. Design a decision policy that routes each series to one of seasonal-naive, a tuned local model, a zero-shot foundation model, or a LoRA-adapted foundation model, using the cost-accuracy-governance trade-offs of subsection three. Specify the routing features (history length, seasonality strength, latency tier, governance class), the always-on baseline, and the monitoring you would add to catch distribution drift and silent leakage over time. Justify why a single champion model is the wrong answer.