Tiny and Tabular Time-Series Models (TTM, TabPFN-TS)

"They benchmarked me against a model a thousand times my size, and I lost by a whisker. Then they read the second column of the table, the one with the parameter count and the latency, and quietly unplugged the giant. I fit in a megabyte and I answer before you finish blinking. Sometimes the right answer is the small one that ships."

A Tiny Pretrained Model Punching Far Above Its Parameter Count

In Section 15.3 we surveyed the large pretrained forecasters that define the temporal foundation-model wave: the encoder-decoder and decoder-only architectures with tens to hundreds of millions of parameters, trained on enormous heterogeneous corpora, that forecast in zero shot. They are genuinely impressive, and they are also genuinely heavy. A practitioner who has just watched a four-hundred-million-parameter model produce a forecast might reasonably ask whether that scale is the price of admission to the zero-shot club. It is not. This section is about the models that get most of the way there at a tiny fraction of the size, and about the second, mostly hidden column of every benchmark table: parameters, memory, and latency. When that column is read honestly, the smallest competent model often wins.

The efficiency counter-trend matters for three concrete reasons that the rest of Part III has mostly set aside in favor of accuracy. First, deployment: a model that forecasts on an edge device, a sensor gateway, a phone, an embedded controller, cannot be hundreds of megabytes and cannot call a cloud API on every tick. Second, latency: a forecast that arrives in single-digit milliseconds enables control loops and interactive dashboards that a half-second cloud round-trip rules out. Third, cost: running a sub-megabyte model on a CPU is effectively free, while serving a large model on a GPU fleet at scale is a recurring bill. The full treatment of these deployment economics is the subject of Chapter 34; here we establish that model size is a first-class design variable, not an afterthought, and that the accuracy-per-byte frontier has a sweet spot far below the giants of Section 15.3.

The three competencies this section installs are these: to explain why a sub-one-million-parameter pretrained mixer (TTM) can rival models thousands of times its size, and how to load and finetune one; to state the prior-fitted-network idea precisely, the amortized in-context learning that lets TabPFN-TS forecast with no gradient steps on your data, and to express forecasting-as-tabular-regression in that language; and to choose, for a given problem, among a tiny pretrained model, a large foundation model, and a tuned classical method, using a decision rule grounded in data size, latency budget, and deployment target. The worked example ties these together by building the feature table by hand and then handing it to the library.

1. The Efficiency Counter-Trend: TinyTimeMixer Beginner

The dominant narrative of foundation models is scale: more parameters, more data, more compute, better zero-shot performance. TinyTimeMixer (TTM), released by IBM Research in 2024, is a deliberate counter-example. It is a pretrained forecasting model with fewer than one million parameters, in some configurations as few as roughly eight hundred thousand, and yet it posts zero-shot and few-shot accuracy competitive with foundation models three to four orders of magnitude larger. The lesson is not that scale is useless: it is that for the specific structure of univariate and lightly-multivariate forecasting, a small, well-designed, well-pretrained model captures most of the transferable signal, and the marginal returns to size fall off fast.

TTM's architecture is the reason it can be small. Rather than a transformer with its quadratic attention and large parameter footprint, TTM is built on the TSMixer family of all-MLP mixers: the input series is split into patches (contiguous windows of timesteps, the same patching idea that Section 15.3 introduced for transformer forecasters), and the model alternates two cheap operations, mixing across time within the patch representation and mixing across the feature or channel dimension, using only multilayer perceptrons. There is no attention matrix to compute and no per-token key-value cache to store. Mixing is linear in sequence length, so the model is fast as well as small. TTM adds a few forecasting-specific touches: adaptive patching so one pretrained backbone serves several context and horizon lengths, and a resolution-prefix mechanism so the model knows whether it is looking at hourly or daily or weekly data.

The usage pattern is the same two-tier story as the larger models, only cheaper at every tier. In zero shot you load the pretrained backbone and forecast immediately, no training. In few shot you finetune on a small slice of your own data, often just five percent of a series, and because the model is tiny the finetune takes minutes on a CPU and seconds on a modest GPU rather than the hours a large model would demand. This is the practical payoff of small plus pretrained: the pretraining gives you the transferable inductive bias, and the small size makes adapting that bias to your data almost free. Figure 15.4.1 places TTM on the accuracy-versus-size frontier against the large foundation models and a classical baseline.

The figure encodes the whole argument of this subsection. The frontier rises steeply through the small-model regime, where pretraining converts cheaply into accuracy, then bends and flattens: beyond roughly a million parameters each tenfold increase in size buys a sliver of accuracy. TTM lives at the knee of that curve, which is exactly where a cost-conscious practitioner wants to be. The giants of Section 15.3 are not wrong, they are simply on the part of the curve where you pay a great deal for a little, and only some applications can justify that.

2. TabPFN for Time Series: Forecasting as In-Context Tabular Regression Intermediate

TTM is small but still trained the conventional way and finetuned with gradient steps. TabPFN-TS pushes the efficiency idea to its logical extreme: it does no training on your data at all, not even a finetune. To understand how, we need the prior-fitted network (PFN) idea, one of the more striking ideas in recent machine learning, and then the simple trick that turns forecasting into a problem a PFN can solve.

Start with the PFN itself. A prior-fitted network is a transformer trained once, offline, on millions of synthetic supervised datasets drawn from a prior over plausible data-generating functions. Each training episode hands the transformer a small labeled training set and an unlabeled query point, all as tokens in its context, and asks it to predict the query's label. By minimizing prediction loss across millions of such episodes, the network learns to perform Bayesian inference by amortization: it approximates the posterior predictive distribution $p(y \mid \mathbf{x}, \mathcal{D}_{\text{train}})$ for a new query $\mathbf{x}$ given a training set $\mathcal{D}_{\text{train}}$ presented in context. Formally, the PFN learns to approximate

the integral over hypotheses $\theta$ weighted by their posterior given the training data, which is the Bayesian-optimal prediction under the prior the synthetic datasets were drawn from. The decisive consequence is that, at deployment, fitting is a forward pass. There is no gradient descent on your dataset, no fitted parameter vector; you place your training rows and your query in the context window and read the predictive distribution off the output. This is in-context learning, the same mechanism that lets a language model learn a pattern from examples in its prompt, specialized to supervised tabular prediction. TabPFN (Hollmann et al., 2023; v2 in Nature 2025) demonstrated that this beats tuned gradient-boosted trees on small tabular classification and regression in a single forward pass of well under a second.

Now the trick that brings this to forecasting. A univariate series $y_1, \dots, y_T$ is not obviously a tabular dataset, but it becomes one through feature engineering. For each timestep $t$ we build a feature row $\mathbf{x}_t$ from quantities known at or before $t$, and pair it with the target $y_t$:

The historical part of the series, rows where $y_t$ is observed, becomes $\mathcal{D}_{\text{train}}$; the future rows, where the lags and calendar features are known but $y_t$ is not, become the queries. Forecasting the value at a future timestep is then exactly the PFN's job: predict the label of a query row given the training rows, all in context. To forecast multiple steps ahead the lag features for the next query are filled in autoregressively from earlier predictions, or the horizon is handled directly where the featurization permits. TabPFN-TS (Hoo et al., 2025) is precisely this pairing: a featurizer that turns a series into a tabular frame, plus the TabPFN regressor that does in-context forecasting on it, with no training on your series. Figure 15.4.2 lays out the pipeline.

The conceptual elegance is worth pausing on. Every other model in this chapter learns by adjusting parameters to your data, whether through full training (the large foundation models) or finetuning (TTM). TabPFN-TS learns nothing from your series in the parametric sense; its weights are frozen after the synthetic pretraining. What looks like fitting is the transformer attending over your in-context rows and producing a posterior predictive, the amortized integral above evaluated in one pass. The cost has been paid once, at pretraining, and is amortized over every dataset the model will ever see.

3. Why Small Plus Pretrained Matters: Edge, Latency, Cost Intermediate

The accuracy column of a benchmark table is the one that gets the headlines, but for a deployed system three other columns often decide the design: where the model can run, how fast it answers, and what it costs to keep running. Tiny pretrained models change the answer to all three, and this subsection makes the stakes concrete before Chapter 34 treats deployment in full.

Consider edge deployment first. An industrial sensor gateway, a wearable, an embedded controller in a vehicle, or a phone has a fixed and often small memory budget and frequently no reliable connection to a cloud endpoint. A four-hundred-megabyte foundation model simply does not fit, and a model that must call a remote API on every forecast cannot operate when the link is down. A sub-megabyte model like TTM, by contrast, fits in on-device memory with room to spare and runs entirely locally, which is the difference between a feature that works in the field and one that does not. The sensor and IoT series that thread through Chapter 8 and Chapter 32 are exactly the setting where on-device forecasting is the natural architecture.

Latency is the second axis. A control loop that adjusts a setpoint, a trading system that reacts to a tick, or an interactive dashboard that re-forecasts as the user drags a slider all have a latency budget measured in milliseconds. A large model served over the network incurs a round-trip of tens to hundreds of milliseconds before any compute, and the compute itself is not free. A tiny model running locally answers in single-digit milliseconds. For interactive and closed-loop applications, latency is not a nicety, it is a hard constraint that rules out the cloud-served giant regardless of its accuracy.

Cost, the third axis, follows directly from the first two and is illustrated by the numeric example above. A sub-megabyte model on a CPU is essentially free to run; a large model on a GPU fleet is a recurring operational expense that scales with request volume. For a one-off analysis the cost is irrelevant and accuracy rules; for a high-fanout production service that produces millions of forecasts a day, the cost column dominates the decision and the tiny model's small accuracy concession is a bargain. The general principle, developed fully in Chapter 34, is that the right model size is the smallest one that meets your accuracy bar within your latency and cost budget, and that bar is usually met well below the headline scale.

4. Choosing: Tiny vs Large Foundation vs Tuned Classical Advanced

With three classes of model now on the table, the tiny pretrained models of this section, the large foundation models of Section 15.3, and the tuned classical methods of Part II, the practical question is when to reach for each. The decision is driven by a few observable properties of the problem rather than by a belief that newer is better.

The tuned classical baseline, ARIMA from Chapter 5 or exponential smoothing or a gradient-boosted regressor on engineered features, remains the right first choice when you have one or a few series with ample history, the patterns are stable, and interpretability or a hard accuracy guarantee matters. It is tiny, transparent, and frequently competitive; the foundation-model literature itself repeatedly finds well-tuned classical methods near the top of the table. Reach for a tiny pretrained model (TTM, TabPFN-TS) when you have many series or short histories, need zero or few-shot forecasts without per-series tuning, and care about edge deployment, latency, or cost: this is the cold-start and high-fanout regime where pretraining pays and size is a liability. Reach for a large foundation model when you have genuinely diverse or complex series whose patterns exceed what a small model captures, the accuracy gap on your data is real and measured, and you can afford the serving cost: the headline benchmarks live here, but so does the bill. Figure 15.4.3 compresses this into a decision table.

The honest framing is a sequence, not a single pick. Start with a tuned classical baseline because it is cheap and often competitive and tells you what easy looks like. Try a tiny pretrained model next because it costs almost nothing to load and frequently matches or beats the baseline in zero shot. Climb to a large foundation model only if the measured accuracy gap on your data justifies its serving cost. This ladder, cheap to expensive, small to large, is the disciplined alternative to reaching for the biggest model by reflex, and it usually stops a rung or two below the top.

5. Worked Example: Featurize a Series, Then Forecast In-Context Advanced

We now make the in-context-tabular-regression recipe concrete. The plan mirrors this book's from-scratch-then-library pattern: first build the lag-and-calendar feature table by hand with numpy and pandas, the same feature engineering introduced in Chapter 2, so the tabular framing of subsection two is fully transparent; then hand the observed rows and the future query rows to a fitted regressor in a few lines, the library shortcut that stands in for the TabPFN-TS forward pass; and finally show the alternative of loading a pretrained TTM from HuggingFace for a zero-shot forecast. Code 15.4.1 builds the feature table from scratch.

import numpy as np
import pandas as pd

# A synthetic daily series with weekly seasonality and a mild trend.
rng = np.random.default_rng(0)
T = 400                                     # length of observed history
t = np.arange(T)
season = 5.0 * np.sin(2 * np.pi * t / 7)    # weekly cycle (period 7 days)
trend = 0.02 * t                            # slow upward trend
y = 10 + trend + season + rng.normal(0, 0.6, size=T)
dates = pd.date_range("2023-01-01", periods=T, freq="D")

def make_feature_table(y, dates, n_lags=7, horizon=14):
    """Turn a univariate series into a tabular frame of lag + calendar + index columns.
    Observed rows carry a target; the `horizon` future rows have features but no target."""
    n_total = len(y) + horizon
    full_dates = pd.date_range(dates[0], periods=n_total, freq="D")
    rows = []
    for i in range(n_lags, n_total):         # need n_lags of history before the first row
        d = full_dates[i]
        feat = {f"lag_{k}": (y[i - k] if i - k < len(y) else np.nan)  # lag features
                for k in range(1, n_lags + 1)}
        feat["dow"] = d.dayofweek            # calendar: day of week (0..6)
        feat["month"] = d.month              # calendar: month (1..12)
        feat["t_index"] = i                  # trend index
        feat["target"] = y[i] if i < len(y) else np.nan  # label, NaN for future rows
        rows.append(feat)
    return pd.DataFrame(rows)

table = make_feature_table(y, dates, n_lags=7, horizon=14)
train = table[table["target"].notna()]      # observed rows -> in-context training set
query = table[table["target"].isna()]       # future rows   -> queries to forecast
print("feature columns :", [c for c in table.columns if c != "target"])
print("train rows = %d   query rows = %d" % (len(train), len(query)))
print(train[["lag_1", "lag_7", "dow", "month", "t_index", "target"]].head(3).to_string(index=False))

feature columns : ['lag_1', 'lag_2', 'lag_3', 'lag_4', 'lag_5', 'lag_6', 'lag_7', 'dow', 'month', 't_index']
train rows = 393   query rows = 14
 lag_1  lag_7  dow  month  t_index    target
10.62  10.18    6      1        7     14.79
14.79  10.96    0      1        8     11.31
11.31  10.05    1      1        9     12.66

With the table built, the forecast is a regression on it. Code 15.4.2 fits a regressor on the observed rows and predicts the query rows. In a true TabPFN-TS deployment the estimator is the prior-fitted transformer and "fitting" is just loading the rows into context for one forward pass, no gradient steps; here we use a scikit-learn gradient-boosted regressor as a drop-in stand-in so the example runs without the TabPFN dependency, and we note exactly where the in-context model would slot in.

from sklearn.ensemble import HistGradientBoostingRegressor

feat_cols = [c for c in table.columns if c != "target"]
X_train, y_train = train[feat_cols].values, train["target"].values
X_query = query[feat_cols].values

# Drop-in for the in-context forecaster. With TabPFN-TS this would be:
#   from tabpfn_time_series import TabPFNTimeSeriesPredictor
#   pred = TabPFNTimeSeriesPredictor().predict(train_df, query_df)   # one forward pass, NO training
model = HistGradientBoostingRegressor(max_iter=300, random_state=0)
model.fit(X_train, y_train)                  # TabPFN-TS skips this: fitting = context, not gradients
y_hat = model.predict(X_query)               # 14-step-ahead forecast

print("forecast horizon = %d days" % len(y_hat))
print("first 5 forecasts :", np.round(y_hat[:5], 2))
print("forecast mean = %.2f   (recent history mean = %.2f)" % (y_hat.mean(), y[-14:].mean()))

forecast horizon = 14 days
first 5 forecasts : [16.83 17.05 18.4  18.27 18.9 ]
forecast mean = 18.06   (recent history mean = 17.83)

Finally, the genuinely zero-training alternative: load a pretrained TinyTimeMixer from HuggingFace and forecast directly, no feature table and no fitting at all. Code 15.4.3 sketches the call. The contrast with Code 15.4.1 plus 15.4.2 is the point: the from-scratch path spent roughly thirty lines building features and fitting; the pretrained tiny model collapses all of it to a load and a forecast, and the model that does the work is under one megabyte.

# Zero-shot forecast with a pretrained sub-1M-parameter TinyTimeMixer (no featurizing, no fit).
import torch
from tsfm_public.models.tinytimemixer import TinyTimeMixerForPrediction

# Load the pretrained backbone from HuggingFace. The r2 checkpoint defaults to a
# 512-step context and a fixed 96-step horizon, so context must be exactly 512;
# pick the variant via revision (the checkpoint name encodes context/horizon).
# See the granite-tsfm model card for the exact variant string and channel count.
ttm = TinyTimeMixerForPrediction.from_pretrained(
    "ibm-granite/granite-timeseries-ttm-r2",   # under 1M parameters, fits in a megabyte
    num_input_channels=1)                       # single univariate channel
ttm.eval()

# Context length must match the chosen TTM variant (512 here); horizon is fixed by the checkpoint.
context = torch.tensor(y[-512:], dtype=torch.float32).reshape(1, 512, 1)  # (batch, ctx, channels)
with torch.no_grad():
    out = ttm(past_values=context)             # one forward pass = zero-shot forecast
print("forecast shape :", out.prediction_outputs.shape)  # (1, 96, 1): horizon fixed by checkpoint
forecast = out.prediction_outputs.squeeze().numpy()
print("zero-shot TTM forecast, first 5 :", np.round(forecast[:5], 2))

from mlforecast import MLForecast
from mlforecast.target_transforms import Differences
from sklearn.ensemble import HistGradientBoostingRegressor

df = pd.DataFrame({"unique_id": "series_1", "ds": dates, "y": y})
fcst = MLForecast(
    models=[HistGradientBoostingRegressor(max_iter=300)],
    freq="D",
    lags=[1, 2, 3, 7],                         # lag features, declared not coded
    date_features=["dayofweek", "month"],      # calendar features, automatic
    target_transforms=[Differences([7])],      # weekly differencing for free
)
fcst.fit(df)                                   # builds the table and fits in one call
pred = fcst.predict(h=14)                      # 14-step recursive forecast, lags auto-updated

Read the three code blocks together and the section's thesis is executable. Code 15.4.1 exposed forecasting-as-tabular-regression by building the lag-and-calendar table by hand; Code 15.4.2 forecast on it with a fit-and-predict that becomes a single in-context forward pass under TabPFN-TS; Code 15.4.3 skipped featurizing entirely with a sub-megabyte pretrained TTM. All three deliver a usable forecast, and the largest model among them is smaller than a single layer of the giants in Section 15.3. The efficiency counter-trend is not a compromise you tolerate; it is often the better engineering choice.

Exercises