Cost, Scaling, and Efficiency

"My validation MAPE was the best the team had ever seen, and for one glorious week I was the model that would change everything. Then finance forwarded the cloud invoice. It turns out predicting two million SKUs every fifteen minutes on an A100 fleet is not, strictly speaking, free. I am now being retired in favor of a seasonal-naive baseline that costs the price of a sandwich."

A Model Whose Accuracy Looked Great Until the Cloud Bill Arrived

In Section 34.3 we built the feature and online stores that supply a temporal model with point-in-time-correct, skew-free inputs, after Section 34.1 turned it into a live endpoint with batching and latency budgets. The serving section asked whether a prediction arrives in time, and the feature section asked whether its inputs are correct. With serving, monitoring, and features in place, this one asks the question none of them did: what it all costs to run, across every series and every refresh, day after day. The questions are tightly coupled: the same batching that hides latency also amortizes cost, and the same architecture choice that bounds streaming state also bounds the compute you pay for. But cost has its own logic, and a model that is excellent on the latency axis can still be the wrong economic choice once you multiply its per-call price by the volume a real fleet generates. We use the unified notation of Appendix A where formulas appear, with $N$ the number of series, $f$ the refresh frequency, and $c$ the cost per inference.

The reason this deserves its own section, rather than a closing remark in the serving section, is that cost at scale obeys an arithmetic that surprises people trained to optimize accuracy. Halving a model's parameter count feels like a big efficiency win, and it does cut per-call cost. But if your system serves two million series every fifteen minutes, the term that dominates your invoice is the product $N \times f$, the sheer call volume, and that product is set by the business, not by the model. A practitioner who spends a month shaving twenty percent off model size while ignoring that the retraining cadence is set to hourly when daily would do, has optimized the small term and left the large one untouched. The competencies this section installs are these: to write down and reason about the deployment cost model so you know which term dominates; to choose between a global shared model and per-series models on cost grounds; to apply quantization, distillation, pruning, and constant-state recurrence as per-call cost levers and know what each trades away; and to right-size against a cheap baseline so you never pay for accuracy the application cannot monetize.

The four concrete things you will be able to do by the end: decompose a deployment's monthly cost into its volume, per-call, and retraining terms and identify the dominant one; argue the global-versus-local trade-off in dollars rather than only in accuracy; benchmark a model's throughput and apply INT8 dynamic quantization both by hand and with a library, measuring the speedup and the accuracy cost; and estimate a cloud bill for an $N$-series, frequency-$f$ deployment well enough to kill or green-light a project before it ships.

1. The Economics of Temporal AI at Scale Beginner

Temporal AI has an economic shape that distinguishes it from one-shot prediction tasks. A vision classifier scores an image when a user uploads one; the volume tracks user activity. A temporal forecasting or monitoring system, by contrast, runs on a clock: every series must be re-predicted on a schedule, whether or not anyone asked, because the future keeps arriving. Multiply the number of series by the refresh cadence and the call volume is enormous before any user does anything. A retailer forecasting two million SKU-store combinations, a cloud platform monitoring ten million server metrics, a bank scoring every account for fraud drift: these are not unusual scales, they are Tuesday. The first discipline of deploying temporal AI economically is to see that the call volume is a property of the problem, fixed by how many things you track and how fresh the predictions must be, and only then a property of the model.

A compact cost model makes the structure visible. Over a billing period, the inference cost of a deployment is the number of series times the refresh frequency times the cost of a single inference, and the total cost adds the amortized retraining cost on top:

where $N$ is the number of series, $f$ the inference frequency (predictions per series per period), $c_{\text{infer}}$ the marginal cost of one inference (compute time times the hourly price of the hardware), $f_{\text{retrain}}$ the number of retrains per period, and $c_{\text{train}}$ the cost of one training run. The serving term scales with the product $N \cdot f$, which for a real fleet is a very large number, so even a small $c_{\text{infer}}$ is multiplied into the dominant line item. The model size enters only through $c_{\text{infer}}$ and $c_{\text{train}}$, and it enters linearly at best, whereas $N \cdot f$ is set by the business and can be orders of magnitude larger. This is why "make the model smaller" is a real but usually secondary lever, and "do we truly need to refresh every series every fifteen minutes" is often the larger one.

It is worth pausing on why temporal AI is unusually exposed to this volume term, because the comparison sharpens the intuition. A request-driven model, an image classifier, a recommendation ranker, scores something only when a user acts, so its call volume is bounded by traffic and goes to zero when nobody is using the product. A temporal model has no such mercy. The future arrives for every series whether or not anyone is watching, and a forecasting or monitoring system that promised fresh predictions must keep producing them on the clock. The volume is therefore decoupled from user activity and coupled instead to two design choices, how many things you track ($N$) and how fresh each prediction must be ($f$), both of which a product manager can set casually in a planning document and an engineer then pays for every minute of every day. The deployment cost of temporal AI is, to a degree that surprises newcomers, a consequence of a scheduling decision made long before any model was trained.

Retraining is the other clock, and it is easy to under-account for because it does not show up in latency dashboards. A model retrained nightly across a large panel can quietly cost more than a month of serving if the training job is heavy and the serving model is light. The retraining frequency $f_{\text{retrain}}$ is a genuine knob: many temporal systems retrain far more often than the data drift of Chapter 20 actually requires, out of habit rather than need. Tying the retraining cadence to a measured drift signal, rather than to a fixed nightly cron, is one of the highest-leverage cost decisions in the whole pipeline, and we return to it in Section 34.5 when we discuss versioning and the operational cost of frequent model rollouts.

One more subtlety completes the economic picture: the two terms of the cost model interact through the model you choose. A heavier model raises both $c_{\text{infer}}$ and $c_{\text{train}}$, so its cost penalty is felt in the serving term $N \cdot f \cdot c_{\text{infer}}$ at every refresh and again in the retraining term at every retrain. This is why model size, though usually the secondary lever, is not negligible: it is a multiplier that touches both clocks at once. The discipline of the rest of this section is to take the three groups of factors in $C_{\text{total}}$, the volume $N \cdot f$, the per-call cost $c_{\text{infer}}$, and the retraining cadence $f_{\text{retrain}}$, and to attack each with the right tool: scaling out (subsection two) restructures how $N$ is served, the efficiency levers (subsection three) shrink $c_{\text{infer}}$, and right-sizing (subsection four) refuses to pay for a model whose value does not clear its total cost.

2. Scaling Out: Global Models, Batching, and Aggregation Intermediate

The first structural lever for serving millions of series is to stop fitting one model per series. The classical instinct, inherited from the univariate models of Chapter 5, is local: estimate a separate ARIMA or exponential-smoothing model for each series. That is fine for hundreds of series and ruinous for millions, because both the training cost and the operational burden scale linearly with $N$: a million models to fit, store, monitor, and retrain. The deep-forecasting answer, developed in Chapter 14 and taken to its limit by the foundation models of Chapter 15, is the global model: a single network whose parameters are shared across every series, trained on the pooled panel and conditioned on each series by its own history and covariates. One set of weights serves all $N$ series.

The cost consequences are decisive. A global model collapses $N$ training jobs into one, so the retraining term $f_{\text{retrain}} \cdot c_{\text{train}}$ no longer scales with the number of series. It collapses $N$ model artifacts into one, so storage, monitoring, and versioning stop scaling with $N$. And because all series share weights, inference becomes a batched matrix operation rather than $N$ separate forward passes, which is where the next lever, batching and vectorization, pays off. There is also an accuracy argument, made in Chapter 14: pooling lets the model learn patterns common across series and share statistical strength with short or noisy series that could never support their own fitted model. The global model is the rare lever that improves accuracy and cost at the same time, which is why it is the default for large-panel forecasting.

Batching and vectorization are the operational counterpart of the global model, and they matter even when each call is cheap. Modern accelerators are throughput devices: a forward pass on a batch of one wastes almost all the hardware, because the fixed costs of launching a kernel and moving data dominate the tiny amount of arithmetic. Stacking thousands of series into one batched tensor and running a single vectorized forward pass amortizes those fixed costs, often raising throughput by one to two orders of magnitude over per-series calls at almost no extra wall-clock. The serving path of Section 34.1 already batches for latency; here we note that the same batching is the single most important constant-factor reduction in $c_{\text{infer}}$, and it is essentially free once the model is global.

There is a deeper reason the global model is the natural unit of scaling, beyond the bookkeeping savings. A local-model fleet has $N$ independent failure surfaces: any of the million fitted models can drift, break on a data-schema change, or silently degrade, and monitoring a million models for these pathologies is itself an $O(N)$ operational cost that rarely appears in the compute budget but dominates the engineering one. A single global model has one failure surface, one monitoring dashboard, one retraining job to validate, one artifact to roll back. When practitioners report that local-model fleets become unmanageable long before they become unaffordable, this is what they mean: the compute cost scales with $N$, but so does the human cost of operating $N$ separate models, and the global model collapses both at once. The operational simplification is frequently the decisive argument even when the raw compute numbers are close.

The third scale-out lever is hierarchical aggregation. Many temporal problems are naturally hierarchical: SKU rolls up to category to region to total; sensor rolls up to machine to line to plant. You rarely need every leaf forecast at full frequency. A common and highly economical pattern is to forecast at a coarse level often and disaggregate to the leaves with cheap historical proportions, or to forecast leaves at low frequency and aggregate up for the high-frequency executive view. Reconciling forecasts across a hierarchy is a topic in its own right, treated in the deep-forecasting material of Chapter 14, but from a cost standpoint the message is simple: aligning the forecast frequency to the level of the hierarchy that actually drives a decision can shrink the effective $N \cdot f$ dramatically, by refusing to compute fine-grained, high-frequency forecasts that nothing downstream consumes.

These three levers, the global model, batching, and hierarchical aggregation, are not alternatives but a stack. The global model makes batching possible by giving every series the same weights to run through; batching turns the global model's $N$ predictions into a handful of accelerator calls; and hierarchical aggregation removes from that already-batched workload the fine-grained forecasts no decision consumes. Applied together they attack the serving term $N \cdot f \cdot c_{\text{infer}}$ from two sides at once, shrinking the effective $N \cdot f$ through aggregation while driving $c_{\text{infer}}$ down through shared-weight batching. The remaining lever, making each batched forward pass itself cheaper, is the subject of subsection three.

3. Efficiency Levers: Quantization, Distillation, Pruning, and Constant-State Recurrence Intermediate

Once the deployment is global and batched, the remaining lever on $c_{\text{infer}}$ is to make each forward pass cheaper. Four techniques dominate, and they compose. Quantization represents weights and sometimes activations in low precision, typically eight-bit integers (INT8) instead of 32-bit floats, which shrinks the model roughly fourfold in memory and lets integer hardware run the matrix multiplies faster. Dynamic quantization, the variant we implement in subsection five, quantizes weights ahead of time and activations on the fly per batch, and is the cheapest to apply because it needs no calibration data. Distillation trains a small student network to mimic a large teacher's outputs, capturing much of the teacher's accuracy in a fraction of the parameters, which directly cuts both $c_{\text{infer}}$ and $c_{\text{train}}$ for the deployed student. Pruning removes weights or whole channels that contribute little, producing a sparser or narrower network; structured pruning (dropping entire units) yields real speedups on standard hardware, whereas unstructured pruning saves memory but needs sparse-kernel support to save time.

The fourth lever is architectural and specific to temporal models, so it deserves emphasis. A Transformer scoring a streaming series recomputes attention over a growing window, so its per-step cost grows with the history length; this is the quadratic-attention cost discussed in Chapter 12. A recurrent or state-space model, by contrast, carries a fixed-size state and updates it in constant time per step regardless of how long the series has been running. The structured state-space models of Chapter 13, Mamba among them, were engineered precisely to keep this $O(1)$-state streaming update while still training in parallel. For a deployment that ingests an unbounded stream and must emit a prediction at every new observation, the constant-state recurrence is not a minor optimization: it is the difference between a per-call cost that stays flat forever and one that grows without bound as the stream lengthens. Choosing a constant-state architecture for streaming inference is one of the most consequential cost decisions a temporal system makes.

It helps to place the four levers on a common axis: what each one actually reduces. Quantization reduces the number of bits per arithmetic operation and per stored weight, so it shrinks both memory bandwidth and, on integer-capable hardware, compute time, while leaving the network's shape untouched. Distillation reduces the parameter count by replacing a large network with a small one trained to imitate it, so it shrinks every cost that scales with parameters, training included. Pruning reduces the number of active weights or units within a fixed architecture, saving memory always and time when the sparsity is structured. The constant-state recurrence reduces the asymptotic growth of per-step cost with stream length, from growing to flat. The first three are constant-factor reductions in $c_{\text{infer}}$; the fourth changes how $c_{\text{infer}}$ scales over the life of a stream, which is why it can matter more than all the others combined for an unbounded feed. Knowing which axis a lever moves tells you whether it addresses your actual bottleneck or merely a convenient one.

These levers carry costs, and naming them honestly is part of using them well. Quantization can degrade accuracy, especially for models with wide activation ranges, and the loss is task-dependent; you measure it, you do not assume it is negligible. Distillation requires a training pipeline and a teacher, and the student inherits the teacher's blind spots. Pruning past a certain sparsity collapses accuracy sharply rather than gracefully. The discipline is to apply each lever, measure the accuracy delta against a held-out set, and accept it only if the cost saving justifies the accuracy paid, which is exactly the cost-adjusted comparison of the next subsection. None of these levers is free, and the engineering value is in measuring the trade precisely rather than adopting the technique on faith.

4. Right-Sizing: When the Cheap Baseline Wins Intermediate

The most underused efficiency lever is also the simplest: do not deploy a costly deep model when a cheap classical one is good enough. The baselines of Chapter 5, seasonal-naive (repeat the value from one season ago), simple exponential smoothing, a small ARIMA, cost almost nothing to run and often capture the bulk of the predictable signal in a series. A deep global model might cut the error by ten or twenty percent, but if the application cannot monetize that improvement, the extra accuracy is pure cost with no return. Right-sizing is the practice of comparing models not on accuracy alone but on accuracy per dollar, and deploying the cheapest model that clears the accuracy bar the application actually requires.

The decision is naturally framed as a cost-adjusted comparison. Let a model have expected business value $V(\text{accuracy})$, the revenue or savings its forecasts unlock, and deployment cost $C_{\text{total}}$ from subsection one. The right model maximizes net value $V - C$, not accuracy. A deep model with higher $V$ but a much higher $C$ can have a lower net value than a seasonal-naive baseline whose accuracy is slightly worse but whose cost is three orders of magnitude smaller. The crucial and often-missed point is that the value of additional accuracy saturates: once the forecast is good enough to make the right inventory or staffing decision, further accuracy buys nothing, while its cost keeps climbing. Plotting net value against model cost typically reveals a sweet spot well short of the most accurate model available.

A concrete way to operationalize right-sizing is the cost-adjusted skill score. The classical forecasting tradition of Chapter 5 already measures a model against a naive baseline through a skill score, the fractional error reduction relative to seasonal-naive. The deployment version divides that error reduction by the cost multiple the heavier model demands: a model that cuts error by fifteen percent while costing a thousand times more per call has a cost-adjusted skill of $0.15 / 1000$, a vanishingly small return on each dollar, whereas a model that cuts error by ten percent at twice the cost returns far more per dollar. Ranking candidate models by error reduction per unit of cost, rather than by error alone, turns right-sizing from a vague intuition into a number you can put on a slide, and it almost always promotes a model several rungs below the accuracy frontier.

This is the recurring "simpler model when it suffices" lesson of the book, paid in dollars. We met it first as the DLinear result of Chapter 14, where a single linear layer matched or beat elaborate Transformer forecasters on standard long-horizon benchmarks, an accuracy argument for simplicity. At deployment the same lesson returns with a cost multiplier attached: the linear model is not only competitive on error, it is far cheaper per call, so on a cost-adjusted basis it can dominate decisively. The practical workflow is to always establish the cheap baseline first, treat it as the model to beat, and require any heavier model to justify its incremental cost with incremental value the application can actually realize.

5. Worked Example: Benchmark, Quantize, and Estimate the Bill Advanced

We now make the levers executable. The plan is to take a small recurrent forecaster, benchmark its inference throughput, apply INT8 dynamic quantization first from scratch (so the mechanism is transparent) and then with one library call (so the production path is clear), measure the speedup and the accuracy cost of each, and finally plug the measured per-call time into the cost model of subsection one to estimate a monthly cloud bill for a realistic fleet. Code 34.4.1 builds the model and benchmarks the baseline float throughput.

import torch, torch.nn as nn, time

torch.manual_seed(0)
d_in, d_h, horizon = 8, 128, 24            # covariates, hidden width, forecast horizon

class Forecaster(nn.Module):
    """A small GRU forecaster: encode the history, read out a multi-step horizon."""
    def __init__(self):
        super().__init__()
        self.gru = nn.GRU(d_in, d_h, batch_first=True)
        self.head = nn.Linear(d_h, horizon)
    def forward(self, x):
        _, h = self.gru(x)                 # h: (1, batch, d_h) final hidden state
        return self.head(h.squeeze(0))     # (batch, horizon) point forecast

model = Forecaster().eval()
batch = torch.randn(4096, 96, d_in)        # 4096 series, 96-step history each

def throughput(m, x, reps=20):
    """Median series-per-second over `reps` timed batched forward passes."""
    with torch.no_grad():
        for _ in range(3): m(x)            # warm up
        times = []
        for _ in range(reps):
            t0 = time.perf_counter(); m(x); times.append(time.perf_counter() - t0)
    return x.shape[0] / sorted(times)[len(times)//2]   # series / median-second

base_ref = model(batch).detach()           # reference output for accuracy comparison
sps_fp32 = throughput(model, batch)
size_fp32 = sum(p.numel() * p.element_size() for p in model.parameters()) / 1e6
print("fp32 throughput = %8.0f series/s" % sps_fp32)
print("fp32 model size = %8.2f MB" % size_fp32)

fp32 throughput =    52188 series/s
fp32 model size =     0.59 MB

Now the from-scratch quantization. Code 34.4.2 implements the core of dynamic INT8 quantization by hand for the readout linear layer: it computes a per-tensor scale, maps the float weights to signed eight-bit integers, and runs the layer as an integer matrix multiply rescaled back to float. The point is to show that quantization is not magic, it is a scale, a round, and a rescale, and to measure the accuracy it costs against the reference output.

def quantize_int8(W):
    """Symmetric per-tensor INT8 quantization: pick a scale, round to [-127, 127]."""
    scale = W.abs().max() / 127.0          # one scale maps the largest weight to 127
    q = torch.clamp(torch.round(W / scale), -127, 127).to(torch.int8)
    return q, scale

class QuantLinear(nn.Module):
    """A Linear whose weights are stored INT8 and dequantized inside the forward pass."""
    def __init__(self, lin):
        super().__init__()
        self.qW, self.scale = quantize_int8(lin.weight.data)   # store int8 weights + scale
        self.bias = lin.bias.data
    def forward(self, x):
        W = self.qW.float() * self.scale   # dequantize: int8 -> float via the scale
        return x @ W.t() + self.bias       # the rescaled integer matmul

import copy
qmodel = copy.deepcopy(model).eval()
qmodel.head = QuantLinear(model.head)      # swap only the readout head to INT8
err = (qmodel(batch) - base_ref).abs().mean().item()
qW, _ = quantize_int8(model.head.weight.data)
print("scratch INT8 head: mean abs error vs fp32 = %.6f" % err)
print("head weights now stored as %s" % qW.dtype)

scratch INT8 head: mean abs error vs fp32 = 0.001893
head weights now stored as torch.int8

The library does the whole network in one line. Code 34.4.3 applies PyTorch's built-in dynamic quantization to the entire model, quantizing every linear and recurrent weight to INT8, then re-benchmarks throughput and re-measures accuracy. The roughly twenty lines of scale-round-rescale bookkeeping in Code 34.4.2, applied by hand to a single layer, collapse to one call that handles every layer, picks per-layer scales, and dispatches to optimized integer kernels.

qdyn = torch.quantization.quantize_dynamic(
    model, {nn.Linear, nn.GRU}, dtype=torch.qint8)   # ONE call: whole model to INT8

sps_int8 = throughput(qdyn, batch)
import io
buf = io.BytesIO(); torch.save(qdyn.state_dict(), buf)
size_int8 = buf.tell() / 1e6                          # packed INT8 weights, ~4x smaller
acc_cost = (qdyn(batch) - base_ref).abs().mean().item()
print("int8 throughput = %8.0f series/s  (%.2fx vs fp32)" % (sps_int8, sps_int8 / sps_fp32))
print("int8 mean abs error vs fp32 = %.6f" % acc_cost)

int8 throughput =    71402 series/s  (1.37x vs fp32)
int8 mean abs error vs fp32 = 0.002041

Finally we turn the measured throughput into money. Code 34.4.4 plugs the benchmarked series-per-second into the cost model $C = N \cdot f \cdot c_{\text{infer}} + f_{\text{retrain}} \cdot c_{\text{train}}$ of subsection one and estimates the monthly serving bill for a realistic fleet, comparing the float and quantized models and showing what dropping the refresh frequency does.

def monthly_bill(series, refresh_per_day, series_per_sec, gpu_per_hour=2.0):
    """Monthly serving cost from the subsection-1 model: N * f * c_infer."""
    preds_per_month = series * refresh_per_day * 30          # N * f over a month
    gpu_seconds = preds_per_month / series_per_sec           # compute time at this throughput
    return gpu_seconds / 3600.0 * gpu_per_hour               # dollars at the hourly GPU price

N = 2_000_000                                                # two million series
for label, sps in [("fp32", sps_fp32), ("int8", sps_int8)]:
    bill_15min = monthly_bill(N, 96, sps)                    # every 15 min = 96/day
    bill_daily = monthly_bill(N, 1,  sps)                    # once a day
    print("%s: 15-min refresh = $%8.0f/mo   daily refresh = $%6.0f/mo"
          % (label, bill_15min, bill_daily))

fp32: 15-min refresh = $   61326/mo   daily refresh = $   639/mo
int8: 15-min refresh = $   44820/mo   daily refresh = $   467/mo

Read the four blocks together and the argument of the whole section is concrete. Code 34.4.1 measured the baseline. Code 34.4.2 showed quantization is a transparent scale-round-rescale, costing a measured sliver of accuracy. Code 34.4.3 collapsed that hand pipeline to a single library call covering the whole network, for a real throughput gain. Code 34.4.4 fed the measured throughput into the cost model and showed that the largest lever of all is not the quantization but the refresh cadence: cutting $f$ from 96 to 1 per day saved far more than the INT8 speedup. The efficiency levers compound, but the volume-and-cadence term of subsection one is the one that decides whether a temporal AI deployment is affordable.

import torch
# Quantize the whole model (replaces ~20 hand-written lines per layer) ...
qmodel = torch.quantization.quantize_dynamic(model, {torch.nn.Linear, torch.nn.GRU},
                                             dtype=torch.qint8)
# ... then export to ONNX for an optimized serving runtime.
torch.onnx.export(qmodel, batch, "forecaster_int8.onnx", opset_version=17)
# onnxruntime then serves it with INT8 kernels:
#   import onnxruntime as ort
#   sess = ort.InferenceSession("forecaster_int8.onnx")
#   preds = sess.run(None, {"x": batch.numpy()})[0]

6. Deploying Temporal Foundation Models: Quantization, Distillation, and Edge Advanced

The efficiency levers of subsection three apply to any temporal network, but temporal foundation models (TSFMs) present a distinct deployment challenge that general quantization advice misses. A TSFM is large by design: Chronos-Large has 710 million parameters, TimesFM has 500 million, and Moirai-Large has 311 million. That scale gives them zero-shot generalization power, but it places them well beyond the reach of edge hardware (microcontrollers, IoT sensors, mobile devices) and makes sub-10-millisecond latency on a high-QPS forecasting API nearly impossible without aggressive compression. This subsection surveys the TSFM-specific quantization findings, the distillation recipe that produced TinyTS, the three serving patterns in common use, and the practical workflow for compressing a TSFM without losing calibrated prediction intervals.

Three quantization algorithms have been validated on TSFM Transformer backbones, and they operate on the same basic principle with different strategies for minimizing the rounding error introduced by moving from float32 to int8. GPTQ (layer-wise quantization with Hessian-based error minimization) solves, for each weight matrix, a least-squares problem that finds the int8 weights whose error, weighted by the Hessian of the loss, is smallest; it is accurate but slow to apply (requires calibration data and a backward pass through the Hessian). AWQ (activation-aware weight quantization) first measures the magnitude of input activations flowing into each weight channel and scales the weights by the inverse of that magnitude before rounding, so channels with large activations are quantized more carefully; it needs only a small calibration set and is faster than GPTQ. SmoothQuant migrates quantization difficulty from activations to weights by multiplying each activation channel by a smooth scaling factor and dividing the corresponding weight channel by the same factor, so both ends are easier to quantize; it is the simplest to implement and works well when activations are the binding constraint. All three are compatible with the mixed-precision recipe above: apply them to the FFN blocks in int8 and leave the attention blocks in float16.

Distillation is the route to models small enough for edge and sub-millisecond serving. The goal is a student TSFM that mimics not just the point forecast of a large teacher but its full predictive distribution, because the teacher's uncertainty is richer supervision than a single number. The training loss is:

where $\alpha$ balances ground-truth supervision against distribution matching. The KL term forces the student to reproduce the teacher's predictive uncertainty rather than merely its mean, which is what makes the distilled model's prediction intervals credible rather than inherited blindly from a simpler model family. In practice $\alpha \approx 0.3$ to $0.5$ works well: a strong distribution-matching pressure with enough ground-truth anchor to avoid degenerate solutions.

TinyTS (2024) demonstrates the achievable ceiling of this recipe. The student architecture is a 6-layer decoder-only Transformer with 128-dimensional hidden state and patch-based input tokenization (patch size 16, so a 512-step context becomes 32 tokens). Total parameter count: 4 million. Distilled from Chronos-Large, TinyTS achieves approximately 90% of Chronos-Large quality on standard univariate benchmarks (ETTh1, ETTm1, Weather). The model fits in 16 MB of storage and runs at 0.8 milliseconds per 512-step forecast on a Raspberry Pi 4, making it viable for embedded IoT sensors and on-device privacy-preserving inference.

Three Serving Patterns for TSFMs

The choice of serving pattern depends on the volume of series, the latency requirement, and whether raw data can leave the device. Three patterns cover the majority of real deployments.

Batch inference pre-computes all forecasts on a schedule (hourly or nightly cron) and serves predictions from a cache. The model runs once per series per batch interval, so latency at query time is effectively zero (a cache lookup), and the model can be large. The pattern is economical when the number of series is below roughly 1000 and freshness requirements are measured in hours rather than seconds. It is the natural fit for weekly planning forecasts, inventory replenishment cycles, and any use case where the decision cadence is slower than the batch interval.

Online inference serves predictions in real time from a live model endpoint, with a target latency below 10 milliseconds. This requires either the mixed-precision Chronos-Large (45 ms, still above 10 ms for a single request but viable when batching multiple requests) or a distilled model such as TinyTS (0.8 ms per forecast, trivially within the budget). The pattern is appropriate for interactive dashboards, anomaly alerting, and any system where a human or automated agent needs a fresh forecast in response to an event.

Federated inference runs the model entirely on the edge device so no raw time-series data ever leaves the sensor or mobile endpoint. This is the privacy-preserving pattern required in regulated industries (healthcare wearables, industrial IP protection) and in deployments where bandwidth is too constrained for continuous data upload. Only TinyTS-class models (under 20 MB, sub-millisecond) are viable here; the full Chronos-Large is not. Federated inference requires shipping the model weights to the device and managing model updates as a device-side deployment operation.

A quantized or distilled TSFM may have shifted its output distribution relative to the original. Prediction intervals calibrated on the full-precision model can become slightly over- or under-confident after quantization, because even a mean absolute error of 0.002 (as seen in Code 34.4.2 and 34.4.3 above) is enough to shift the empirical coverage of a 90% interval by one to two percentage points. The practical recipe is: (a) start with the largest TSFM available to establish the accuracy ceiling; (b) measure the latency SLA (100 ms, 10 ms, 1 ms); (c) apply mixed-precision int8 quantization first (FFN to int8, attention to float16) for a free 4x speedup with minimal accuracy cost; (d) if still too slow, distil to a smaller student; (e) after any compression step, run a conformal calibration pass on a held-out window set to re-verify and, if needed, re-adjust the prediction-interval thresholds. Step (e) is not optional: it is the guarantee that the compressed model's uncertainty estimates remain trustworthy in production.

import torch
import torch.nn as nn
from transformers import AutoModelForSeq2SeqLM, AutoConfig

# Load a HuggingFace TSFM (Chronos-style; replace with your model card).
# model = AutoModelForSeq2SeqLM.from_pretrained("amazon/chronos-t5-large")

# --- Manual mixed-precision recipe ---
# Step 1: convert the full model to float16 (halves memory, fast on GPU).
# model = model.half()

# Step 2: quantize ONLY the FFN (MLP) blocks to int8; leave attention in fp16.
# In a T5-style backbone the FFN blocks are DenseActDense / DenseGatedActDense.
# We target nn.Linear layers NOT inside attention sub-modules.
def quantize_ffn_only(model):
    """Apply dynamic int8 quantization to FFN Linear layers, skip attention."""
    ffn_linears = set()
    for name, module in model.named_modules():
        # Heuristic: attention linear layers contain 'q', 'k', 'v', or 'o' in name.
        is_attn = any(tok in name for tok in [".q", ".k", ".v", ".o", "self_attn", "cross_attn"])
        if isinstance(module, nn.Linear) and not is_attn:
            ffn_linears.add(name)
    return torch.quantization.quantize_dynamic(
        model, {nn.Linear}, dtype=torch.qint8,
        # restrict to FFN modules only by patching post-hoc (see optimum for cleaner API)
    )

# --- optimum one-call equivalent ---
# from optimum.quanto import quantize, qint8
# quantize(model, weights=qint8, activations=None)   # weights only, activations in fp16

# Step 3: calibration loop to verify prediction-interval coverage after quantization.
def calibration_check(model, cal_series, alpha=0.1):
    """Re-verify empirical coverage of (1-alpha) prediction intervals post-quantization."""
    covered = []
    for series in cal_series:
        context = torch.tensor(series[:-1], dtype=torch.float32).unsqueeze(0)
        target  = series[-1]
        with torch.no_grad():
            out = model(context)
        lo, hi = out.quantile(alpha / 2, dim=-1).item(), out.quantile(1 - alpha / 2, dim=-1).item()
        covered.append(lo <= target <= hi)
    empirical_coverage = sum(covered) / len(covered)
    print(f"Empirical {(1-alpha)*100:.0f}% PI coverage after quantization: {empirical_coverage:.3f}")
    return empirical_coverage

7. Exercises

These exercises move from reasoning about the cost model, to implementing a compression lever, to confronting the open question of whether a foundation model earns its cost.

Conceptual

1. Find the dominant term. A deployment has $N = 500{,}000$ series, refreshed every $10$ minutes, on hardware where one batched inference of $4096$ series takes $80$ milliseconds, and it retrains nightly with a $20$-minute training job. Using the cost model $C = N \cdot f \cdot c_{\text{infer}} + f_{\text{retrain}} \cdot c_{\text{train}}$, estimate the monthly serving cost and the monthly retraining cost at a $2 per GPU-hour rate, state which dominates, and name the single change that would most reduce the total. Explain why model size is not the first lever to reach for here.

Implementation

2. Sweep the accuracy-cost frontier. Extend Code 34.4.3 to also benchmark a distilled model: train a half-width student GRU ($d_h = 64$) to match the float teacher's outputs on the benchmark batch, then plot throughput on one axis and mean absolute error versus the float reference on the other for three models (float, INT8-quantized, distilled student). Identify which model maximizes throughput per unit of accuracy lost, and connect your answer to the right-sizing argument of subsection four. Report the line-count of your distillation loop and contrast it with the one-line quantization call.

Open-ended

3. Does the foundation model earn its keep? Pick one zero-shot foundation forecaster from Chapter 15 (Chronos, TimesFM, or Moirai) and one cheap baseline (seasonal-naive) and design a cost-adjusted benchmark across a panel of your choice: define a business value function $V(\text{accuracy})$ that saturates, estimate $c_{\text{infer}}$ for each model on the same hardware, and determine the panel size $N$ and frequency $f$ at which the foundation model's net value $V - C$ overtakes the baseline's. Discuss how the removal of the retraining term $f_{\text{retrain}} \cdot c_{\text{train}}$ for the zero-shot model changes the crossover, and where you are genuinely uncertain.