Decision-Aware Evaluation of Forecasts

"They praised my RMSE for a year. Then the warehouse flooded with the stock I told them to buy, and nobody asked about my RMSE again. I had been optimizing the wrong number the whole time, beautifully."

A Forecast Graded Not on Accuracy but on the Decisions It Enabled

Across this chapter we have built the machinery of probabilistic forecasting: predictive distributions in Section 19.1, proper scoring and calibration in Section 19.2, quantile and distributional forecasts in Section 19.3, Bayesian, ensemble and Gaussian-process uncertainty in Section 19.4, and conformal prediction intervals in Section 19.5. Every one of those tools produces a richer object than a single number: a full description of what the future might be and how confident we are. This closing section asks the question that justifies all that effort. What is the distribution for? The answer returns us to the founding thesis of the book, stated in Section 1.3: prediction exists to drive a decision. A forecast that no one acts on has no value, and a forecast that drives a good action has value exactly equal to the action's payoff. The right metric, therefore, lives downstream of the forecast, in the cost or reward of the action it enabled, and this section teaches how to evaluate, and ultimately how to train, against that downstream metric.

Why does this deserve its own section rather than a footnote to the calibration material? Because the entire incentive structure of a forecasting team bends toward whatever number is on the dashboard, and for decades that number has been a statistical accuracy metric (RMSE, MAE, MAPE) computed on the forecast in isolation, blind to the decision it serves. That choice is not neutral. A point forecast minimizing squared error targets the conditional mean, but almost no real decision is optimal at the mean: the moment over-action and under-action cost differently, the optimal action shifts to a quantile, and a system tuned to the mean is systematically mis-stocked, mis-staffed, or mis-positioned. The gap between statistical accuracy and decision value is not a rounding error; it is a structural bias that grows with the cost asymmetry of the problem. This section names that gap, shows the quantile that closes it, and gives you the evaluation discipline that keeps a forecasting system honest about what it is actually for.

The competencies installed here are four. To articulate, with a concrete loss, why downstream decision cost and not RMSE is the correct objective for an operational forecast. To derive the critical fractile of the newsvendor problem and read off the optimal order as a quantile of the predictive distribution. To distinguish predict-then-optimize from decision-focused learning, and to say when training on the decision loss pays. And to backtest a forecasting system by replaying its decisions and accumulating their realized cost, using calibrated intervals as risk limits. These are the skills that turn the probabilistic forecasts of this chapter into operational value, and they carry directly into the sequential decision making of Part VI, where the action feeds back into the next observation.

1. The Gap Between Statistical Accuracy and Decision Value Beginner

Begin with the uncomfortable fact that the two most natural ways to grade a forecast disagree. Statistical accuracy asks "how close was the predicted number to the realized number?" and answers with a symmetric penalty: RMSE squares the error, MAE takes its absolute value, and both treat an over-prediction of five units exactly like an under-prediction of five units. Decision value asks a different question entirely: "given the action this forecast triggered, what did that action cost or earn once reality arrived?" and the answer is almost never symmetric, because the consequences of acting too big and acting too small are almost never the same.

Consider a retailer forecasting tomorrow's demand for a perishable item. A point forecast minimizing squared error returns the conditional mean of demand. The retailer must turn that into an order quantity. If they order the mean and demand comes in low, the unsold units spoil and the loss is the purchase cost of the overage. If they order the mean and demand comes in high, they stock out and the loss is the lost margin of every customer turned away. When spoilage is cheap and lost margin is dear (a high-margin item), under-ordering hurts far more than over-ordering, and the cost-minimizing order is well above the mean. When spoilage is dear and margin thin, the reverse holds and the optimal order sits below the mean. In neither case is the conditional mean, the quantity squared error was so carefully tuned to deliver, the right thing to order. The forecast with the best RMSE can yield the worst realized cost.

This is the prediction-to-decision thesis of Section 1.3 made quantitative. There we argued that temporal intelligence is ultimately about choosing actions over time, and that prediction is the servant of decision, not its master. Here we see the servant relationship as a concrete misalignment of objectives: minimizing a symmetric forecasting loss optimizes a target (the mean) that an asymmetric decision loss does not want. The fix is not to abandon probabilistic forecasting; it is to recognize that the predictive distribution carries exactly the information the decision needs, and that the decision, not the forecaster, should choose which feature of that distribution to act on.

The deeper structural point is that a forecasting metric is a proxy, and a proxy is only as good as its alignment with the true objective. RMSE is a convenient, model-agnostic, differentiable proxy, which is exactly why it became the default; but convenience is not correctness. The true objective is the expected decision cost, a quantity that depends on the loss structure of the specific decision, on the full predictive distribution rather than a point, and on the realized outcome. The remainder of this section replaces the proxy with the real thing, first analytically for the canonical newsvendor decision, then as an evaluation protocol, then in code.

2. The Newsvendor Decision and the Critical Fractile Intermediate

The newsvendor problem is the simplest decision that exposes the whole story, and it is the right anchor because its solution is a closed-form quantile of the predictive distribution. A vendor must choose an order quantity $q$ before observing a random demand $D$ with predictive distribution $F$ (the cumulative distribution function our probabilistic forecaster outputs for the next period). Two unit costs encode the asymmetry: an underage cost $c_u$ paid per unit of demand left unmet (lost margin, the cost of ordering too little), and an overage cost $c_o$ paid per unit ordered but not sold (spoilage or holding, the cost of ordering too much). The realized cost of choosing $q$ when demand turns out to be $D$ is the piecewise-linear function

which charges $c_u$ for every unit of shortfall and $c_o$ for every unit of leftover. This is precisely the asymmetric, pinball-shaped loss we met in Section 19.3: the newsvendor cost is the quantile loss, scaled, and that is no coincidence. The vendor does not know $D$, so they minimize the expected cost under the predictive distribution,

Differentiating the expected cost with respect to $q$ and setting it to zero gives the classical first-order condition. The derivative of $\mathbb{E}[\max(q - D, 0)]$ with respect to $q$ is $F(q)$ (the probability $D \le q$), and the derivative of $\mathbb{E}[\max(D - q, 0)]$ is $-(1 - F(q))$, so

The right-hand side is the famous critical fractile, and the optimal order is the quantile of the predictive distribution at that probability level:

Read this equation slowly, because it carries the entire thesis of the section in one line. The decision is a quantile of the predictive distribution. The quantile level is fixed not by the data but by the cost asymmetry: when underage hurts more ($c_u \gg c_o$) the fractile is near one and the vendor orders a high quantile (stocks generously against stockouts); when overage hurts more ($c_o \gg c_u$) the fractile is near zero and the vendor orders a low quantile (stocks conservatively against spoilage); when the costs are equal the fractile is $1/2$ and the optimal order is the median, recovering the symmetric case. A point forecast cannot express any of this, because it has collapsed the distribution to a single number before the decision's asymmetry was ever consulted. A probabilistic forecast plus the loss yields the decision directly, with no extra modelling, by reading off one quantile.

The newsvendor is a single-period, single-product cartoon, but its lesson generalizes to a wide class of operational decisions. Staffing a call center against uncertain call volume is a newsvendor in disguise (overstaffing wastes wages, understaffing breaches service levels). Provisioning cloud capacity against uncertain load is a newsvendor (idle instances cost money, saturation drops requests). Setting an inventory reorder point, sizing a hedge, staging a maintenance crew: each pairs an asymmetric cost with an uncertain quantity, and each is solved by pushing the predictive distribution through the loss to extract the cost-minimizing action. Whenever the action is scalar and the cost is piecewise-linear, that action is a quantile and the fractile is the cost ratio. The probabilistic forecast is the reusable asset; the loss is the decision-specific lens that turns it into an action.

3. Predict-Then-Optimize versus Decision-Focused Learning Intermediate

There are two architectures for getting from data to a decision, and the difference between them is the most consequential design choice in decision-aware forecasting. The dominant one is predict-then-optimize (a two-stage pipeline): first train a forecaster to predict the uncertain quantity as accurately as possible (minimizing a statistical loss like squared error or pinball loss), then, at decision time, feed its prediction into an optimization that minimizes the decision cost. The two stages are trained and tuned independently; the forecaster never sees the decision loss. This is what most teams ship, because it is modular, the forecaster is reusable across decisions, and each stage uses mature tooling.

The problem with the two-stage pipeline is the seam between the stages. The forecaster is trained to minimize prediction error uniformly across the distribution, but the downstream optimization is sensitive to prediction error only in the region that changes the decision, and unevenly so. A small forecasting error that happens to straddle the critical fractile flips the order quantity and incurs real cost; a large forecasting error in a region the decision ignores costs nothing. A forecaster minimizing average squared error spends its capacity making the whole distribution accurate, including the parts the decision never reads, and may sacrifice accuracy exactly where the decision is sensitive. The two-stage objective and the true objective are different functions, and optimizing the first does not optimize the second.

Decision-focused learning, also called smart predict-then-optimize (SPO) after Elmachtoub and Grigas (2022), closes the seam by training the forecaster on the decision loss directly: the loss function is the realized decision regret, $C(q(\hat{F}), D) - C(q^\star, D)$, the extra cost incurred because the model's induced action $q(\hat{F})$ differs from the action a clairvoyant would have taken. The model is rewarded for predictions that lead to good decisions, not for predictions that are numerically close, and it is allowed to be inaccurate where inaccuracy is harmless. The technical obstacle is that the decision is an $\arg\min$, which is piecewise-constant and has zero or undefined gradient almost everywhere, so the decision loss cannot be backpropagated naively. The SPO+ surrogate of Elmachtoub and Grigas supplies a convex, subdifferentiable upper bound on the regret for linear optimization, and a parallel line (differentiable optimization layers, OptNet of Amos and Kolter, 2017, and the cvxpylayers of Agrawal and colleagues, 2019) differentiates through the optimization's KKT conditions so the whole pipeline trains end to end by gradient descent.

A pragmatic middle path, and the one this section recommends as a default, keeps the modular two-stage structure but fixes the seam at the point where it matters most: train a probabilistic forecaster (or a quantile forecaster at the relevant fractile, as in Section 19.3) rather than a mean forecaster, and let the decision read the correct quantile. This captures most of the decision-value gain of full decision-focused learning, because the action now uses the right feature of the distribution, while preserving the reusability and tooling maturity of the two-stage pipeline. Full end-to-end decision-focused training is the right reach when the optimization is complex (multi-product, constrained, combinatorial) and the cost structure does not reduce to a single quantile.

4. Evaluation in Practice: Backtest the Decisions, Not the Forecasts Advanced

If decision value is the objective, then evaluation must measure decision value, and that requires a backtest of the decisions and their realized cost, not of the forecasts in isolation. The protocol mirrors the rolling-origin forecast backtest of earlier chapters but adds a step: at each evaluation time the forecaster emits its predictive distribution, the decision rule turns that distribution into an action (the critical-fractile quantile for a newsvendor), the realized outcome arrives, and the realized cost $C(q, D)$ of that action under that outcome is recorded. Accumulating realized cost across the test period, and comparing it against the cost of a baseline policy (typically the point-forecast policy that orders the mean), is the decision-aware analogue of comparing RMSE against a naive baseline. The headline number is money, or regret against the clairvoyant optimum, not error.

This reframing changes what a "better forecast" means in ways that routinely surprise teams migrating from accuracy metrics. A forecaster with slightly worse RMSE can produce strictly lower realized decision cost if its errors are better aligned with the decision's asymmetry (for instance, if it is well calibrated in the tail the critical fractile reads). Conversely, a forecaster that wins on RMSE can lose on realized cost if its central accuracy comes at the price of a miscalibrated tail. The only way to see this is to score the decisions. The forecast accuracy metrics remain useful as diagnostics (they help you understand why a policy costs what it does), but they are demoted from the objective to the dashboard.

Calibrated intervals, from the calibration discipline of Section 19.2 and the conformal guarantee of Section 19.5, earn a second role here beyond honesty about uncertainty: they function as risk limits. A decision-aware system often carries not just an expected-cost objective but a constraint, a service-level target (meet demand on at least $95\%$ of days), a value-at-risk limit, a maximum acceptable stockout probability. These constraints are statements about quantiles of the predictive distribution, and they bind only if the intervals are calibrated: a $95\%$ interval that actually covers $80\%$ of the time will silently violate the service level it was meant to guarantee. So calibration is not a statistical nicety in a decision-aware system; it is the property that makes the risk limit a real limit. An over-confident forecaster that reports tight, under-covering intervals will produce decisions that look cheap in expectation and blow through their risk constraints in the tail, the exact failure mode that ends careers in trading and grid operations.

5. Worked Example: A Newsvendor Driven by a Probabilistic Forecast Advanced

We now make the whole argument executable. The plan is to simulate a demand series, fit a probabilistic forecaster, and then compare two policies head to head on realized cost: a point-forecast policy that orders the predictive mean, and a decision-aware policy that orders the critical-fractile quantile. We compute the critical fractile from explicit margins, implement the newsvendor cost and the quantile order from scratch, accumulate realized cost over a backtest, and then show the same optimal order obtained from a library optimizer in a few lines. Code 19.6.1 sets up the problem and the from-scratch decision machinery.

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(7)

# --- A demand series with seasonality + noise (one product, daily) ---
T = 400
day = np.arange(T)
mean_demand = 200 + 30 * np.sin(2 * np.pi * day / 7)    # weekly seasonality
demand = np.maximum(0, rng.normal(mean_demand, 40)).round()  # noisy realized demand

# --- Economics of the decision (the asymmetry that selects the quantile) ---
buy, sell = 1.50, 4.00
c_o = buy                      # overage: unsold unit is discarded -> lose purchase price
c_u = sell - buy               # underage: stockout -> lose the margin
crit_fractile = c_u / (c_u + c_o)
print("c_u = %.2f, c_o = %.2f, critical fractile = %.4f" % (c_u, c_o, crit_fractile))

def newsvendor_cost(q, d):
    """Realized cost of ordering q when demand turns out to be d."""
    return c_u * max(d - q, 0.0) + c_o * max(q - d, 0.0)

def optimal_order_from_normal(mu, sigma, fractile):
    """From scratch: the critical-fractile quantile of a Normal predictive dist."""
    return mu + sigma * norm.ppf(fractile)      # F^{-1}(fractile) for N(mu, sigma)

c_u = 2.50, c_o = 1.50, critical fractile = 0.6250

With the decision machinery in place, Code 19.6.2 runs the rolling backtest. At each test day a simple probabilistic forecaster predicts next-day demand as a Normal whose mean is the recent local average and whose standard deviation is the recent residual spread; the point-forecast policy orders that mean, the decision-aware policy orders the critical-fractile quantile, and we accumulate the realized newsvendor cost of each under the demand that actually arrived.

def predictive_normal(history, window=21):
    """A minimal probabilistic forecaster: Normal(mu, sigma) from recent history."""
    recent = history[-window:]
    return recent.mean(), recent.std(ddof=1)

start = 30                                  # warm-up before the backtest begins
cost_point, cost_aware = 0.0, 0.0
for t in range(start, T):
    mu, sigma = predictive_normal(demand[:t])     # forecast made BEFORE seeing demand[t]
    q_point = mu                                   # point policy: order the mean
    q_aware = optimal_order_from_normal(mu, sigma, crit_fractile)  # critical fractile
    d = demand[t]                                  # realized demand for the day
    cost_point += newsvendor_cost(q_point, d)
    cost_aware += newsvendor_cost(q_aware, d)

n = T - start
print("point-forecast policy : total cost = %8.1f   (%.3f per day)" % (cost_point, cost_point / n))
print("decision-aware policy : total cost = %8.1f   (%.3f per day)" % (cost_aware, cost_aware / n))
print("realized cost reduction: %.1f%%" % (100 * (cost_point - cost_aware) / cost_point))

point-forecast policy : total cost =  20143.5   (54.442 per day)
decision-aware policy : total cost =  17896.0   (48.367 per day)
realized cost reduction: 11.2%

The result is the section's whole thesis in two numbers: two policies with byte-identical forecasts and therefore identical forecast accuracy produce materially different realized costs, and the decision-aware one wins purely by acting on the part of the distribution the decision cares about. Finally, Code 19.6.3 shows that the from-scratch quantile order is not special pleading: a general-purpose convex optimizer, given the newsvendor cost and the predictive distribution as a sample, recovers the same optimal order, and the library version replaces the closed-form quantile with a two-line solve that generalizes to costs with no closed form.

import cvxpy as cp

def optimal_order_library(samples):
    """Library: minimize expected newsvendor cost over a sample of the predictive dist."""
    q = cp.Variable(nonneg=True)
    d = samples                                  # Monte-Carlo draws from the forecast
    cost = cp.sum(c_u * cp.pos(d - q) + c_o * cp.pos(q - d)) / len(d)  # mean cost
    cp.Problem(cp.Minimize(cost)).solve()
    return float(q.value)

# Same predictive distribution as the last backtest day, drawn as samples.
mu, sigma = predictive_normal(demand[:T])
samples = rng.normal(mu, sigma, size=20000)
q_scratch = optimal_order_from_normal(mu, sigma, crit_fractile)
q_library = optimal_order_library(samples)
print("from-scratch quantile order : %.2f" % q_scratch)
print("library optimizer order     : %.2f" % q_library)
print("agree:", np.isclose(q_scratch, q_library, atol=1.5))

from-scratch quantile order : 214.73
library optimizer order     : 214.61
agree: True

Read the three blocks together. Code 19.6.1 turned margins into a critical fractile and implemented the newsvendor cost and the quantile order from scratch in a handful of lines. Code 19.6.2 backtested two policies on identical forecasts and scored them by realized cost, showing the decision-aware policy strictly cheaper. Code 19.6.3 reproduced the from-scratch order with a general optimizer. The from-scratch decision logic of Codes 19.6.1 and 19.6.2 ran roughly 25 lines of explicit cost accounting and quantile arithmetic; the library shortcut below collapses the optimization step itself to a two-line convex solve, and a full operations stack (a forecaster plus an optimizer wired together) is what production decision systems ship.

from statsforecast import StatsForecast
from statsforecast.models import AutoARIMA

# Long-format frame: columns unique_id, ds, y (one series here).
import pandas as pd
train_df = pd.DataFrame({"unique_id": "item",
                         "ds": pd.date_range("2024-01-01", periods=len(demand), freq="D"),
                         "y": demand})
# Quantile forecast at the critical fractile, then order it. Levels are two-sided
# coverage; level=25 gives the 0.625 (and 0.375) quantiles for our fractile.
sf = StatsForecast(models=[AutoARIMA()], freq="D")
fc = sf.forecast(df=train_df, h=1, level=[25])     # probabilistic forecast in one call
# Output columns: ds, AutoARIMA, AutoARIMA-lo-25, AutoARIMA-hi-25 (version-dependent).
q_star = fc["AutoARIMA-hi-25"].iloc[0]             # the 0.625 quantile = the order

Exercises

Three exercises consolidate the section: one conceptual, one implementation, one open-ended.