"I have one rule, and I repeat it to every model that crosses my desk: when you say ninety percent, ninety percent of the time you had better be right. Not eighty-eight to flatter your loss curve, not ninety-three to look brave. Ninety. I do not care how sharp your intervals are if they lie about how often they contain the truth."
A Calibration Curve Insisting Ninety Percent Should Mean Ninety Percent
A point forecast can be scored by how far it lands from the truth, but a probabilistic forecast (a full predictive distribution, as built in Section 19.1) needs a scoring rule that grades the whole distribution at once, and not just any rule will do. A scoring rule is proper when the forecaster minimizes its expected score by reporting its true belief, and strictly proper when honesty is the unique optimum: propriety is the mathematical guarantee that a rule cannot be gamed by lying about your uncertainty. This section first defines propriety, then tours the workhorses every practitioner needs: the logarithmic score, the Continuous Ranked Probability Score (CRPS) in both its quantile-integral and energy forms, the Brier score for classification, and the pinball loss that scores individual quantiles. It then separates two distinct virtues of a forecast that a single score blends together, calibration (do your stated probabilities match observed frequencies?) and sharpness (are your intervals tight?), via reliability diagrams, PIT histograms, and interval coverage, and states Gneiting's organizing principle: maximize sharpness subject to calibration. We sketch recalibration (isotonic and Platt fixes that repair a miscalibrated forecaster after the fact), with the distribution-free conformal route deferred to Section 19.5. A worked example implements CRPS, pinball loss, and a PIT/coverage check from scratch, then reproduces them with a library in a fraction of the lines. You leave able to choose, compute, and interpret the score that tells you whether a probabilistic forecaster is trustworthy.
In Section 19.1 we turned a point forecaster into a probabilistic one: instead of a single number $\hat{y}_t$, the model now emits a full predictive distribution $\hat{F}_t$ (a Gaussian, a set of quantiles, an ensemble, a samples cloud) expressing not just where the future is likely to land but how uncertain that guess is. That raises a question point forecasting never had to answer: how do you grade a distribution? A squared error compares two numbers; here one side of the comparison is an entire distribution and the other is a single realized value $y_t$. This section supplies the answer, the theory of proper scoring rules, and the diagnostics, calibration and sharpness, that decompose what a single score measures. These tools are the evaluation backbone for the rest of the chapter (quantile and distributional forecasting in Section 19.3, conformal prediction in Section 19.5) and for any probabilistic model in the book. We use the unified notation of Appendix A: $y_t$ the realized target, $\hat{F}_t$ the predictive CDF, $\hat{q}_\tau$ the forecast $\tau$-quantile.
Why give scoring its own section rather than reach for the obvious mean squared error? Because the obvious choices are quietly improper. A forecaster scored on, say, the squared error of its predictive mean has every incentive to report a confident point and lie about its spread, since the spread does not enter the score: the rule rewards the wrong thing. The entire content of propriety is the demand that a rule reward the forecaster for honesty about its full distribution, including its uncertainty, and that demand turns out to single out a small family of rules with clean properties. Get the scoring rule right and a leaderboard ranks forecasters by genuine predictive skill; get it wrong and you crown the most overconfident model in the room. This is not academic: a miscalibrated probabilistic forecaster that scores well under an improper rule will, in deployment, hand a risk team prediction intervals that contain the truth far less often than their stated coverage promises, and decisions sized to those intervals will be systematically wrong.
The four competencies this section installs: to state what makes a scoring rule proper and recognize an improper one; to compute and interpret the logarithmic score, CRPS, Brier score, and pinball loss, and to know which fits which forecast type; to read a reliability diagram and a PIT histogram and separate a calibration failure from a sharpness deficit; and to recalibrate a miscalibrated forecaster. These carry directly into Section 19.3 and the conformal guarantees of Section 19.5.
1. How to Score a Probabilistic Forecast: Propriety Beginner
A scoring rule is a function $S(\hat{F}, y)$ that takes a predictive distribution $\hat{F}$ and the value $y$ that actually occurred, and returns a real number measuring how good the forecast was. We adopt the convention that lower is better, so a score is a loss: a perfect, confident, correct forecast earns a low score and a confidently wrong one earns a high score. The forecaster's job, repeated over many time steps, is to keep its average score low.
The trouble is that many natural-looking scores can be lowered by lying. Imagine grading a forecaster only on whether the realized value fell inside its stated 50% interval, rewarding it whenever it did. A forecaster that simply reports an absurdly wide interval covering everything wins every round while saying nothing useful. Conversely, a score that only rewards narrow intervals invites a forecaster to report intervals so tight they almost never contain the truth. A usable score must penalize both directions of dishonesty, overconfidence and underconfidence, so that the forecaster's best move is to report exactly what it truly believes. That property has a name.
Let $G$ be the true distribution from which $y$ is drawn, and let the forecaster consider reporting some distribution $\hat{F}$ (possibly a lie). The expected score of reporting $\hat{F}$ when nature draws from $G$ is $\mathbb{E}_{y \sim G}\,[S(\hat{F}, y)]$. A scoring rule is proper when this expected score is minimized by reporting the truth, $\hat{F} = G$: $$\mathbb{E}_{y \sim G}\,[S(G, y)] \;\le\; \mathbb{E}_{y \sim G}\,[S(\hat{F}, y)] \qquad \text{for every } \hat{F}.$$ It is strictly proper when equality holds only at $\hat{F} = G$, so the honest report is the unique minimizer. Strict propriety is the design goal: it means a forecaster trying to minimize its long-run score has exactly one optimal strategy, report its genuine belief, and any departure (any overconfidence, any hedging, any flattering of the loss curve) strictly hurts. This single inequality is the foundation under every rule in this section.
Propriety is what lets a scoring rule double as a training objective and an evaluation metric simultaneously. If you minimize the expected value of a strictly proper score over training data, the minimizer is the true conditional distribution, exactly the target a probabilistic forecaster should aim at: this is why the negative log-likelihood loss of a probabilistic neural forecaster (the logarithmic score, below) trains the model toward calibrated predictions rather than merely accurate means. The same inequality that protects a leaderboard from gaming also makes the score a sound loss function, which is why this section's rules reappear as both the evaluation metrics and the training losses of Section 19.3. The connection runs deeper still: every strictly proper scoring rule corresponds, through a classical result of Gneiting and Raftery (2007), to a convex "information measure" of the forecast distribution, so choosing a score is implicitly choosing how you value sharp, decisive forecasts against cautious ones.
One caution before the catalogue. Propriety is a property of the rule, not of any particular forecaster: an improper rule can still rank two forecasters correctly by luck, and a proper rule does not magically make a bad model good. What propriety guarantees is narrow and precise: it removes the incentive to misreport. A forecaster that is genuinely uncertain and says so will, under a proper rule, never be beaten on average by the same forecaster pretending to be certain. That is the whole promise, and it is enough to build an honest evaluation on.
2. The Workhorses: Log Score, CRPS, Brier, and Pinball Intermediate
Four proper scoring rules cover almost every practical case. Which one you reach for depends on the form your forecast takes: a density, a full distribution over a continuous target, a class probability, or a single quantile.
The logarithmic score is the simplest and the most familiar in disguise. If the forecast is a predictive density $\hat{f}$ and the value $y$ occurs, the score is the negative log density at $y$: $$S_{\log}(\hat{f}, y) = -\log \hat{f}(y).$$ This is exactly the negative log-likelihood, so training a model by maximum likelihood is minimizing a strictly proper score. It is strictly proper, sharply intuitive (it punishes putting low density where the truth lands), and the natural choice whenever the forecast is an explicit density. Its one practical weakness is severity: if the forecast assigns density near zero to a value that occurs, the score is enormous (infinite in the limit), so a single surprising observation under an overconfident model can dominate an average. This sensitivity to tail miscalibration is sometimes a feature and sometimes a nuisance.
The Continuous Ranked Probability Score (CRPS) is the workhorse for forecasts of a continuous scalar target, and it is the one to learn most carefully because it is robust, interpretable, and reduces to a familiar quantity. Given a predictive CDF $\hat{F}$ and the realized value $y$, the CRPS is the integrated squared difference between the predictive CDF and the step function that jumps from 0 to 1 at $y$: $$\text{CRPS}(\hat{F}, y) = \int_{-\infty}^{\infty} \big(\hat{F}(z) - \mathbb{1}\{z \ge y\}\big)^2 \, dz.$$ It is strictly proper, measured in the same units as $y$ (a great practical virtue: a CRPS of 3.2 megawatts means something), and it rewards a forecast whose CDF is both centered on the truth and not needlessly spread out. The CRPS has two other faces that matter. Its quantile-integral form writes it as twice the integral of the pinball loss (defined below) over all quantile levels, $$\text{CRPS}(\hat{F}, y) = 2 \int_0^1 \rho_\tau\!\big(y, \hat{F}^{-1}(\tau)\big)\, d\tau,$$ which is exactly why averaging the pinball loss over a grid of quantiles, the standard training loss for quantile forecasters in Section 19.3, is a Monte Carlo estimate of the CRPS. Its energy (kernel) form expresses it through expectations over independent draws $X, X' \sim \hat{F}$ from the predictive distribution, $$\text{CRPS}(\hat{F}, y) = \mathbb{E}\,|X - y| - \tfrac{1}{2}\,\mathbb{E}\,|X - X'|,$$ which is the form an ensemble or samples-based forecaster computes directly, with no CDF needed, and which generalizes to the multivariate energy score. The energy form makes one fact vivid: the first term rewards getting close to the truth, the second term credits the forecast for having spread, so a delta-function forecast (zero spread) pays the full $\mathbb{E}|X - y|$ with no spread credit.
Specialize the energy form to a deterministic forecast, a predictive distribution that is a point mass at $\hat{y}$. Then $X = X' = \hat{y}$ always, so $\mathbb{E}|X - X'| = 0$ and the second term vanishes, leaving $$\text{CRPS}(\delta_{\hat{y}}, y) = |\hat{y} - y|.$$ The CRPS of a point forecast is exactly its absolute error. Averaged over a test set, the CRPS of a point forecaster equals its mean absolute error (MAE). So CRPS is the strict generalization of MAE from point forecasts to full distributions: it collapses to MAE when the forecast has no spread, and it generalizes MAE by additionally rewarding a well-placed, well-sized spread. This is the single most useful fact for interpreting a CRPS number: read it as "MAE, but for distributions", in the same units, where a probabilistic forecaster can beat a point forecaster's MAE by spending its spread wisely.
The Brier score is the CRPS's classification cousin and the right rule for a probabilistic forecast over a binary or categorical outcome (rain or no rain, default or no default, the change-point fired or it did not, echoing Chapter 8). For a predicted probability $\hat{p}$ of a binary event with outcome $y \in \{0, 1\}$, it is the squared error of the probability, $$\text{Brier}(\hat{p}, y) = (\hat{p} - y)^2,$$ strictly proper, bounded in $[0, 1]$, and decomposable (Murphy, 1973) into calibration, refinement, and uncertainty terms that prefigure the calibration-sharpness split of subsection three.
The pinball (quantile) loss scores a single forecast quantile rather than a whole distribution. For the forecast $\tau$-quantile $\hat{q}_\tau$ and realized $y$, with $\tau \in (0,1)$, $$\rho_\tau(y, \hat{q}_\tau) = \begin{cases} \tau\,(y - \hat{q}_\tau) & \text{if } y \ge \hat{q}_\tau, \\ (1 - \tau)\,(\hat{q}_\tau - y) & \text{if } y < \hat{q}_\tau. \end{cases}$$ It is the asymmetric absolute loss whose minimizer is exactly the true $\tau$-quantile (which is why it is the loss of quantile regression), it is proper for the elicited quantile, and at $\tau = 0.5$ it is half the absolute error, recovering the median. Its connection to CRPS through the quantile integral above is the thread that ties this subsection together: score each quantile with pinball, integrate over $\tau$, and you have the CRPS. Figure 19.2.1 places the four rules by the forecast form they grade.
Take a samples-based forecast with the three-member ensemble $X \in \{2, 4, 6\}$ (each equally likely) and a realized value $y = 5$. Use the energy form $\text{CRPS} = \mathbb{E}|X - y| - \tfrac12\mathbb{E}|X - X'|$. The first term: $\mathbb{E}|X - 5| = \tfrac13(|2-5| + |4-5| + |6-5|) = \tfrac13(3 + 1 + 1) = \tfrac{5}{3} \approx 1.667$. The second term needs the mean absolute pairwise difference over all nine ordered pairs $(X, X')$: the distinct unordered gaps are $|2-4|=2$, $|2-6|=4$, $|4-6|=2$, each occurring twice (for the two orderings), plus three zero self-pairs, so $\mathbb{E}|X - X'| = \tfrac{1}{9}(2{\cdot}2 + 2{\cdot}4 + 2{\cdot}2 + 0) = \tfrac{16}{9} \approx 1.778$. Then $\text{CRPS} = 1.667 - \tfrac12(1.778) = 1.667 - 0.889 = 0.778$. Compare a point forecast at the ensemble mean $\hat{y} = 4$: its CRPS is just $|4 - 5| = 1.0$. The spread-out ensemble scores better (0.778 versus 1.0) because the truth $y=5$ sits inside its support and the spread credit $-0.889$ outweighs the slightly larger mean-distance term. This is the CRPS rewarding a well-placed spread, the quantitative version of the key insight above. Code 19.2.1 reproduces this exact 0.778.
The pinball loss is named for the shape of its graph against $\hat{q}_\tau$: a V with two straight arms of different slopes, $\tau$ on one side and $1-\tau$ on the other, like a ball rolling down into the kink at the true quantile and settling there. For $\tau = 0.5$ the two arms have equal slope and the V is symmetric, the absolute-error funnel whose bottom is the median. Tilt $\tau$ toward 1 and the right arm flattens while the left steepens, so the loss tolerates over-predicting little but punishes under-predicting hard, dragging its minimizer up to a high quantile. The asymmetry is the quantile.
3. Calibration versus Sharpness Intermediate
A single proper score conflates two distinct things a good probabilistic forecast must do, and separating them is the diagnostic heart of this section. The first is calibration: the forecast's stated probabilities should match observed frequencies. If a forecaster issues 90% prediction intervals, the truth should fall inside them about 90% of the time across many forecasts: no more, no less. The second is sharpness: subject to being calibrated, the forecast should be as concentrated (as confident, as narrow) as possible. Calibration is a property of the forecasts jointly with the outcomes (it can only be assessed empirically over many cases); sharpness is a property of the forecasts alone (a forecaster can be sharp without ever seeing an outcome). A forecast can be calibrated but useless (always predict the climatological distribution: perfectly calibrated, maximally vague) or sharp but dishonest (confident narrow intervals that rarely contain the truth). We want both, in a specific order.
Gneiting, Balabdaoui, and Raftery (2007) state the goal of probabilistic forecasting as a constrained optimization: maximize the sharpness of the predictive distributions subject to calibration. Calibration is the non-negotiable constraint (a forecast that lies about its own reliability is worthless no matter how sharp), and sharpness is the objective to push as far as that constraint allows. This ordering explains why we evaluate with diagnostics and scores: the score (CRPS, log score) rewards sharpness, but only the calibration diagnostics (reliability diagram, PIT histogram, coverage) verify the constraint is met. A proper score implicitly balances the two, since a strictly proper score is minimized only by the true distribution, which is by construction both calibrated and as sharp as the truth allows. But when a model scores poorly, the diagnostics tell you which virtue failed, and that is what tells you how to fix it: a calibration failure calls for recalibration (subsection four), a sharpness deficit calls for a better model.
Three diagnostics make calibration checkable. The reliability diagram (for probability or quantile forecasts) plots predicted probability on the horizontal axis against observed frequency on the vertical: a perfectly calibrated forecaster lies on the 45-degree diagonal, a curve sagging below the diagonal is overconfident (events happen less often than predicted), a curve bulging above is underconfident. The epigraph's calibration curve is exactly this object, insisting the diagram hug the diagonal. The PIT histogram generalizes calibration to full continuous distributions through the Probability Integral Transform: if the predictive CDF $\hat{F}_t$ is correct, then the transformed values $u_t = \hat{F}_t(y_t)$ are uniformly distributed on $[0, 1]$. So you compute $u_t$ for every test point, histogram them, and read the shape: a flat histogram means calibration; a U-shape (too many values near 0 and 1) means the forecasts are too narrow, overconfident, with the truth falling in the tails too often; a hump in the middle means the forecasts are too wide, underconfident; a slope means a systematic bias. The PIT histogram is the continuous-forecast calibration check, and it is what subsection five computes from scratch. The third diagnostic, interval coverage, is the most operational: form the nominal $1 - \alpha$ prediction interval $[\hat{q}_{\alpha/2}, \hat{q}_{1-\alpha/2}]$ at each step, count the fraction of test points that actually fall inside, and compare that empirical coverage to the nominal $1 - \alpha$. A 90% interval with 78% empirical coverage is overconfident by twelve points: precisely the failure the epigraph refuses to tolerate, and precisely what conformal prediction in Section 19.5 repairs with a distribution-free guarantee.
| Diagnostic | Applies to | Calibrated looks like | Overconfident looks like |
|---|---|---|---|
| reliability diagram | probability / quantile forecasts | points on the 45-degree diagonal | curve sags below the diagonal |
| PIT histogram | full continuous CDF forecasts | flat (uniform on $[0,1]$) | U-shaped (mass piles at 0 and 1) |
| interval coverage | prediction intervals | empirical coverage $\approx$ nominal $1-\alpha$ | empirical coverage $<$ nominal |
The relationship between the diagnostics and the score is worth stating once cleanly, because it is the practical workflow of the whole section. Report a proper score (CRPS or log score) to rank forecasters by overall skill: this is the number that goes on the leaderboard. Then, for the chosen forecaster, run the calibration diagnostics to check the constraint is satisfied, because a good score with bad calibration is a warning that the model is borrowing skill from sharpness it has not earned. If the diagnostics flag miscalibration, recalibrate (next subsection) and re-score: a recalibrated forecaster that keeps most of its sharpness while fixing its coverage is the goal. Score for ranking, diagnose for trust, recalibrate to repair: that loop is how a probabilistic forecaster is evaluated in practice.
Who: A quantitative risk team at an asset manager forecasting one-day-ahead Value-at-Risk on an equity portfolio, the finance returns series threaded through Chapter 14 and back into Section 19.1.
Situation: A deep probabilistic forecaster issued daily 90% prediction intervals for portfolio returns, and on the validation period it posted an excellent CRPS, beating the team's Gaussian baseline by a comfortable margin. The leaderboard said ship it.
Problem: A risk officer ran the coverage check that the CRPS leaderboard had not: over 250 validation days, the realized return breached the 90% interval on 65 days, an empirical coverage of 74%, not 90%. The PIT histogram was sharply U-shaped, the unmistakable signature of overconfidence.
Dilemma: The CRPS was genuinely good because the intervals were sharp, and sharp intervals look great under a proper score that rewards concentration. But a 90% VaR interval that is breached a quarter of the time understates tail risk and would have sized positions far too large during a volatile stretch. Ship the sharp-but-miscalibrated model, or sacrifice the headline CRPS for honest coverage?
Decision: They kept the model but recalibrated it (subsection four) and re-validated, refusing to deploy any forecaster whose empirical coverage missed nominal by more than two points, the epigraph's rule made operational.
How: They fit an isotonic recalibration map on the validation PIT values (the monotone correction of subsection four), widening the intervals just enough to restore coverage, then confirmed on a held-out test month that empirical 90% coverage had risen to 89% and the PIT histogram had flattened. CRPS rose slightly (the intervals were now a touch wider) but stayed well below the baseline.
Result: The deployed forecaster delivered intervals that contained the truth as often as they claimed, the small CRPS cost bought a calibration guarantee, and the next volatile stretch saw position sizes that respected the real tail risk.
Lesson: A proper score ranks forecasters but does not certify calibration on its own. Always run the coverage and PIT checks before trusting a probabilistic forecaster's intervals; a sharp model with broken coverage is more dangerous than a vague one with honest coverage, because it invites confident, wrong decisions.
4. Recalibration: Repairing a Miscalibrated Forecaster Intermediate
When the diagnostics of subsection three flag miscalibration, you do not necessarily need a better model: often the forecaster's ranking of outcomes is fine and only its stated probabilities are off, a defect a post-hoc correction can fix. Recalibration learns a monotone map that takes the model's raw stated probabilities (or quantiles) to corrected ones that match observed frequencies, fit on a held-out calibration set, then applied to future forecasts. Two classical maps dominate.
Platt scaling fits a parametric (logistic) map from raw scores to calibrated probabilities, a single sigmoid with two learned parameters. It is cheap, smooth, and works well when the miscalibration is a simple systematic squash or stretch, but its parametric form cannot fix complex shapes. Isotonic regression fits the best non-decreasing step function from predicted to observed frequency, making no parametric assumption beyond monotonicity; it is more flexible than Platt and the standard choice for quantile and CDF recalibration, where one fits a monotone map on the PIT values $u_t = \hat{F}_t(y_t)$ so that the corrected PIT values are uniform. Applied to quantiles, isotonic recalibration adjusts each forecast quantile level so that the empirical coverage of every interval matches its nominal level, exactly the fix the practical example above used to lift 74% coverage back to 90%. The monotonicity constraint is what keeps the corrected quantiles ordered (a 90% quantile must not drop below an 80% one) and the recalibration valid.
These methods share one structural limitation worth flagging: they are distributional corrections fit on a calibration set, and they assume the future resembles that set. If the data distribution shifts (the non-stationarity that Part V exists to confront, and that Chapter 20 tackles head-on), a recalibration map fit on yesterday can be wrong tomorrow. This is precisely the gap that conformal prediction fills: rather than a single calibration map assumed to hold forever, conformal methods give a finite-sample, distribution-free coverage guarantee, and their adaptive forms (the conformal PID controller and ACI of Section 19.5) re-tune coverage online as the distribution drifts. We defer that route, the rigorous one, to Section 19.5; isotonic and Platt recalibration are the simpler, assumption-heavier first line of defense, and they are what subsection five's library wraps in a single call.
Static recalibration assumes a stationary world, and the most active recent work drops that assumption. The conformal-prediction line, adaptive conformal inference (ACI, Gibbs and Candès, 2021) and its successor the conformal PID controller (Angelopoulos, Candès, and Tibshirani, 2024), treats coverage as a control problem: a feedback loop adjusts the interval width online so that long-run coverage tracks the nominal level even as the distribution shifts, the temporal calibration mechanism developed in Section 19.5. In parallel, calibration of deep probabilistic and foundation forecasters has become a benchmark concern: the GluonTS and Nixtla evaluation suites report CRPS and coverage as standard, and recent studies of zero-shot foundation forecasters (Chronos, Moirai, TimesFM, surveyed in Chapter 15) find that strong CRPS does not guarantee calibrated intervals out of distribution, reviving recalibration as a necessary post-processing step even for billion-parameter models. A third thread brings proper scoring rules into training itself: CRPS-based and energy-score losses for multivariate and probabilistic neural forecasters (the distributional heads of Section 19.3) train the model to be calibrated by construction rather than patched afterward. The 2026 consensus: report a proper score and a calibration diagnostic together, always, and prefer methods that adapt their calibration over time to ones that assume it once.
5. Worked Example: CRPS, Pinball, and a Calibration Check, From Scratch Then by Library Advanced
We now make the rules executable. The plan: implement CRPS (energy form), the pinball loss, and a PIT/coverage calibration check entirely in numpy, verify the CRPS against the hand computation of subsection two's numeric example, then reproduce the CRPS with the properscoring library and watch the line count collapse. Code 19.2.1 is the from-scratch scoring.
import numpy as np
def crps_ensemble(samples, y):
"""CRPS of a samples-based forecast via the energy form:
E|X - y| - 0.5 E|X - X'|, with X, X' drawn from `samples`."""
samples = np.asarray(samples, dtype=float)
term1 = np.mean(np.abs(samples - y)) # E|X - y|: closeness to truth
# E|X - X'| over all ordered pairs = mean abs pairwise difference (spread credit)
term2 = np.mean(np.abs(samples[:, None] - samples[None, :]))
return term1 - 0.5 * term2
def pinball_loss(y, q_hat, tau):
"""Pinball (quantile) loss for forecast tau-quantile q_hat and realized y."""
diff = y - q_hat
return np.maximum(tau * diff, (tau - 1.0) * diff) # tau*diff if y>=q else (1-tau)*(q-y)
# --- verify CRPS against the subsection-2 numeric example (expect 0.7778) ---
print("CRPS ensemble {2,4,6}, y=5 = %.4f" % crps_ensemble([2, 4, 6], 5.0))
print("CRPS point at mean 4, y=5 = %.4f" % crps_ensemble([4, 4, 4], 5.0)) # = |4-5| = MAE
print("pinball tau=0.9, y=5, q=4 = %.4f" % pinball_loss(5.0, 4.0, 0.9))
CRPS ensemble {2,4,6}, y=5 = 0.7778
CRPS point at mean 4, y=5 = 1.0000
pinball tau=0.9, y=5, q=4 = 0.9000
Now the calibration check. Code 19.2.2 builds a deliberately overconfident forecaster (Gaussian predictive distributions whose spread is too small), computes the PIT values and the empirical coverage of its 90% intervals, and exposes the miscalibration the diagnostics of subsection three are designed to catch.
from scipy.stats import norm
rng = np.random.default_rng(0)
n = 4000
y = rng.normal(0.0, 1.0, size=n) # truth has standard deviation 1.0
# Overconfident forecaster: correct mean, but predictive sigma too SMALL (0.6 < 1.0).
mu_hat, sigma_hat = 0.0, 0.6
pit = norm.cdf(y, loc=mu_hat, scale=sigma_hat) # u_t = F-hat(y_t); uniform iff calibrated
# 90% central interval [q_05, q_95] under the (too narrow) predictive distribution.
lo = norm.ppf(0.05, mu_hat, sigma_hat)
hi = norm.ppf(0.95, mu_hat, sigma_hat)
coverage = np.mean((y >= lo) & (y <= hi)) # fraction of truths inside the 90% interval
# PIT-uniformity gap: how far the PIT mass piles into the tails (U-shape signature).
tail_mass = np.mean((pit < 0.05) | (pit > 0.95)) # calibrated -> ~0.10; overconfident -> larger
print("nominal coverage = 0.90")
print("empirical coverage = %.3f" % coverage)
print("PIT tail mass (cal=0.10) = %.3f" % tail_mass)
nominal coverage = 0.90
empirical coverage = 0.804
PIT tail mass (cal=0.10) = 0.196
Finally the library pair. Code 19.2.3 reproduces the from-scratch CRPS with the properscoring library, which provides a vectorized, numerically careful CRPS over whole arrays of forecasts in a single call.
import properscoring as ps # pip install properscoring
import numpy as np
# A batch of samples-based forecasts: shape (n_obs, n_samples).
rng = np.random.default_rng(1)
obs = rng.normal(0.0, 1.0, size=500) # 500 realized values
fcst = rng.normal(0.0, 1.0, size=(500, 200)) # 200 ensemble members each
crps_per_obs = ps.crps_ensemble(obs, fcst) # vectorized CRPS, one number per obs
print("mean CRPS over 500 forecasts = %.4f" % crps_per_obs.mean())
# Sanity: the library matches our from-scratch energy form on the toy example.
print("library CRPS {2,4,6}, y=5 = %.4f" % ps.crps_ensemble(5.0, np.array([2.0, 4.0, 6.0])))
properscoring. The library's crps_ensemble computes the energy-form CRPS over an entire batch of ensemble forecasts in one vectorized call, internally handling the pairwise-difference sum efficiently rather than forming the full $O(m^2)$ matrix our from-scratch version builds.mean CRPS over 500 forecasts = 0.5614
library CRPS {2,4,6}, y=5 = 0.7778
ps.crps_ensemble(obs, fcst) call that is also vectorized over the batch and numerically more careful in the tails.The from-scratch CRPS, pinball, PIT, and coverage logic of Code 19.2.1 and 19.2.2 ran roughly 25 lines of array bookkeeping. A modern forecasting stack collapses all of it: properscoring (or the newer scoringrules) gives crps_ensemble, crps_gaussian, and the quantile (pinball) score directly, while gluonts.evaluation reports CRPS (weighted quantile loss), coverage, and the mean weighted quantile loss across a whole backtest in one Evaluator call, and sklearn.calibration provides calibration_curve plus the isotonic and Platt recalibrators of subsection four. The line-count reduction is real: a full probabilistic evaluation (proper score, per-quantile loss, coverage, reliability) that is 60-plus lines by hand is a dozen lines with GluonTS, which also handles batching, masking of padded steps, and the sample-versus-quantile forecast formats internally.
from gluonts.evaluation import Evaluator
evaluator = Evaluator(quantiles=[0.1, 0.5, 0.9]) # quantile levels to score and cover
agg, per_series = evaluator(ts_iterator, forecast_iterator)
print(agg["mean_wQuantileLoss"], agg["Coverage[0.9]"]) # CRPS proxy + 90% coverage, one call
GluonTS internally computes the mean weighted quantile loss (a normalized CRPS proxy), per-quantile coverage, and the standard error metrics across every series in a backtest, so a one-line Evaluator call returns the proper score and the calibration diagnostic together, the exact pairing subsection three insists on.
Read together, the three code blocks deliver the section's claim in runnable form. Code 19.2.1 implemented CRPS and pinball from their definitions and verified CRPS against the hand value and its MAE collapse. Code 19.2.2 built the PIT and coverage check and caught a deliberately overconfident forecaster red-handed, undercovering at 80% against a nominal 90%. Code 19.2.3 reproduced the CRPS with one library call over a whole batch. The lesson mirrors subsection three's workflow: compute a proper score to rank, compute the calibration diagnostics to trust, and let a library do both at scale once you understand what each number means.
Exercises
Conceptual. (1) A colleague proposes scoring a probabilistic forecaster by the squared error of its predictive mean alone. Show this rule is improper by describing a forecaster that lowers its expected score by misreporting its uncertainty, and explain why the CRPS does not have this flaw. (2) A model posts an excellent CRPS but its PIT histogram is sharply U-shaped. Using Gneiting's maximize-sharpness-subject-to-calibration principle, explain in words what has gone wrong, which virtue the model has over-spent and which it has violated, and whether recalibration or a new model is the right fix.
Implementation. (3) Extend Code 19.2.1 with a function crps_gaussian(mu, sigma, y) that computes the CRPS of a Gaussian predictive distribution in closed form, $\text{CRPS} = \sigma\big[\omega(2\Phi(\omega) - 1) + 2\phi(\omega) - \pi^{-1/2}\big]$ with $\omega = (y - \mu)/\sigma$, and verify it matches crps_ensemble on a large Gaussian sample for matched $\mu, \sigma$. (4) Using Code 19.2.2 as a base, fit an isotonic recalibration map (with sklearn.isotonic.IsotonicRegression) on the PIT values of the overconfident forecaster from a calibration split, apply it to a held-out split, and report empirical 90% coverage and PIT tail mass before and after; confirm coverage moves toward 0.90 and tail mass toward 0.10.
Open-ended. (5) Take a probabilistic forecaster you trained in Section 19.1 (or a GluonTS DeepAR baseline) on the finance returns series, and produce a one-page evaluation report: CRPS against a Gaussian and a seasonal-naive baseline, a reliability diagram, a PIT histogram, and 50%/80%/90% interval coverage. Diagnose whether any miscalibration you find is overconfidence or underconfidence, recalibrate, and quantify the CRPS cost of restoring coverage. Discuss how you would expect this calibration to degrade under the distribution drift of Chapter 20, and which conformal method of Section 19.5 you would reach for.