Hidden States and the State-Space Formulation

"They never measure me. They measure my shadow on the wall, blurred by whatever noise the day brought, and from that smudge they reconstruct me anyway. I have stopped resenting it. A good estimate, I have decided, is the sincerest form of being seen."

A Latent State Nobody Ever Measures Directly

The previous chapter, on multivariate and econometric models, took the observed series at face value: the vector autoregression of Section 6.1 forecast every series directly from the lagged values of every series, and the cointegration and factor methods of Section 6.4 already gestured at hidden common drivers behind a panel. This chapter makes that hidden layer the protagonist. We stop modelling the observation and start modelling the thing behind the observation, the state, with the observation demoted to a noisy, possibly incomplete window onto it. The reader should arrive comfortable with linear-Gaussian algebra (multivariate normals, covariance matrices, linear maps), the lag operator, and the companion-form rewrite of Section 6.1, because the state vector here is the direct descendant of that stacked companion state. Throughout we use the symbols of the unified notation table in Appendix A: $\mathbf{z}_t \in \mathbb{R}^d$ for the latent state, $\mathbf{x}_t \in \mathbb{R}^K$ for the observation, $F$ for the transition matrix, $H$ for the observation matrix, and $Q, R$ for the process and measurement noise covariances.

1. The Core Idea: A Hidden State That Summarizes the Past Beginner

Start with the physical intuition before any equation. A drone has a true position and velocity at every instant; that is its state. We never observe the state. We observe GPS pings and accelerometer readings, each a corrupted, partial function of the true state. A patient has a true underlying physiological condition; we observe a handful of noisy vitals sampled when a nurse happens to take them. A market has a true latent volatility; we observe only the returns it produces. In every case there is a quantity that genuinely drives the system, evolving by its own rules, and a separate quantity that we actually get to measure, related to the driver but blurred by noise and often lower-dimensional than it. The state-space model is the formal commitment to keeping these two quantities distinct: a state that evolves, and an observation that reports on the state through a haze. Figure 7.1.3 dramatizes the same split as a puppeteer whose hidden hand we infer only from the puppet it moves.

The conceptual heart of the formulation is what the state is for. The state $\mathbf{z}_t$ is defined to be a sufficient summary of the entire history up to time $t$: everything in the past that is relevant to predicting the future is compressed into $\mathbf{z}_t$, so that conditioning on the full past is no better than conditioning on the current state alone. Formally, the model assumes the Markov property on the state,

and that each observation depends only on the contemporaneous state,

These two conditional-independence statements are the entire structural content of a state-space model; the matrices and noise distributions to come are just one particular linear-Gaussian way of filling them in. The first says the state carries all the memory: the future is independent of the deep past once you know the present state. The second says the observation is a momentary readout of the state with no memory of its own. Together they license the most important computational fact in this book: we can process a stream one step at a time, carrying forward only a fixed-size belief about the current state, never re-reading the history. That recursive, constant-memory processing is what makes the Kalman filter (and later the RNN, and the structured state-space model) tractable on arbitrarily long sequences.

Contrast this with the direct modelling of the previous chapters to see what is genuinely new. An autoregression carries memory by literally storing the last $p$ observations and regressing on them; its "state" is the raw recent past, and to extend the memory you must lengthen the window and pay for more lags. A state-space model instead carries memory in a compressed latent vector whose dimension is fixed regardless of how far back the relevant history reaches: a single state component evolving as a slow random walk can summarize an arbitrarily long trend, and a small rotating block can summarize a season of any past length. The state decouples how much history matters from how much you must store, which is precisely the decoupling that lets the same idea, scaled up and learned, handle sequences of thousands of steps in the deep-learning chapters without a thousand-tap window. This is also why a state-space model can absorb an irregular or gapped series gracefully: a missing observation simply means a step with no update, the state coasting forward on its dynamics, whereas a fixed-lag autoregression chokes on the hole in its window.

• AR memory: store the last $p$ raw values, and lengthen the window whenever the relevant past reaches further back.
• State-space memory: store one fixed-size state vector that summarizes the past no matter how far back it reaches.

It is worth being precise about the word "sufficient," because it is the idea that elevates the state from a modelling convenience to a deep principle. A sufficient statistic is a function of the data that loses no information about a quantity of interest. Here the quantity of interest is the future of the series, and the claim is that $\mathbf{z}_t$ is a sufficient statistic of the past $\mathbf{x}_{1:t}$ for that future. A long series might have thousands of past observations, yet a state of dimension two or three can carry all the predictively relevant information in them. The state is a lossless compression of the past with respect to prediction: it may forget the exact values the series took, but it cannot forget anything that would have helped forecast what comes next. This is exactly the property we will later ask a recurrent network to learn rather than be told.

2. The Linear-Gaussian State-Space Model Intermediate

The conditional-independence structure of subsection one admits many concrete realizations; the linear-Gaussian one, the workhorse of this chapter, fills in both conditionals with linear maps and Gaussian noise. It is specified by two equations. The transition equation (also called the state equation) governs how the hidden state evolves,

and the observation equation (also called the measurement equation) governs how each state is reported,

The two noise streams $\mathbf{w}_t$ and $\mathbf{v}_t$ are assumed mutually independent, independent across time, and independent of the initial state $\mathbf{z}_0 \sim \mathcal{N}(\mathbf{m}_0, P_0)$. Because every operation is linear and every noise is Gaussian, the entire joint distribution of states and observations is Gaussian, which is the property the Kalman filter exploits to keep its beliefs Gaussian forever. Each object in these two lines has a precise and worth-memorizing meaning, set out next.

It pays to read the two equations as a generative recipe and write down the conditional distributions they induce, because those two Gaussians are literally the linear-Gaussian realization of the abstract conditionals $p(\mathbf{z}_t \mid \mathbf{z}_{t-1})$ and $p(\mathbf{x}_t \mid \mathbf{z}_t)$ of subsection one. The transition equation says the next state is the old state pushed through $F$ plus a Gaussian kick, so

Because of the Markov and observation-independence structure, the full joint density of an entire run of states and observations factorizes into a product of exactly these two kinds of term, one transition factor and one observation factor per time step, anchored by the initial-state prior,

This factorization is the whole model on one line, and it is what every inference algorithm in this chapter operates on: filtering, smoothing, and likelihood evaluation are all just different ways of summing and conditioning this product of Gaussians. Reading it left to right is sampling (the generative direction, state then observation); reading it right to left, solving for the states given the observations, is inference (the direction the filter runs). Each object in these two equations has a precise and worth-memorizing meaning, set out next.

The transition matrix $F \in \mathbb{R}^{d \times d}$ encodes the system's intrinsic dynamics: how the state at one instant maps into the state at the next, absent any disturbance. Its eigenvalues control stability exactly as the companion matrix of Section 6.1 did: eigenvalues inside the unit circle give a mean-reverting state, an eigenvalue exactly at one gives a random walk or integrator, and eigenvalues outside the unit circle give an explosive state. The process-noise covariance $Q \in \mathbb{R}^{d \times d}$ measures how much the state is buffeted by unmodelled disturbances between steps; a large $Q$ says the state can move a lot on its own, which makes the filter trust new data quickly, while a small $Q$ says the state is nearly deterministic, which makes the filter rely on its dynamics. The observation matrix $H \in \mathbb{R}^{K \times d}$ projects the (often higher-dimensional) state down into the measured quantities; it is where partial observability lives, because $H$ can read out some components of the state and ignore others entirely, exactly as a GPS reports position but not velocity. The measurement-noise covariance $R \in \mathbb{R}^{K \times K}$ is the precision of the sensor; a large $R$ is a noisy instrument the filter should distrust, a small $R$ is a precise one it should believe. The interplay between $Q$ and $R$, how much to trust the dynamics versus the data, is the single most important tuning decision in all of state-space filtering, and the Kalman gain of Section 7.2 is precisely the optimal resolution of that trade-off.

A subtlety the matrices quietly encode is the difference between the state dimension $d$ and the observation dimension $K$. Nothing forces them to be equal, and in the interesting cases they are not. A constant-velocity tracker carries a four-dimensional state (two position coordinates, two velocity coordinates) but observes only the two positions, so $H$ is a $2 \times 4$ matrix that reads position and discards velocity; the velocity is inferred entirely from how the observed positions change, never measured. This is the formal meaning of a hidden state: components of $\mathbf{z}_t$ that no row of $H$ ever touches, recoverable only through the dynamics in $F$ that couple them to the observed components. When $H$ is square and invertible and $R$ is tiny, the state is essentially observed and the model degenerates toward the directly-modelled series of earlier chapters; the state-space machinery earns its keep precisely when $H$ is fat (fewer observations than state components) or $R$ is large.

3. The Three Inference Tasks: Filtering, Prediction, Smoothing Intermediate

Once a model is specified, every question we can ask about the hidden state is a request for a conditional distribution of some state given some set of observations. The three canonical questions differ only in which observations condition the estimate of which state, and naming them precisely now prevents a great deal of later confusion. Let $\mathbf{x}_{1:s}$ denote the observations from time $1$ through time $s$. The three tasks are as follows.

Filtering estimates the current state from all data up to and including now,

the best estimate of where the system is, given everything observed so far. This is the online, real-time task: a tracker reporting the drone's position as each measurement arrives uses the filtering distribution, because at time $t$ it has access to exactly $\mathbf{x}_{1:t}$ and nothing from the future. Prediction estimates a future state from data only up to now,

the forecasting task: project the current filtered belief forward through the dynamics $F$ for $h$ steps, with the uncertainty growing at each step because process noise accumulates and no new measurement arrives to rein it in. Smoothing estimates a past or present state using all the data, including observations that came after the state in question,

where $T$ is the end of the available record. Smoothing is the offline, retrospective task: revisiting time $t$ with the benefit of hindsight, using future observations to sharpen the estimate of the past. The distinction is not academic. The same state $\mathbf{z}_t$ has a different, and almost always more certain, smoothed estimate than filtered estimate, because the smoother conditions on strictly more information ($\mathbf{x}_{1:T} \supseteq \mathbf{x}_{1:t}$). A useful mnemonic: filtering looks only backward, prediction looks forward from now, and smoothing looks both ways.

A fourth quantity rides along with these three and deserves a mention now, because it is what lets us fit a state-space model at all: the marginal likelihood of the data. As the filter runs, each step produces a one-step-ahead prediction of the next observation, $\hat{\mathbf{x}}_{t\mid t-1} = H F \,\mathbf{m}_{t-1\mid t-1}$, and the difference between what we predicted and what actually arrived, the innovation $\boldsymbol{\nu}_t = \mathbf{x}_t - \hat{\mathbf{x}}_{t\mid t-1}$, is a zero-mean Gaussian with a covariance $S_t$ the filter also computes. The log-likelihood of the whole series is then nothing but the sum of the log-densities of these innovations,

a decomposition (the prediction-error decomposition) that turns the running filter into a likelihood machine. This is the quiet workhorse of the whole chapter: maximizing this expression over the unknown entries of $F, H, Q, R$ is how every statsmodels state-space model, including the worked example below, estimates its parameters, and comparing it across models by AIC and BIC is how we choose among them, exactly as in Chapter 5. So the same forward pass that filters also, for free, scores the model.

These three distributions (and the likelihood) are not computed by three unrelated algorithms. Remarkably, all of them flow from a single forward-backward recursion on the linear-Gaussian model: the forward pass is the Kalman filter, which produces the filtering distributions and, as a free by-product of its predict step, the one-step predictions; the backward pass is the Kalman (Rauch-Tung-Striebel) smoother, which revisits each state with the future folded in. Prediction beyond one step is just the filter's predict step iterated without any update. The unity is the subject of Section 7.2 and Section 7.3; what matters now is that the three tasks are one model interrogated three ways, distinguished only by their conditioning sets.

4. Why State-Space Is Unifying Advanced

The reason this chapter sits where it does, after the univariate, frequency-domain, and multivariate chapters rather than before them, is that the state-space form is the common denominator of nearly everything that came before. A startling number of classical models that look distinct on the surface turn out to be the same linear-Gaussian state-space model wearing different $F$, $H$, $Q$, $R$. Once each is written in state-space form, the single inference engine of Section 7.2 estimates, forecasts, and computes the likelihood of all of them, with no per-model algorithm required. We survey the four most important reductions.

ARIMA models (the univariate workhorses of Chapter 5) have an exact state-space representation: the state is a stacked vector of current and lagged values plus accumulated moving-average shocks, and $F$ is essentially the companion matrix of Section 6.1. Writing an ARMA($p,q$) in state-space form lets the Kalman filter evaluate its exact Gaussian likelihood even with missing observations, which is precisely how statsmodels fits ARIMA internally. Exponential smoothing (ETS) models, the level-trend-season forecasters, are state-space models whose state holds the current level, trend, and seasonal components and whose transition adds the trend to the level and rotates the seasonal indices; the "smoothing parameters" $\alpha, \beta, \gamma$ are reparameterizations of the noise covariances. Unobserved-components models are state-space by construction: they posit that an observed series is the sum of latent level, trend, seasonal, and cycle components, each given its own little transition, and ask the filter to tease them apart, which is exactly the decomposition our worked example performs. Dynamic factor models, the high-dimensional reduction we glimpsed in Section 6.4, put a small latent factor vector in the state and let a tall observation matrix $H$ (the factor loadings) generate many observed series from those few factors, the literal embodiment of "few hidden drivers, many noisy readouts."

The intellectual economy here is the entire reason the state-space view is worth the abstraction. Rather than a separate estimation routine, a separate forecasting formula, and a separate likelihood for ARIMA, for ETS, for unobserved components, and for factor models, we write each as a triple $(F, H, Q, R)$ and hand it to one filter. The filter returns the filtered states (for nowcasting), the predicted states (for forecasting), the smoothed states (for decomposition and signal extraction), and the Gaussian log-likelihood (for fitting the parameters by maximum likelihood, and for comparing models by AIC and BIC exactly as in Chapter 5). This is the same consolidating move the vector autoregression made within multivariate modelling, taken one level higher: where the VAR unified many regressions under one design matrix, the state-space form unifies many model families under one filter.

5. Worked Example: The Local-Level Model From Scratch and Through statsmodels Advanced

We make all of this concrete with the simplest non-trivial state-space model, the local-level model, also called the random-walk-plus-noise model. It supposes a true underlying level that drifts as a random walk, observed through additive measurement noise. In the matrices above it is the one-dimensional case $F = 1$, $H = 1$, with scalar process variance $Q = \sigma_w^2$ and measurement variance $R = \sigma_v^2$:

Despite its simplicity it is genuinely useful (it is the state-space form of simple exponential smoothing, and a sensible nowcaster for a slowly drifting quantity) and it isolates the core phenomenon: a hidden level we want, buried in observation noise we do not. This is the healthcare running thread of Part II in miniature, a true physiological level (say a patient's baseline glucose) drifting slowly, sampled through a noisy meter. Code 7.1.1 simulates the hidden state and its noisy observations and assembles the state-space matrices explicitly, so the two-layer structure is concrete rather than asserted.

import numpy as np

rng = np.random.default_rng(7)
n = 200
sigma_w, sigma_v = 0.30, 1.50      # process std (state drift) and measurement std (sensor noise)

# --- Build the linear-Gaussian state-space matrices explicitly (local-level model) ---
F = np.array([[1.0]])              # transition: the level is a random walk
H = np.array([[1.0]])             # observation: we read the level directly...
Q = np.array([[sigma_w**2]])      # ...but the level itself drifts (process noise)
R = np.array([[sigma_v**2]])      # ...and the reading is corrupted (measurement noise)

# --- Simulate the hidden state z_t and the observed x_t from those matrices ---
z = np.zeros(n); x = np.zeros(n)
z[0] = 10.0                        # true starting level
x[0] = z[0] + sigma_v * rng.standard_normal()
for t in range(1, n):
    z[t] = (F @ [[z[t-1]]])[0, 0] + sigma_w * rng.standard_normal()   # state equation
    x[t] = (H @ [[z[t]]])[0, 0] + sigma_v * rng.standard_normal()     # observation equation

print("hidden state z_t  (first 5):", np.round(z[:5], 3))
print("observed   x_t  (first 5):", np.round(x[:5], 3))
print("noise-to-signal: meas std =", sigma_v, " vs drift std per step =", sigma_w)

hidden state z_t  (first 5): [10.    10.057 10.16  10.305 10.18 ]
observed   x_t  (first 5): [ 9.218 11.49   8.97  11.4    9.62 ]
noise-to-signal: meas std = 1.5  vs drift std per step = 0.3

Now we recover the hidden level from the observations alone. Code 7.1.2 implements the scalar Kalman filter for this model from scratch, the forward recursion that produces the filtering distribution $p(z_t \mid x_{1:t})$ of subsection three. It is a preview of the general derivation in Section 7.2; here we simply run the predict-then-update loop and watch the noisy observations collapse onto the hidden truth. The point of writing it by hand is to see that filtering is nothing more than the two model equations applied recursively with bookkeeping for uncertainty.

# From-scratch scalar Kalman filter for the local-level model (recovers z from x).
def local_level_filter(x, F, H, Q, R, m0=0.0, P0=1e4):
    n = len(x)
    m = np.zeros(n)                       # filtered state mean  E[z_t | x_1:t]
    P = np.zeros(n)                       # filtered state variance
    m_pred, P_pred = m0, P0               # belief before seeing x[0]
    f, h, q, r = F[0,0], H[0,0], Q[0,0], R[0,0]
    for t in range(n):
        if t > 0:                         # PREDICT: push the state through the dynamics
            m_pred = f * m[t-1]
            P_pred = f * P[t-1] * f + q   # uncertainty grows by the process noise q
        S = h * P_pred * h + r            # innovation (prediction-error) variance
        K = P_pred * h / S                # Kalman gain: how much to trust this measurement
        m[t] = m_pred + K * (x[t] - h * m_pred)   # UPDATE: correct toward the observation
        P[t] = (1 - K * h) * P_pred       # uncertainty shrinks after seeing data
    return m, P

m, P = local_level_filter(x, F, H, Q, R)
rmse_obs  = np.sqrt(np.mean((x - z)**2))   # error if we just trusted the raw observations
rmse_filt = np.sqrt(np.mean((m - z)**2))   # error of the filtered estimate
print(f"RMSE of raw observations vs truth : {rmse_obs:.3f}")
print(f"RMSE of filtered estimate vs truth: {rmse_filt:.3f}")
print(f"steady-state Kalman gain (last K) : {P[-1]/(P[-1]+R[0,0]):.3f}")

RMSE of raw observations vs truth : 1.486
RMSE of filtered estimate vs truth: 0.512
steady-state Kalman gain (last K) : 0.196

To make the hidden-versus-observed contrast unmistakable, Code 7.1.3 renders the simulation as an inline plot of the three series: the true hidden state, the noisy observations scattered around it, and the filtered estimate tracking the truth through the noise. The figure is the visual statement of the whole section.

import matplotlib.pyplot as plt

t_axis = np.arange(n)
fig, ax = plt.subplots(figsize=(9, 4))
ax.plot(t_axis, x, '.', color='0.6', ms=4, label='observed  $x_t$ (noisy)')
ax.plot(t_axis, z, color='C0', lw=2.2, label='hidden state  $z_t$ (truth)')
ax.plot(t_axis, m, color='C3', lw=1.8, label='filtered  $E[z_t \\mid x_{1:t}]$')
ax.fill_between(t_axis, m - 2*np.sqrt(P), m + 2*np.sqrt(P),
                color='C3', alpha=0.15, label=r'$\pm 2\sigma$ filter band')
ax.set_xlabel('time $t$'); ax.set_ylabel('level')
ax.set_title('Local-level model: a hidden state seen only through noise')
ax.legend(loc='upper left', fontsize=9); fig.tight_layout()
fig.savefig('local_level_filter.png', dpi=130)

That was the transparent machinery: explicit matrices, a hand-rolled recursion, a plot. In practice you would not hand-code the filter, the likelihood, and the maximum-likelihood estimation of $\sigma_w$ and $\sigma_v$; statsmodels packages the entire local-level model behind one object that builds the state-space form, runs the Kalman filter and smoother, and estimates the unknown variances by maximum likelihood. Code 7.1.4 is the library shortcut.

import numpy as np
from statsmodels.tsa.statespace.structural import UnobservedComponents

# 'local level' IS the random-walk-plus-noise model; the library builds F,H,Q,R internally.
model = UnobservedComponents(x, level='local level')
res = model.fit(disp=False)                 # MLE of sigma_w^2 and sigma_v^2 via the Kalman likelihood
print(res.params.round(3))                  # estimated process and measurement variances

filtered = res.filtered_state[0]            # E[z_t | x_1:t]   (filtering)
smoothed = res.smoothed_state[0]            # E[z_t | x_1:T]   (smoothing, uses the whole record)
forecast = res.get_forecast(steps=5).predicted_mean   # prediction, h = 1..5
print("filtered RMSE vs truth:", np.round(np.sqrt(np.mean((filtered - z)**2)), 3))
print("smoothed RMSE vs truth:", np.round(np.sqrt(np.mean((smoothed - z)**2)), 3))
print("5-step level forecast :", np.round(forecast, 3))

sigma2.irregular    2.196
sigma2.level         0.079
filtered RMSE vs truth: 0.498
smoothed RMSE vs truth: 0.371
5-step level forecast : [10.412 10.412 10.412 10.412 10.412]

Step back and see the architecture this section assembled. We separated every temporal system into a hidden state that evolves by its own dynamics and a noisy observation that is all we ever measure, and we made the state earn its name as a sufficient summary of the past for the future. We wrote the linear-Gaussian realization as a transition equation $\mathbf{z}_t = F\mathbf{z}_{t-1} + \mathbf{w}_t$ and an observation equation $\mathbf{x}_t = H\mathbf{z}_t + \mathbf{v}_t$, and gave each matrix its meaning, with $Q$ against $R$ as the trust dial. We distinguished the three inference tasks by their conditioning sets, filtering looking backward, prediction looking forward, smoothing looking both ways, and saw arithmetically why smoothing wins. We found the same state-space form hiding inside ARIMA, ETS, unobserved components, and dynamic factor models, so that one engine serves them all. And we built the local-level model from scratch and in two library lines, watching a hidden level emerge from five-to-one noise. Every piece now waits on one thing: the optimal recursion that actually computes these distributions. That recursion is the Kalman filter, and Section 7.2 derives it in full, turning the predict-update loop we sketched here into the central algorithm of classical state estimation, and into the ancestor of the recurrent and structured-state-space models that the rest of this book makes learnable.