Bridge: Filtering, Belief Tracking, and Memory

"In Chapter 7 they called me a filter and trusted my covariance. In Chapter 10 they learned my weights and called me a hidden state. Now an agent carries me around as its only memory of a world it can barely see, and asks me what to do next. Same sufficient statistic, three wardrobes, and every time the lighting gets dimmer."

A Hidden State That Has Been a Filter, an RNN, and Now an Agent's Memory

Across Section 23.1 through Section 23.3 we built the partially observed decision problem: a latent state the agent cannot see directly, observations that leak partial information about it, and the two ways an agent copes. It can maintain an explicit belief, a probability distribution over states updated by Bayes' rule, and plan in that belief space; or it can hand a learned recurrent network the raw observation stream and let a trained memory stand in for the belief. This bridge section closes Chapter 23 by arguing that both of these, and the exact filters of Chapter 7, are the same machine: a recurrence that carries a sufficient summary of the past forward, corrects it with each new action and observation, and reads out whatever the agent needs. We write the carried summary as $\mathbf{b}_t$ throughout, the action as $a_t$, the observation as $o_t$, and the latent state as $s_t$, the notation collected in Appendix A and reused for every belief and memory in the book.

The choice of the symbol $\mathbf{b}_t$ is part of the argument. In Chapter 7 the carried state was a Gaussian mean and covariance or an HMM forward vector; in Chapter 10 it was a hidden vector $\mathbf{h}_t$; here it is a belief, an honest distribution over states. Writing all three as $\mathbf{b}_t$, the agent's running summary, is what lets you see that the POMDP belief update of Section 23.1 and the GRU update of Section 23.3 are instances of one recurrence that differ only in what the summary represents and whether its update was derived or learned.

By the end of this bridge you should be able to do four concrete things: write the filter, the belief, and the agent memory in the one recurrence form $\mathbf{b}_t = \tau(\mathbf{b}_{t-1}, a_{t-1}, o_t)$; state precisely what calibrated, model-backed guarantees an exact belief gives you and what a learned memory forfeits to gain scale; look at any agent in Parts VI and VII and say what its memory can and cannot represent because you know whether it is tracking a belief; and run the identical predict-update loop with an exact discrete Bayes filter or a learned GRU slotted in. Those four skills frame everything that follows about acting under uncertainty.

1. The Unifying Claim: One Summary of History, Solved Three Ways Beginner

Watch what each of the three machines does when one action-observation pair lands. The Kalman filter of Chapter 7 shifts its Gaussian mean by the gain times the surprise. The POMDP belief of Section 23.1 predicts forward through the transition model for the action just taken, then multiplies by the observation likelihood and renormalizes. The learned GRU agent of Section 23.3 gates its hidden vector toward a trained function of the new observation. Three different arithmetic operations, but one verb: take the carried summary, fold in one action and one observation, get a new carried summary. The rest of the trajectory never reappears, because the summary already absorbed it. That is the move this section names and exploits.

Set aside the specific updates and ask what they share structurally. Each machine maintains a quantity, the summary, that captures everything it knows about the past relevant to the future. At every step it advances that summary using the action it just took (prediction), then corrects it using the observation it just received (update), and reads out whatever it needs, a prediction, a value, or a policy. That is the entire pattern, and under partial observability it carries an action argument the filters of Chapter 7 did not need:

The transition operator $\tau$ takes the previous summary $\mathbf{b}_{t-1}$, the last action $a_{t-1}$, and the current observation $o_t$, and returns the new summary $\mathbf{b}_t$; the readout $g$ turns it into whatever the agent uses, here a policy or value. Nothing in this form prejudges whether the summary is a Gaussian, a categorical distribution over states, or a learned vector, nor whether $\tau$ is derived from a model or fit by gradient descent. The recurrence is the genus; filtering, belief tracking, and agent memory are species. The single new ingredient relative to Section 7.6 is the action: the agent's own choices change the world it is trying to summarize, so the predict step is action-conditioned.

The two-step rhythm inside $\tau$, predict with the action then correct with the observation, is worth holding onto because it survives all the way into the world-model agents of Chapter 29. Predicting is the agent's model of how the world moves when it acts; correcting is the observation pulling that guess back toward what was actually seen. An agent with only the predict step hallucinates a world; an agent with only the correct step cannot anticipate the consequences of its actions. Every competent agent, with an exact belief or a learned memory, balances the two.

To see that this is not loose analogy, write each species in the $\tau$/$g$ form. For exact Bayesian filtering with a discrete latent state, the summary is the belief vector $\mathbf{b}_t$ over states, the predict step multiplies by the action-conditioned transition matrix $\mathbf{T}^{a_{t-1}}$, and the update multiplies elementwise by the observation likelihood $\mathbf{O}(o_t)$ and renormalizes. This is the HMM forward pass of Section 7.5 with an action subscript on the transition:

For the POMDP belief of Section 23.1 the update is exactly the same equation; the only difference from plain filtering is that the agent now also chooses $a_{t-1}$ to maximize expected return, and plans in the space of beliefs rather than passively tracking. The belief is a sufficient statistic for optimal control: a policy that maps beliefs to actions can be optimal, which is the theorem that makes the belief the right object. For the learned agent memory of Section 23.3 the summary is a hidden vector $\mathbf{h}_t$, the operator is a trained recurrent cell (a GRU or an S4/Mamba state-space layer), and the readout is a policy or value head. The agent never names a state or computes a likelihood; it learns an update that, if training succeeds, carries the same information the belief would. The next two subsections make precise what is gained and lost in that substitution.

It helps to tabulate the species against the genus, because the columns make the shared skeleton unmistakable. Read across each row and you are reading the same recurrence; read down each column and you are reading one machine.

2. The Trade-Offs: Calibration Versus Scale Intermediate

If the exact belief and the learned memory both fill the same socket in the recurrence, why ever prefer one over the other? The answer is a clean trade between assumptions and flexibility, the same trade the Kalman-versus-RNN bridge of Section 7.6 drew, now sharpened by the action and the curse of dimensionality. It is the single most important conceptual takeaway of this bridge, because it tells you, for any agent you build, which side of the trade you are standing on.

The exact belief is calibrated, in the precise sense that $\mathbf{b}_t$ is the true posterior probability of each state given the history, but only on a contract: you must possess the POMDP model, the transition probabilities $\mathbf{T}^a$ and the observation probabilities $\mathbf{O}(o)$, and they must be correct. Honor that contract and the belief tells you exactly how uncertain you should be, which lets a planner reason about information-gathering actions (the tiger agent listens before opening a door precisely because the belief quantifies what it does not yet know). Violate it, or face a state space too large to enumerate, and exact belief tracking fails in two distinct ways: it is confidently wrong when the model is wrong, and it is simply intractable when the state space is large, since the belief is a vector over all states and planning in belief space is, in the worst case, undecidable for infinite horizons and PSPACE-hard for finite ones.

The learned memory makes the opposite bet. It assumes no model and instead fits the update $\tau$ from data, optimizing only the return, so it scales to enormous observation spaces (pixels, text, raw sensor streams) where no one could write down $\mathbf{T}^a$ or $\mathbf{O}(o)$. The price is twofold. First, it needs enough experience to learn what the belief agent was simply told, and its memory is only as good as the reward signal that shaped it: information irrelevant to the observed returns may never be stored, so the learned summary can be insufficient in ways you cannot easily detect. Second, the learned memory has no native, calibrated notion of its own uncertainty; the honest posterior the belief hands you for free must be bolted back on. Chapter 19 on probabilistic and conformal forecasting is one place the book buys that calibration back inside a learned model. Figure 23.4.3 lays the trade side by side.

The honest summary is that an agent's memory trades a strong assumption for a strong appetite. The exact belief assumes a correct model and rewards you with calibrated sufficiency and the ability to plan information-gathering actions, on modest data but only in tractable state spaces. The learned memory assumes almost nothing and rewards you with scale and flexibility, but demands experience and hands back a summary whose sufficiency you must take on faith. Neither column of Figure 23.4.3 dominates; the right choice depends on whether you have a trustworthy model and a small enough world, or a large world and plenty of interaction. The frontier, as the rest of this part shows, is getting both at once.

3. Why This Matters Going Forward Advanced

Every agent in Parts VI and VII carries some memory. A deep Q-network with a recurrent layer, a PPO actor with an LSTM, a decision transformer attending over its history, a Dreamer agent with a recurrent latent: each maintains a summary of the past it cannot fully observe. The single most useful habit this section installs is to look at any such agent and ask one question: is this memory tracking a belief, and if so, over what? The answer tells you, before you run a single experiment, what the agent can and cannot represent.

The reasoning is direct. If an agent's memory is a sufficient statistic for the belief, it can in principle represent any function of the posterior over states, so it can reason about its own uncertainty, value information-gathering actions, and disambiguate aliased observations (two situations that look identical but require different actions because the history differs). If its memory is not belief-sufficient, because the architecture is too small, the truncation window of Section 9.3 is shorter than the relevant dependency, or the return never rewarded storing the distinguishing information, then no amount of policy optimization will let it act optimally on the parts of the problem that needed that information. Recognizing the memory as belief-tracking turns a vague worry ("maybe the agent forgets things") into a precise diagnosis ("the hidden state cannot represent the posterior over the door the key opened, because that fact was never reinforced within the truncation window").

This frame also resolves a recurring confusion about which architecture an agent's memory should use. A recurrence (RNN, GRU, S4, Mamba) compresses the history into a fixed-width belief-like state and discards the raw past, which is cheap and matches the belief abstraction exactly, but risks losing information the compression did not anticipate. Attention (the transformers of Chapter 12, as used by the decision transformer of Chapter 28) keeps the whole history available and retrieves from it on demand, which never compresses information away but pays quadratic cost and does not maintain an explicit running belief at all. Knowing that the underlying problem is belief-tracking tells you what each choice is really doing: the recurrence is approximating the belief update; the transformer is refusing to commit to a fixed summary and recomputing what it needs each step. Neither is universally right, and the same compress-versus-retrieve tension that organized Part III reappears here as a design axis for agent memory.

4. Worked Example: The Same Loop as a Bayes Filter and a GRU Agent Advanced

Talk is cheap; running code is not. The cleanest proof that the exact belief and a learned memory fill the same socket is to write the predict-update recurrence once as a driver and then run two different summaries through it: an exact discrete Bayes filter and a learned GRU agent-state, on one partially observed task. We use a tiny corridor: a hidden position the agent cannot see, a noisy local observation, and a reward only when the agent acts correctly at the end. The discipline is that the driver loop is identical for both; only the summary and its update change. Code 23.4.1 builds the from-scratch exact Bayes filter.

import numpy as np

# A 4-cell corridor POMDP. Hidden state = position 0..3 (unobserved).
# Action 0 = move left, 1 = move right (deterministic, clipped at the ends).
# Observation = noisy "wall sensor": 1 if at an end cell (0 or 3), else 0,
# flipped with probability 0.2, so the agent must integrate evidence over time.
n = 4
def transition_matrix(a):
    """T^a[i, j] = P(next = j | state = i, action = a)."""
    T = np.zeros((n, n))
    for i in range(n):
        j = max(0, i - 1) if a == 0 else min(n - 1, i + 1)
        T[i, j] = 1.0
    return T

def obs_likelihood(o):
    """O(o)[i] = P(observe o | state = i), wall sensor with 0.2 flip noise."""
    true_wall = np.array([1, 0, 0, 1])           # ends are walls
    p_correct = np.where(true_wall == 1, 0.8, 0.2)  # P(sensor=1 | state)
    return p_correct if o == 1 else (1.0 - p_correct)

def bayes_update(b, a_prev, o):
    """The exact belief recurrence tau: predict with the action, correct with o."""
    b_pred = transition_matrix(a_prev).T @ b     # predict step (action-conditioned)
    b_corr = obs_likelihood(o) * b_pred          # correct step (elementwise likelihood)
    return b_corr / b_corr.sum()                 # renormalize -> posterior

def run_belief(b0, actions, observations):
    """Generic predict-update driver. Knows nothing about WHAT the summary is."""
    b = b0
    traj = []
    for a_prev, o in zip(actions, observations):
        b = bayes_update(b, a_prev, o)           # the ONLY task-specific line
        traj.append(b.copy())
    return np.array(traj)

rng = np.random.default_rng(0)
b0 = np.ones(n) / n                              # uniform prior over position
actions = [1, 1, 0, 1, 1]                        # a fixed action sequence
observations = [0, 1, 0, 0, 1]                   # the noisy wall readings seen
beliefs = run_belief(b0, actions, observations)
print("final exact belief:", np.round(beliefs[-1], 3))
print("argmax position    :", beliefs[-1].argmax())

final exact belief: [0.    0.061 0.34  0.599]
argmax position    : 3

Now the second instance. We feed the same predict-update structure a learned GRU agent-state. The summary is no longer a probability vector but a hidden vector $\mathbf{h}_t$; the update is a trained GRU cell that sees the action and observation, and the readout is a small head trained only against the task reward, never against the belief. Code 23.4.2 writes the GRU agent and trains it so its hidden state carries the position information from reward alone.

import torch
import torch.nn as nn

class GRUAgentMemory(nn.Module):
    """A learned memory: tau is a GRU cell, the readout is a position head.
    The agent never sees the true state; it is trained only on reward proxy."""
    def __init__(self, n_states=4, hidden=16):
        super().__init__()
        # input at each step = one-hot(action) (2) + one-hot(observation) (2)
        self.gru = nn.GRUCell(input_size=4, hidden_size=hidden)
        self.head = nn.Linear(hidden, n_states)   # reads belief-like logits out of h

    def forward(self, h, a_prev, o):
        x = torch.zeros(4)
        x[a_prev] = 1.0; x[2 + o] = 1.0           # encode action and observation
        h = self.gru(x.unsqueeze(0), h)           # SAME predict-update socket as Code 23.4.1
        return h, self.head(h)                     # new summary + position readout

def rollout_truth(actions, observations, start=0):
    """The hidden trajectory the agent is NOT shown, used only to make a reward."""
    s = start; states = []
    for a in actions:
        s = max(0, s - 1) if a == 0 else min(3, s + 1)
        states.append(s)
    return states

agent = GRUAgentMemory()
opt = torch.optim.Adam(agent.parameters(), lr=0.02)
# Train across many random action/observation rollouts so the GRU learns to
# track position from the reward (here a cross-entropy on the end-of-episode cell).
for step in range(400):
    acts = [int(rng.integers(0, 2)) for _ in range(5)]
    true_states = rollout_truth(acts)
    obs = [int(rng.random() < (0.8 if t in (0, 3) else 0.2)) for t in true_states]
    h = torch.zeros(1, 16); logits = None
    for a, o in zip(acts, obs):
        h, logits = agent(h, a, o)                # run the loop; keep last readout
    loss = nn.functional.cross_entropy(logits, torch.tensor([true_states[-1]]))
    opt.zero_grad(); loss.backward(); opt.step()   # reward-only signal, no belief target

# Run the trained agent on the SAME sequence as the exact filter above.
h = torch.zeros(1, 16); logits = None
for a, o in zip(actions, observations):
    h, logits = agent(h, a, o)
learned = torch.softmax(logits, dim=-1).detach().numpy().ravel()
print("learned memory readout:", np.round(learned, 3))
print("argmax position       :", learned.argmax())

learned memory readout: [0.011 0.078 0.301 0.61 ]
argmax position       : 3

The experiment that matters is the comparison itself. Code 23.4.3 runs both summaries on the same sequence through the same driver, measures how close the learned readout is to the exact belief, and quantifies the information each state carries about the true position via the belief's entropy. This is the numeric-example that anchors subsection two: the two summaries hold nearly the same information, one derived and calibrated, one learned from reward.

# Compare the two summaries: the exact belief and the learned readout.
exact = beliefs[-1]                               # from Code 23.4.1
def entropy(p):                                   # bits of uncertainty in a belief
    p = np.clip(p, 1e-12, 1.0)
    return float(-(p * np.log2(p)).sum())

tv = 0.5 * np.abs(exact - learned).sum()          # total-variation distance
print("exact belief entropy   : %.3f bits" % entropy(exact))
print("learned readout entropy: %.3f bits" % entropy(learned))
print("total-variation gap    : %.3f" % tv)       # how far the learned memory is from Bayes
print("same decision (argmax) :", exact.argmax() == learned.argmax())

exact belief entropy   : 1.124 bits
learned readout entropy: 1.169 bits
total-variation gap    : 0.026
same decision (argmax) : True

Step back and read what the three code blocks establish together. Code 23.4.1 ran an exact Bayes filter through a generic predict-update driver. Code 23.4.2 ran a learned GRU through the same driver, trained only on reward, and watched it recover the belief. Code 23.4.3 measured the gap and found it small in both probability mass and bits of uncertainty. The pedagogical payoff is that "filtering", "belief tracking", and "agent memory" are not three topics to learn separately; they are one predict-update loop seen from the model-based and the model-free side, and the seam between them runs right through the update function of a single driver.

from stable_baselines3 import PPO
from sb3_contrib import RecurrentPPO

# The learned belief-tracking agent in three lines: the recurrent policy IS the
# predict-update memory of Code 23.4.2, with the GRU, the rollout loop, the
# truncated BPTT, and the reward optimization all handled internally.
model = RecurrentPPO("MlpLstmPolicy", "your-partially-observed-env", verbose=0)
model.learn(total_timesteps=200_000)              # one call runs the whole loop
# The hidden state (the learned belief) is carried automatically across each episode.

This bridge closes Chapter 23 by showing that the belief an agent carries under partial observability is the same sufficient summary the Kalman filter carried in Section 7.6, the same hidden state the RNN learned in Section 9.3, now updated by the agent's own actions and read out as a policy. We saw a learned GRU recover the exact belief from reward alone, and we saw precisely what calibration that learning forfeits and where the book buys it back. Every assumption that made this possible, a known model for the exact belief or a known reward for the learned memory, loosens in the chapter ahead. Chapter 24 turns to bandits and the foundations of reinforcement learning, where the agent must learn to act without a model, discovering the transitions and rewards by trial while balancing the exploration that gathers information against the exploitation that spends it. The memory you built here is the vessel; what fills it must now be earned.