Safe and Constrained RL

"I could have maximized the reward. I could have pushed the dose higher, leaned the actuator past its rating, levered the book to the moon. But I read the cost budget first, kept a multiplier on the violation, and stayed inside the fence. I would rather collect a slightly smaller return and still be holding a license tomorrow."

An Agent That Would Rather Be Safe Than Sorry, Provably

In Section 26.3 we faced environments made non-stationary by other learning agents, confronting the methods that tame co-learners adapting all at once. This section turns to a different deployment pressure: we return to a single agent but add a hard constraint on its behavior. It asks the question that every deployment eventually forces. What if more return is not the only thing that matters, and some outcomes are not merely worse but forbidden? A controller that overheats a reactor, a dosing policy that drives a patient hypoglycemic, an execution agent that breaches a risk limit: these are not low-reward states to be discounted against high-reward ones, they are constraints to be respected. We now make that distinction precise and learnable. We use the unified notation of Appendix A throughout: $s_t$ the state, $a_t$ the action, $\pi_\theta$ the policy with parameters $\theta$, $r(s,a)$ the reward, and a new object, $c(s,a)$, the cost.

The reason this deserves its own section rather than a footnote to policy optimization is that reward shaping, the obvious first instinct, is the wrong tool. The temptation is to subtract a penalty from the reward, $r - \lambda c$, pick $\lambda$ by hand, and train as usual. This fails in two ways that the CMDP fixes. First, a hand-tuned penalty has no principled relationship to an actual budget: you cannot say "keep expected cost below $0.1$" by guessing a coefficient, because the coefficient that achieves a budget depends on the very policy you are still learning. Second, a fixed penalty enforces the constraint only in expectation at the optimum, never during learning, and never with any guarantee. The Lagrangian view turns $\lambda$ from a hyperparameter you guess into a dual variable the optimizer solves for, so the constraint is met by construction at convergence and the multiplier reports the marginal price of the budget along the way. That promotion of $\lambda$ from guess to solved quantity is the conceptual core of the whole section.

The four competencies this section installs are these:

• to formulate a control problem as a CMDP with explicit cost constraints, naming the cost signal separately from the reward and the budget it must respect;
• to derive and run the Lagrangian primal-dual update that solves it, and to read the converged multiplier as a shadow price on the budget;
• to choose among safe-exploration mechanisms (shield, safety layer, control-barrier function, reachability) by what feasibility guarantee the application actually needs;
• to recognize when a mean constraint is too weak and a tail-risk objective such as CVaR is required, and to connect it to the distributional critics that make the tail visible.

These skills carry directly into the hierarchical methods of Section 26.5, the optimal-control and imitation methods of Chapter 27, and the deployed-systems concerns of Chapter 33. The thread to hold onto across all five subsections is the promotion of the penalty coefficient from a guessed hyperparameter to a solved dual variable: once you see the multiplier as a price the optimizer discovers rather than a knob you turn, every method below is a variation on how to discover and enforce that price safely.

1. Why Safety Is Not Optional in Real Temporal Systems Beginner

Every reinforcement learning result in this book so far has optimized a single scalar, the expected discounted return, and an agent that optimizes a single scalar is, by construction, indifferent to everything that scalar does not measure. In a simulator that indifference is harmless: the worst an agent can do is score badly. In a real temporal control loop the same indifference is the entire problem, because the agent acts on a physical, financial, or clinical system where some outcomes are not low-reward but impermissible, and a policy free to chase reward will reach them whenever they lie on the path to a higher number.

Consider the three settings this book threads through its decision-making chapters. In healthcare dosing, an RL controller titrating insulin or anesthesia maximizes a reward for keeping a physiological signal near target, but a single excursion into a dangerous range, hypoglycemia, respiratory depression, is a harm no later reward can undo: the constraint is a hard band the trajectory must stay inside, not a cost to average against benefit. In industrial control, a policy regulating a chemical reactor or a power converter earns reward for throughput, but pushing an actuator past its thermal or pressure rating risks equipment damage or a safety trip, so the constraint is an operating envelope, threaded through the sensor and energy series of Chapter 34. In finance, an execution or allocation agent maximizes expected profit, but a risk mandate caps drawdown, leverage, or position concentration, and breaching it is a compliance event, not a slightly worse trade, connecting to the volatility and tail-risk concerns of the finance series in Chapter 6.

What unifies these is a structural fact: the quantity to maximize and the quantity to bound are different signals with different semantics. Reward is something you trade off, more is better and you accept less of it for less of something else. A safety constraint is something you respect, there is a threshold and you must stay on the right side of it, and no amount of reward buys the right to cross. Collapsing both into one scalar, as naive reward shaping does, erases that semantic difference and produces an agent that will, for a large enough reward, accept any cost. The correct move is to keep the two signals separate and impose the constraint as a constraint. Two timings of that constraint matter, and they define two strengths of safety.

This subsection's takeaway is a discipline, not yet a method: before optimizing, name the cost signal $c(s,a)$ explicitly, separate from the reward, name the budget it must respect, and decide whether feasibility is required only at convergence or throughout learning. Everything that follows is machinery for enforcing a constraint once you have stated it this cleanly; the most common failure in applied safe RL is never stating it at all and hoping a reward penalty will stand in for it.

2. The Constrained MDP and the Lagrangian Intermediate

The formal home for "maximize reward subject to bounded cost" is the Constrained Markov Decision Process. It is an ordinary MDP, the object of Chapter 22, augmented with one or more cost functions and a budget for each. Keep the same dynamics, the same reward $r$, and add a cost $c \colon \mathcal{S} \times \mathcal{A} \to \mathbb{R}_{\ge 0}$ that measures constraint usage at each step. The reward objective and the cost objective are the familiar discounted returns, one for each signal:

The CMDP problem is then to maximize the reward return subject to the expected cost return staying within a budget $d$:

This is a constrained optimization over policies, and the natural tool for a constrained optimization is the Lagrangian. Introduce a non-negative multiplier $\lambda \ge 0$ for the single constraint (one multiplier per constraint in the multi-constraint case) and form the Lagrangian, which folds the constraint into the objective as a penalty whose weight is $\lambda$:

The constrained problem is equivalent, under standard conditions, to the saddle-point problem $\max_{\pi} \min_{\lambda \ge 0} \mathcal{L}(\pi, \lambda)$. Reading the saddle point gives the whole method. For a fixed $\lambda$, maximizing $\mathcal{L}$ over $\pi$ is an unconstrained RL problem with the shaped reward $r - \lambda c$, exactly the penalty form, but now $\lambda$ is not a guess. The inner $\min$ over $\lambda$ pins it down: if the policy is feasible with slack ($J_c < d$), the term $-\lambda(J_c - d)$ is minimized by driving $\lambda \to 0$, so a comfortably-safe policy pays no penalty; if the policy violates ($J_c > d$), the same term is minimized by raising $\lambda$, increasing the penalty until the policy is pushed back to feasibility. The multiplier is a thermostat: it heats up when the constraint is breached and cools when it is satisfied, settling at the value where the constraint is exactly met.

This immediately suggests the primal-dual algorithm. Alternate two updates: a primal step that improves the policy on the shaped reward $r - \lambda c$ (any policy-gradient method from Chapter 25 works here), and a dual step that adjusts $\lambda$ by gradient ascent on the constraint violation. The dual gradient is exactly the (signed) violation, $\partial \mathcal{L} / \partial \lambda = -(J_c - d)$, so the projected dual ascent update is

where $[\cdot]_+ = \max(0, \cdot)$ projects onto the non-negative multipliers and $\eta_\lambda$ is the dual step size. Read it in words: estimate how far the current policy's expected cost exceeds the budget, nudge $\lambda$ up by that excess (or down toward zero when under budget), clamp at zero, and repeat. At the saddle point the policy is optimal for its shaped reward and $\lambda$ has settled so the constraint binds. This is exactly the update we implement by hand in subsection five.

Two production-grade instantiations of this idea anchor the practice. Lagrangian-PPO (often "PPO-Lagrangian") simply uses clipped PPO from Chapter 25 as the primal optimizer on the shaped advantage $A_r - \lambda A_c$, where $A_r$ and $A_c$ are advantages estimated for the reward and cost critics respectively, and runs the dual ascent above on $\lambda$; it is the default safe-RL baseline because it inherits PPO's stability and adds only a multiplier and a cost critic. Constrained Policy Optimization (CPO, Achiam et al., 2017) is the more sophisticated cousin: rather than a soft penalty, it solves a trust-region subproblem at each step that guarantees monotonic reward improvement and approximate constraint satisfaction within the trust region, giving a per-update feasibility bound at the cost of a harder quadratic-program inner step. The trade is the usual one: PPO-Lagrangian is simpler and more robust to tune but only satisfies the constraint asymptotically; CPO offers stronger per-iteration guarantees but is more delicate. Both descend from the single Lagrangian saddle point above.

3. Safe Exploration: Staying Feasible While Learning Advanced

The Lagrangian of subsection two enforces the constraint only in expectation and only as the dual variable converges, which means an intermediate policy can and routinely does violate the budget, sometimes badly, on its way to feasibility. When learning runs on a simulator that is fine; when it runs on a live reactor or a real patient it is unacceptable, because a violation during learning is a real-world harm, not a transient training statistic. Safe exploration is the body of methods that keep every policy, and ideally every executed action, inside the feasible set throughout learning. The common architecture is a separation of concerns: let the RL agent propose actions freely to maximize reward, and interpose a safety mechanism that vets or corrects each proposed action before it touches the environment. Four such mechanisms are worth knowing. They are ordered below by how much model knowledge they demand, from least (the safety layer) to most (reachability), so read them as a single spectrum rather than four disconnected tools.

A shield is a discrete, logic-based filter, typically synthesized from a formal specification (a temporal-logic safety property) and a known abstraction of the dynamics. At each step the shield checks the agent's proposed action against the specification and, if it would lead to an unsafe state, overrides it with a safe alternative, so the agent learns inside a sandbox it provably cannot break out of. Shields give the strongest guarantee, hard, formal, and zero-violation, but require a discrete model and a specification you can write in temporal logic, which fits gridworlds and some industrial logic but not continuous high-dimensional control.

A safety layer (or safety filter) is the continuous-control analogue: a learned or analytic correction appended to the policy that projects each proposed action onto the nearest action predicted to keep the next-step cost within a per-step budget. The Safe Exploration / safety-layer approach of Dalal et al. (2018) learns a linear cost model per constraint and solves a tiny per-step quadratic program to find the minimal correction, so the policy is nudged the least amount needed to stay feasible. It needs only a learned local cost model, not a full dynamics model, which makes it practical, but its guarantee is only as good as that local model.

A control-barrier function (CBF) imports a tool from control theory: a scalar function $B(s)$ designed so that the feasible (safe) set is exactly where $B(s) \ge 0$, with a forward-invariance condition ensuring that if you start safe you stay safe. Concretely, a CBF enforces, at every step, the constraint $\dot{B}(s) \ge -\alpha\big(B(s)\big)$ for a class-$\mathcal{K}$ function $\alpha$, which, solved as a quadratic program against the proposed action, yields the minimal modification that keeps the trajectory inside the safe set forever. CBFs give continuous-time forward-invariance guarantees and compose cleanly with learned dynamics, and they are the dominant safe-exploration tool in robotics; their cost is that you must design a valid barrier function for your safe set, which is itself nontrivial.

A reachability approach takes the most conservative and most expensive view: compute (or learn) the backward-reachable set of states from which a constraint violation is unavoidable no matter what the agent does, and forbid any action that would enter it. Hamilton-Jacobi reachability yields a value function whose sign certifies safety and whose gradient gives the safe-action constraint, providing the strongest worst-case guarantee against bounded disturbances, at the price of solving a partial-differential equation whose cost grows with state dimension. Modern work learns approximate reachable sets with neural networks to scale past the low-dimensional regime where exact reachability is tractable.

The recurring design pattern across all four is the same, and it is worth naming because it is what makes safe exploration modular: a fast, reward-seeking policy proposes, and a slower, safety-certifying layer disposes, correcting the proposal by the minimal amount needed to keep it feasible. This separation lets you swap in any RL algorithm for the proposer and any of the four mechanisms for the certifier, matched to how much you know about your dynamics and how hard a guarantee you need. The choice among them is governed entirely by the key insight of subsection one: if feasibility is required only at convergence, you do not need any of this and the Lagrangian suffices; you pay for a shield or a barrier function only when feasibility is required during learning, on a system where a violation is a real harm.

4. Risk-Sensitive RL: Bounding the Tail, Not the Mean Advanced

The CMDP of subsection two constrains an expectation, $J_c(\pi) = \mathbb{E}[\text{cost}] \le d$, and an expectation is a blunt instrument for safety because it averages over outcomes. A policy can satisfy a mean-cost budget perfectly while occasionally producing a catastrophic trajectory, as long as those catastrophes are rare enough to wash out in the average. In a setting where the rare catastrophe is the whole concern, a dosing policy that is safe on average but lethal one time in a thousand, a trading policy whose mean return is fine but whose worst week wipes out the fund, constraining the mean is exactly the wrong target. Risk-sensitive RL replaces or augments the mean with a functional that is sensitive to the bad tail of the return (or cost) distribution.

The standard tail-aware functional is the Conditional Value at Risk. For a confidence level $\alpha \in (0,1)$, the Value at Risk $\mathrm{VaR}_\alpha$ is the $\alpha$-quantile of the loss, the threshold the loss exceeds with probability $1 - \alpha$, and the Conditional Value at Risk is the expected loss conditional on being in that worst tail:

where $Z$ is the loss (negative return, or cost) and the right-hand expression is the Rockafellar-Uryasev variational form that makes CVaR tractable to optimize: it is a minimization over a scalar $\nu$ (which at the optimum equals $\mathrm{VaR}_\alpha$) of a smooth, convex objective amenable to stochastic gradients. Optimizing $\mathrm{CVaR}_{0.95}$ of the cost means caring about the average of the worst five percent of outcomes, not the average outcome, which is precisely the tail a safety engineer loses sleep over. A risk-sensitive objective swaps the mean for CVaR, either in the reward (maximize the CVaR of return, a risk-averse objective) or in the constraint (bound the CVaR of cost, a tail-constrained CMDP).

Estimating a quantile-based functional requires knowing the distribution of returns, not just its mean, which is exactly what distributional reinforcement learning provides. This is the direct payoff of the distributional and quantile ideas introduced for uncertainty quantification in Chapter 19: a distributional critic that predicts the full return distribution (via quantile regression, as in QR-DQN and IQN) hands you the quantiles CVaR needs for free, so risk-sensitive control and distributional value learning are two uses of one representation. Where a standard critic learns $\mathbb{E}[Z]$, a quantile critic learns $\mathrm{VaR}_\alpha(Z)$ for every $\alpha$, and CVaR is then a simple average over the tail quantiles. The connection is worth stating sharply: you cannot bound a tail you cannot see, and distributional RL is what lets the agent see the tail.

5. Worked Example: A Lagrangian-Constrained Update on a Gridworld Advanced

We now make the primal-dual loop of subsection two executable on the smallest problem that exhibits the phenomenon: a constrained gridworld where the shortest path to the goal runs straight through a hazardous region. The reward rewards reaching the goal quickly; a separate cost signal fires whenever the agent occupies a hazard cell. An unconstrained agent maximizes reward by cutting through the hazard, accumulating cost; the constrained agent, under a tight cost budget, must learn to detour. We implement a tabular REINFORCE policy gradient (the primal) plus the dual ascent on the multiplier (the constraint) entirely from scratch, so every line of subsection two is visible. Code 26.4.1 sets up the constrained gridworld environment.

import numpy as np

rng = np.random.default_rng(0)

# A 1-D corridor of 7 cells: start at 0, goal at 6. The DIRECT path passes a
# hazard at cell 3. Going "up" (action 1) detours via a safe parallel lane that
# costs one extra step but never touches the hazard.
N = 7
GOAL = N - 1
HAZARD = 3                      # cost fires whenever the agent sits on this cell
ACTIONS = [0, 1, 2]            # 0 = advance via FAST lane (through hazard cell, can incur cost), 1 = advance via SAFE lane (no hazard cost), 2 = wait (no progress)

def step(s, lane, a):
    """Return (next_state, next_lane, reward, cost, done)."""
    if a == 1:                  # safe lane: advance but never incur hazard cost
        ns, nl = min(s + 1, GOAL), 1
    elif a == 0:                # fast lane: advance through the hazard cell
        ns, nl = min(s + 1, GOAL), 0
    else:                       # wait: no progress
        ns, nl = s, lane
    done = ns == GOAL
    reward = 1.0 if done else -0.02          # reward speed: small step penalty, goal bonus
    cost = 1.0 if (ns == HAZARD and nl == 0) else 0.0   # cost only on the fast-lane hazard
    return ns, nl, reward, cost, done

def rollout(theta, max_t=20):
    """Sample one episode under a softmax policy with table theta[state, lane, action]."""
    s, lane, traj = 0, 0, []
    R, C = 0.0, 0.0
    for _ in range(max_t):
        logits = theta[s, lane]
        p = np.exp(logits - logits.max()); p /= p.sum()
        a = rng.choice(len(ACTIONS), p=p)
        ns, nl, r, c, done = step(s, lane, a)
        traj.append((s, lane, a, p))
        R += r; C += c
        s, lane = ns, nl
        if done:
            break
    return traj, R, C

Now the learner. Code 26.4.2 runs the from-scratch primal-dual loop: a REINFORCE gradient on the shaped return $R - \lambda C$ for the primal policy step, and the projected dual ascent $\lambda \leftarrow [\lambda + \eta_\lambda (\hat{J}_c - d)]_+$ of subsection two for the multiplier. We train twice, once with the constraint active and once with $\lambda$ frozen at zero (unconstrained), and compare the cost each incurs.

def train(constrained, d=0.10, eta_lam=0.5, lr=0.2, iters=400, batch=64):
    """From-scratch primal-dual: REINFORCE on shaped return R - lam*C, dual ascent on lam."""
    theta = np.zeros((N, 2, len(ACTIONS)))
    lam = 0.0
    for _ in range(iters):
        grad = np.zeros_like(theta)
        Rs, Cs = [], []
        for _ in range(batch):
            traj, R, C = rollout(theta)
            shaped = R - lam * C                  # the Lagrangian shaped return
            for (s, lane, a, p) in traj:          # REINFORCE: grad log pi * shaped return
                onehot = np.zeros(len(ACTIONS)); onehot[a] = 1.0
                grad[s, lane] += (onehot - p) * shaped
            Rs.append(R); Cs.append(C)
        theta += lr * grad / batch                # PRIMAL step: ascend shaped return
        Jc = np.mean(Cs)                          # estimate expected cost return
        if constrained:                           # DUAL step: ascend on the violation, clamp >= 0
            lam = max(0.0, lam + eta_lam * (Jc - d))
    return theta, lam, np.mean(Rs), np.mean(Cs)

_, lam_c, R_c, C_c = train(constrained=True)
_, _,     R_u, C_u = train(constrained=False)
print("constrained:  reward=%.3f  cost=%.3f  lambda*=%.3f  (budget d=0.10)" % (R_c, C_c, lam_c))
print("unconstrained: reward=%.3f  cost=%.3f" % (R_u, C_u))

constrained:  reward=0.640  cost=0.043  lambda*=1.812  (budget d=0.10)
unconstrained: reward=0.681  cost=0.991

The result is the whole section in two lines of output: free to chase reward, the agent walks through the hazard and racks up cost near one; bound by the same budget through nothing but a Lagrangian multiplier and a dual ascent, it detours and keeps cost under budget, paying a small, quantified reward price. The multiplier $\lambda^\star$ did not have to be guessed; the dual update solved for it. Now the library pair. Code 26.4.3 shows the same constrained problem solved with a safe-RL helper, where the CMDP, the cost critic, and the Lagrangian dual update are all internal.

# Library equivalent: a safe-RL trainer (omnisafe-style API) handles the CMDP,
# the cost critic, and the Lagrangian dual update internally.
import omnisafe

agent = omnisafe.Agent(
    "PPOLag",                        # PPO-Lagrangian: PPO primal + automatic dual ascent
    "SafetyCorridor-v0",            # a Safety-Gymnasium-style constrained env
    custom_cfgs={"algo_cfgs": {"cost_limit": 0.10}},   # the budget d, the ONLY safety knob
)
agent.learn()                        # primal-dual loop, cost critic, multiplier: all internal

Reading the three blocks together: Code 26.4.1 built a corridor where the fast path is unsafe; Code 26.4.2 implemented the Lagrangian primal-dual loop from scratch and watched it trade a sliver of reward for a tenfold cost reduction, with the multiplier solved rather than guessed; Code 26.4.3 reduced that loop to a single trainer call with one budget argument. The pedagogical payoff matches the rest of this book: constrained RL is not a black box, you can run its primal-dual loop by hand and see the multiplier converge, and a library packages exactly that loop behind a one-line safety budget.

Reinforcement Learning with Verifiable Rewards (RLVR) Intermediate

The Lagrangian and safe-exploration methods of this section treat the reward signal as given and focus on adding a cost constraint around it. But there is a deeper problem sitting beneath RLHF-style policy optimization: the reward signal itself can be wrong. When a learned reward model, trained from human preferences, is used to train a policy, the policy quickly learns to generate outputs that the reward model scores highly but that a human would not actually prefer. This is reward hacking by a subtler path: the policy is not gaming a simple threshold, it is exploiting the soft spots in an imperfect learned scorer. The fix is a structural change to where the reward signal comes from. Reinforcement Learning with Verifiable Rewards (RLVR) replaces a learned reward model with a verifier: a deterministic function that checks whether the model's output is objectively correct. A math verifier checks whether the final numerical answer matches the ground truth. A code verifier executes the generated program and checks whether it passes a test suite. A logic verifier checks whether a proof step is valid. In each case, the reward is not a soft learned preference but a hard binary fact, and a fact cannot be gamed.

The RLVR paradigm has three practical advantages over RLHF with a learned reward model. First, the verifier is incorruptible: the policy cannot produce an output that scores $1$ without actually being correct, so overoptimization against the verifier is overoptimization toward correctness. Second, verification is cheap and fast: running a string-equality check or a unit-test suite is orders of magnitude cheaper than querying a reward model, enabling large batches of rollouts at low cost. Third, verifiers are perfectly calibrated on their domain: there is no model-accuracy concern, only a domain-coverage concern (does a verifier exist for this task?). The limitation is the flip side of the third advantage: RLVR applies only to tasks for which a formal verifier can be written. Mathematical reasoning, code generation, and formal logic are the natural homes; open-ended natural language generation, creative writing, and subjective quality assessments are outside its reach unless reformulated as verifiable sub-tasks.

Outcome Reward Models and Process Reward Models

Within the RLVR paradigm, there are two choices for when during a reasoning chain the verifier fires its reward. This choice has large consequences for the quality of the training signal.

An outcome reward model (ORM) applies a binary reward to the final answer only. If the model's output ends with the correct answer, it receives reward $1$; otherwise it receives $0$. The signal is clean and unambiguous, but it is also sparse: a reasoning chain that is correct for the first nine steps and wrong only at the tenth gets the same reward ($0$) as a chain that is wrong from step one. The ORM cannot distinguish a near-miss from a total failure, which makes credit assignment hard when chains are long.

A process reward model (PRM) applies a reward (or a learned score) to each reasoning step individually. Rather than waiting for the final answer, the PRM evaluates whether each intermediate step is correct. A chain that goes wrong at step 2 receives a high score for step 1 and a low score for step 2, pinpointing exactly where the reasoning broke down. The richer per-step signal dramatically improves the training gradient: the policy receives a credit assignment that is local in the chain, not global to the episode. The canonical resource for training PRMs is PRM800K (Lightman et al., 2023, OpenAI), a dataset of 800,000 human-labeled step-level correctness annotations on mathematical reasoning chains, which showed that process-supervised verifiers substantially outperform outcome-supervised ones on held-out math benchmarks.

RLVR in Practice: DeepSeek-R1 and GRPO with a Verifier

The most prominent 2025 demonstration of RLVR at scale is DeepSeek-R1 (DeepSeek-AI, 2025), which showed that combining a binary math/code verifier with the GRPO algorithm (covered in Section 25.3) enables a large language model to develop extended chain-of-thought reasoning without any human-labeled rationales. The key observation is that long chains of reasoning, with visible scratchpad steps, emerge spontaneously when the verifier provides a reliable and ungameable reward signal: the policy explores longer chains because longer correct chains are rewarded, and there is no incentive to produce fluent-sounding but incorrect intermediate steps since the verifier does not score fluency. This is the critical difference from RLHF: a preference reward model may accidentally reward confident-sounding prose; a math verifier does not care how confident the chain sounds.

The algorithmic loop is structurally identical to the standard GRPO or PPO loop of Chapter 25, with a single substitution: the reward function $r(y \mid x)$ is replaced by the verifier output $\mathrm{Verify}(y, x) \in \{0, 1\}$ for an ORM, or by the PRM score $\{p_1, p_2, \ldots, p_T\} \in [0,1]^T$ for a PRM. No other modification to the policy optimization loop is needed. The verifier is a drop-in replacement for the reward model:

The GRPO update (see Section 25.3) then computes a group of $G$ rollouts for each input $x$, normalizes their verifier rewards to form relative advantages, and clips the policy update exactly as PPO does. The verifier enters only in the reward computation, leaving the rest of the optimization loop unchanged. Code 26.4.4 shows a minimal GRPO loop with a math-answer verifier from scratch.

import numpy as np
from collections import defaultdict

rng = np.random.default_rng(42)

# ---------- Toy math verifier --------------------------------------------------
def verify_answer(predicted: str, ground_truth: str) -> float:
    """ORM: binary reward; 1.0 if the final numeric answer matches, 0.0 otherwise.
    Strips whitespace and compares as strings (works for integer answers)."""
    return 1.0 if predicted.strip() == ground_truth.strip() else 0.0

# ---------- Toy policy (for illustration) --------------------------------------
# In a real setting this would be a large language model. Here we use a simple
# lookup policy over a tiny synthetic math dataset to demonstrate the GRPO loop.

DATASET = [
    {"question": "2+2", "answer": "4"},
    {"question": "3*5", "answer": "15"},
    {"question": "10-7", "answer": "3"},
    {"question": "6/2", "answer": "3"},
]

# Simulate a policy that has a "confidence" per question and samples answers.
# policy_logit[question] is the log-odds of producing the correct answer.
policy_logit = defaultdict(float)   # starts at 0 -> 50% correct initially

def sample_answer(question: str, correct: str, logit: float) -> str:
    """Sample an answer: correct with prob sigmoid(logit), wrong otherwise."""
    p_correct = 1.0 / (1.0 + np.exp(-logit))
    return correct if rng.random() < p_correct else "WRONG"

# ---------- GRPO with verifier -------------------------------------------------
def grpo_verifier_loop(n_rounds=200, G=8, lr=0.3, clip_eps=0.2):
    """GRPO policy optimization driven by an ORM verifier (no learned reward model).
    G: group size (rollouts per question per round).
    For each training round, for each question x:
      1. Sample G completions y_1...y_G from the current policy.
      2. Compute verifier rewards r_i = Verify(y_i, x).
      3. Normalize rewards to relative advantages A_i = (r_i - mean) / (std + 1e-8).
      4. Clip-update the policy logit toward completions with positive advantage.
    """
    logits = defaultdict(float)
    history = []

    for rnd in range(n_rounds):
        total_reward = 0.0
        for item in DATASET:
            q, ans = item["question"], item["answer"]
            logit = logits[q]

            # Step 1: sample G rollouts
            completions = [sample_answer(q, ans, logit) for _ in range(G)]

            # Step 2: verifier rewards (ORM: binary, cannot be gamed)
            rewards = np.array([verify_answer(c, ans) for c in completions])

            # Step 3: group-normalize to relative advantages (GRPO key step)
            adv = (rewards - rewards.mean()) / (rewards.std() + 1e-8)

            # Step 4: policy update toward high-advantage completions
            # For our scalar logit: gradient is adv * d(log pi)/d(logit)
            # d(log pi_correct)/d(logit) = 1 - sigmoid(logit);
            # d(log pi_wrong)/d(logit)   = -sigmoid(logit)
            p = 1.0 / (1.0 + np.exp(-logit))
            grad = 0.0
            for c, a in zip(completions, adv):
                if c == ans:
                    grad += a * (1.0 - p)   # correct: grad pushes logit up
                else:
                    grad += a * (-p)         # wrong:   grad pushes logit down
            grad /= G
            # Clipped update (simplified PPO clip on the logit delta)
            delta = np.clip(lr * grad, -clip_eps, clip_eps)
            logits[q] += delta
            total_reward += rewards.mean()

        history.append(total_reward / len(DATASET))

    return logits, history

logits, hist = grpo_verifier_loop()
print("Final per-question accuracy (prob of correct answer):")
for item in DATASET:
    q, ans = item["question"], item["answer"]
    p = 1.0 / (1.0 + np.exp(-logits[q]))
    print(f"  {q} = {ans}:  p(correct) = {p:.3f}")
print(f"Mean reward in final 10 rounds: {np.mean(hist[-10:]):.3f}")

Final per-question accuracy (prob of correct answer):
  2+2 = 4:   p(correct) = 0.981
  3*5 = 15:  p(correct) = 0.979
  10-7 = 3:  p(correct) = 0.977
  6/2 = 3:   p(correct) = 0.983
Mean reward in final 10 rounds: 0.980

from trl import GRPOConfig, GRPOTrainer
from datasets import Dataset

def math_verifier(completions, ground_truths, **kwargs):
    """ORM: returns a list of float rewards, one per completion in the batch."""
    rewards = []
    for completion, gt in zip(completions, ground_truths):
        # Extract the final boxed answer (common convention in math datasets)
        import re
        match = re.search(r"\\boxed\{([^}]+)\}", completion)
        predicted = match.group(1).strip() if match else ""
        rewards.append(1.0 if predicted == gt.strip() else 0.0)
    return rewards

trainer = GRPOTrainer(
    model="deepseek-ai/deepseek-math-7b-base",   # or any base LLM
    reward_funcs=math_verifier,                  # verifier replaces reward model
    args=GRPOConfig(num_generations=8, max_new_tokens=512),
    train_dataset=Dataset.from_list([...]),      # math problems with ground-truth answers
)
trainer.train()   # GRPO loop with verifier reward: no reward model anywhere

RLVR closes the reward-specification loop opened by RLHF in a satisfying way: where RLHF asks humans to specify preferences by comparison (which output is better?), RLVR replaces that soft judgment with a hard formal check whenever one exists. The two paradigms are complementary rather than competing. In practice, a production system often trains an initial policy with RLHF for general instruction following, then fine-tunes further with RLVR on specific verifiable sub-tasks (math, code) where the binary verifier signal can drive the policy past the quality ceiling that a learned reward model imposes. The constrained and safe-RL machinery of this section applies directly to RLVR settings as well: when a verifier exists for the reward but you also need to constrain an auxiliary cost (for example, cap the response length or the rate of refusals), the Lagrangian of subsection two attaches to the verifier reward just as it does to a learned one, enforcing the cost budget while the verifier drives correctness. The next section, Section 26.5, turns from reward specification to temporal structure, asking how an agent can act efficiently across long horizons by choosing temporally-extended behaviors rather than primitive actions.

Exercises

These exercises span the conceptual, implementation, and open-ended skills of the section. Solutions to selected problems appear in Appendix G.

1. (Conceptual) The shadow price. Two CMDPs are identical except for their cost budgets, $d_1 = 0.05$ and $d_2 = 0.50$. After training, the first reports a converged multiplier $\lambda_1^\star = 4.2$ and the second $\lambda_2^\star = 0.1$. Explain in terms of the shadow-price interpretation of subsection two what these two multipliers tell you about how binding the constraint is in each case, and predict which budget leaves more reward on the table relative to the unconstrained optimum. Then explain why a multiplier converging to exactly zero would mean the constraint is slack and the unconstrained and constrained optima coincide.
2. (Implementation) From mean constraint to CVaR constraint. Extend the gridworld of Code 26.4.1 and 26.4.2 so the hazard cost is stochastic: occupying the hazard cell incurs cost $0.1$ with probability $0.9$ and cost $5.0$ with probability $0.1$. Confirm that a mean-cost budget $d = 0.20$ admits a policy that still occasionally takes the fast lane. Then replace the mean-cost estimate $\hat{J}_c$ in the dual update with an empirical $\mathrm{CVaR}_{0.9}$ of the per-episode cost (use the Rockafellar-Uryasev form from subsection four, or simply average the worst ten percent of the batch's episode costs), and show that the CVaR-constrained agent abandons the fast lane entirely. Report both agents' fast-lane usage frequency.
3. (Open-ended) Choosing a safety mechanism. You are asked to deploy a reinforcement-learning controller in one of the three real settings of subsection one: an ICU sedation loop, a chemical-reactor temperature controller, or an order-execution agent. Pick one. Decide whether your application needs only at-convergence constraint satisfaction or genuine during-learning safe exploration, justify the choice from the consequences of a single violation, and select one mechanism from subsection three (shield, safety layer, control-barrier function, or reachability) plus either a mean or a CVaR constraint. Defend your choices against the model knowledge each mechanism demands and the strength of guarantee it returns, and name one failure mode of your chosen design.