Actor-Critic Methods (A2C/A3C, PPO, SAC)

"The Actor wanted to leap; the Critic kept whispering a number in its ear. Most days the number was wrong, but it was wrong in a consistent direction, and consistently-wrong-by-a-little turned out to beat right-but-deafeningly-noisy. So they argued their way, step by clipped step, to a policy neither of them could have found alone."

An Actor and a Critic Who Argue Their Way to a Good Policy

In Section 25.2 we derived the policy gradient and trained a policy with REINFORCE, weighting the log-probability of each action by the return that followed it and subtracting a baseline to reduce variance. We noted there that the best possible baseline is the state value function $V^\pi(s)$, and that learning that baseline turns REINFORCE into something more powerful. This section makes good on that promise. The learned baseline becomes a full critic, the policy becomes the actor, and the two are trained together. The recurring temporal-AI thread is visible here too: the value function the critic learns is the same Bellman object that the dynamic-programming and temporal-difference methods of Chapter 24 estimated for fixed policies, now used as a moving target inside a policy-optimization loop. We use the unified notation of Appendix A: $s_t$ the state, $a_t$ the action, $r_t$ the reward, $\pi_\theta$ the actor with parameters $\theta$, $V_\phi$ the critic with parameters $\phi$, and $\gamma \in [0,1)$ the discount.

Why does the field overwhelmingly default to actor-critic methods, and to PPO in particular, rather than to the pure policy gradient or to value-based methods like the DQN of Section 25.1? Because actor-critic occupies the productive middle. Value-based methods learn a value function and read a policy off it, which is sample-efficient but awkward for continuous actions and prone to instability when the max operator interacts with function approximation. Pure policy gradients optimize the policy directly and handle any action space, but their variance makes them slow and finicky. Actor-critic keeps the policy-gradient generality (any action space, stochastic policies, direct objective) while borrowing the value-based machinery to tame variance. The result is a recipe that scales from CartPole to robotic locomotion to large-language-model alignment with reinforcement learning from human feedback, where PPO is the standard optimizer.

The four competencies this section installs are these: to write the actor-critic gradient with a learned advantage and explain why it has lower variance than REINFORCE; to compute the generalized advantage estimate and read its $\lambda$ knob as a bias-variance dial; to state PPO's clipped surrogate objective and explain why clipping enforces a trust region cheaply; and to distinguish on-policy PPO from off-policy maximum-entropy SAC and know which to reach for. These skills carry directly into the continuous control of Section 25.4 and the advanced methods of Chapter 26.

1. Actor-Critic: A Policy and a Critic That Cuts Variance Intermediate

The policy gradient from Section 25.2 has the form $\nabla_\theta J(\theta) = \mathbb{E}\big[\sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \Psi_t\big]$, where $\Psi_t$ is some measure of how good action $a_t$ was. REINFORCE uses the Monte-Carlo return $\Psi_t = G_t = \sum_{k \ge t} \gamma^{k-t} r_k$, an unbiased estimate of the action's value but a famously noisy one, because a single rollout's return reflects every random choice made after step $t$, not just the choice at step $t$. The actor-critic idea is to replace that noisy return with a learned, lower-variance signal: the advantage.

Subtracting any function of the state alone, a baseline $b(s_t)$, leaves the gradient unbiased (its expectation is unchanged because $\mathbb{E}[\nabla_\theta \log \pi_\theta(a \mid s)] = 0$) while reducing variance. The variance-minimizing baseline is the state value $V^\pi(s_t)$, and with that choice $\Psi_t = G_t - V^\pi(s_t)$ becomes the advantage. The critic learns to estimate $V^\pi$, so the actor-critic gradient is

The advantage $A^\pi(s_t, a_t)$ answers a sharper question than the raw return does: not "what happened after I took this action?" but "how much better than my average behavior in this state was this particular action?". An action with positive advantage gets its log-probability pushed up; negative advantage pushes it down; an exactly average action gets no push at all. This re-centering is what removes the variance: the large state-dependent baseline that contaminated REINFORCE has been subtracted away, leaving only the action-specific signal.

We rarely have $Q^\pi$ and $V^\pi$ separately. The standard estimator is the one-step temporal-difference (TD) error, which is an unbiased estimate of the advantage when the critic is exact:

This is the same TD error that drove the value-learning methods of Chapter 24, now doing double duty: it trains the critic (by regressing $V_\phi(s_t)$ toward the bootstrap target $r_t + \gamma V_\phi(s_{t+1})$) and it weights the actor's gradient. Advantage actor-critic, A2C, is exactly this: collect a batch of transitions with the current policy, fit the critic to the TD targets, weight the policy gradient by the TD-error advantage, and step both. A3C (asynchronous advantage actor-critic, Mnih et al., 2016) is the same algorithm run by many worker threads in parallel, each computing gradients on its own rollouts and applying them asynchronously to shared parameters; the parallelism both speeds data collection and decorrelates the samples, which stabilizes training. A2C is the synchronous, batched variant that averages the workers before each update, and on modern GPUs it usually matches or beats A3C while being simpler.

The one-step TD advantage is at the low-variance, high-bias end of a spectrum; the full Monte-Carlo advantage $G_t - V_\phi(s_t)$ is at the high-variance, low-bias end. The generalized advantage estimator (GAE; Schulman et al., 2015) interpolates between them with a single parameter $\lambda \in [0,1]$. Writing the per-step TD error as $\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$, the GAE advantage is an exponentially-weighted sum of future TD errors:

The recursive form on the right is what implementations use: a single backward pass over a trajectory accumulates the advantage in $O(T)$. The parameter $\lambda$ is the bias-variance dial promised above. At $\lambda = 0$, GAE collapses to the one-step TD error $\hat{A}_t = \delta_t$, maximally biased by the critic but minimally noisy. At $\lambda = 1$, it telescopes to the Monte-Carlo advantage $G_t - V_\phi(s_t)$, unbiased by the critic but maximally noisy. Intermediate values, with $\lambda \approx 0.95$ the near-universal default, keep most of the variance reduction while limiting the bias the critic injects. GAE is used inside essentially every modern on-policy method, PPO included, and it is the single most important practical refinement of the basic advantage estimate.

2. PPO: The Clipped Surrogate That Became the Default Intermediate

A2C takes one gradient step per batch of data and throws the data away, because the policy gradient is on-policy: the moment $\theta$ changes, the old rollouts no longer reflect the current policy. This is wasteful, and the temptation is to take several gradient steps on the same batch to extract more learning from each expensive environment interaction. But naive reuse is dangerous: after a few steps the policy has moved far enough that the old data badly misestimates the new gradient, and the update can collapse the policy. Proximal policy optimization (PPO; Schulman et al., 2017) is the elegant fix, and it is the reason actor-critic became the field's default.

PPO works with the probability ratio between the new and old policies, $r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)$, which equals one when the policy has not moved. A plain importance-weighted objective $\mathbb{E}[r_t(\theta)\,\hat{A}_t]$ would let a single update push $r_t$ arbitrarily far whenever the advantage is large, which is exactly the instability we want to prevent. PPO clips the ratio to a small interval $[1-\epsilon, 1+\epsilon]$ (typically $\epsilon = 0.2$) and optimizes the pessimistic minimum of the clipped and unclipped objectives:

Read the clip operationally. When the advantage is positive (a good action), the objective rewards increasing $r_t$, but the clip caps the reward once $r_t$ exceeds $1+\epsilon$, so there is no incentive to move the policy further than the clip allows. When the advantage is negative (a bad action), the objective rewards decreasing $r_t$, but the clip floors it at $1-\epsilon$. The outer $\min$ makes the bound one-sided in the safe direction: it never clips in a way that would let the update overshoot, and it does remove the gradient once the ratio leaves the trust region. The practical effect is a soft trust region enforced by a single elementwise clip, with no Hessian, no conjugate gradient, and no line search. The full PPO loss adds a value-function regression term for the critic and an entropy bonus to maintain exploration: $L = L^{\text{CLIP}} - c_1 (V_\phi - V^{\text{target}})^2 + c_2\, \mathcal{H}[\pi_\theta]$.

PPO is the default workhorse for three reasons. It is robust: the clip makes it forgiving of hyperparameters that would destabilize vanilla policy gradients. It is simple: a few dozen lines on top of A2C, with first-order optimization throughout. And it is sample-reusing: the clip makes it safe to take several epochs of minibatch updates on each batch of rollouts, extracting far more learning per environment step than A2C. This combination is why PPO trains everything from MuJoCo locomotion to the reinforcement-learning-from-human-feedback stage of large language models.

PPO is the cheap descendant of trust region policy optimization (TRPO; Schulman et al., 2015), which made the trust-region idea exact by maximizing the surrogate subject to a hard constraint that the average KL divergence between the new and old policies stay below a threshold, $\mathbb{E}[\mathrm{KL}(\pi_{\theta_{\text{old}}} \,\|\, \pi_\theta)] \le \delta$. TRPO solves this constrained problem with a second-order method (a conjugate-gradient solve against the Fisher information matrix plus a backtracking line search), which gives strong monotonic-improvement guarantees but is complex and costly per step. PPO keeps TRPO's trust-region intuition and discards its machinery: the clip is a crude but effective stand-in for the KL constraint, first-order and embarrassingly simple, and in practice it matches or beats TRPO at a fraction of the implementation and compute cost, which is why almost everyone uses PPO.

3. SAC: Off-Policy, Maximum-Entropy Actor-Critic Advanced

PPO and A2C are on-policy: they learn only from data generated by the current policy, and they discard each batch after a few updates. For continuous control on a physical robot or an expensive simulator, where every environment step is precious, on-policy methods can be painfully sample-hungry. Soft actor-critic (SAC; Haarnoja et al., 2018) attacks exactly this, combining three ideas into the most sample-efficient model-free continuous-control method in wide use: it is off-policy (so it reuses every transition from a replay buffer many times), it is actor-critic (an explicit stochastic policy plus learned $Q$-critics), and it is maximum-entropy (it optimizes for reward plus the policy's own entropy).

The maximum-entropy objective augments the usual discounted return with an entropy bonus at every step, scaled by a temperature $\alpha$:

The entropy term has a precise purpose: it rewards the policy for staying as random as it can while still collecting reward, which keeps exploration alive throughout training and produces robust policies that do not collapse prematurely onto a single brittle behavior. The temperature $\alpha$ sets the exploration-exploitation balance, and modern SAC tunes it automatically by treating it as the dual variable of a constraint on the policy's entropy, so the practitioner need not hand-set it. The critics are trained with a soft Bellman backup whose target includes the entropy of the next action, $y = r + \gamma\big(\min_{i=1,2} Q_{\bar\phi_i}(s', a') - \alpha \log \pi(a' \mid s')\big)$ with $a' \sim \pi$, and the actor is updated to maximize the entropy-regularized $Q$ value.

SAC sits at the end of a clear lineage of off-policy actor-critics for continuous action spaces, which it is worth tracing because each member fixed a flaw in the previous one. DDPG (deep deterministic policy gradient; Lillicrap et al., 2016) was the first deep off-policy actor-critic for continuous control: a deterministic actor, a $Q$-critic, a replay buffer, and target networks, essentially DQN's machinery adapted to continuous actions via a deterministic policy gradient. DDPG was sample-efficient but notoriously brittle, prone to overestimating $Q$ values and to hyperparameter sensitivity. TD3 (twin-delayed DDPG; Fujimoto et al., 2018) fixed the overestimation with three tricks now standard across the family: twin critics with a $\min$ to curb overestimation bias, delayed policy updates (update the actor less often than the critics), and target-policy smoothing (add noise to the target action). SAC inherited the twin-critic $\min$ from TD3 and added the maximum-entropy stochastic policy, which gave it both TD3's stability and substantially better exploration, making it the current default for continuous control, the subject of Section 25.4.

4. Choosing a Method: DQN vs A2C vs PPO vs SAC Intermediate

The four algorithms that anchor this chapter span the practical design space of deep reinforcement learning, and choosing among them comes down to a few axes: whether the method is on-policy or off-policy (which governs sample efficiency), what action spaces it handles, and how stable and easy to tune it is. The DQN of Section 25.1 is value-based and discrete-action-only; A2C, PPO, and SAC are the actor-critics of this section. Table 25.3.1 lays the trade-offs side by side.

The practical decision rule that falls out of the table is short. If your actions are discrete and the simulator is cheap, DQN or PPO both work and PPO is usually the safer first try. If you want a simple, parallel-friendly baseline and have many cheap environments, A2C is fine and instructive. For almost everything else, start with PPO, because it is forgiving and robust and you will spend less time tuning it. Switch to SAC the moment two conditions hold together: the action space is continuous, and each environment step is expensive enough that PPO's sample appetite hurts, as on a real robot or a slow physics simulator. We make the on-policy actor-critic concrete next.

5. Worked Example: A2C From Scratch, Then PPO via a Library Advanced

We now make the actor-critic concrete in two stages. First we implement a minimal advantage actor-critic by hand in PyTorch and train it on CartPole-v1, the standard balancing task from Chapter 24: a shared trunk feeding a policy head (the actor) and a value head (the critic), with the policy gradient weighted by the TD-error advantage. Then we solve the same task in a handful of lines with stable-baselines3 PPO, so the from-scratch version and the production version sit side by side. Code 25.3.1 is the from-scratch A2C.

import gymnasium as gym
import torch, torch.nn as nn
import torch.nn.functional as F

env = gym.make("CartPole-v1")
obs_dim = env.observation_space.shape[0]   # 4 state features
n_act = env.action_space.n                 # 2 discrete actions

class ActorCritic(nn.Module):
    """Shared trunk, then a policy head (actor) and a value head (critic)."""
    def __init__(self, obs_dim, n_act, hidden=128):
        super().__init__()
        self.trunk = nn.Sequential(nn.Linear(obs_dim, hidden), nn.Tanh())
        self.actor = nn.Linear(hidden, n_act)   # logits over actions
        self.critic = nn.Linear(hidden, 1)      # state value V(s)

    def forward(self, s):
        z = self.trunk(s)
        return self.actor(z), self.critic(z).squeeze(-1)

net = ActorCritic(obs_dim, n_act)
opt = torch.optim.Adam(net.parameters(), lr=3e-3)
gamma = 0.99

def run_episode():
    """Collect one episode, return per-step log-probs, values, rewards, entropies."""
    s, _ = env.reset()
    logps, values, rewards, entropies = [], [], [], []
    done = False
    while not done:
        st = torch.tensor(s, dtype=torch.float32)
        logits, v = net(st)
        dist = torch.distributions.Categorical(logits=logits)
        a = dist.sample()                       # sample action from the actor
        s, r, term, trunc, _ = env.step(a.item())
        done = term or trunc
        logps.append(dist.log_prob(a))
        values.append(v)
        rewards.append(r)
        entropies.append(dist.entropy())
    return logps, values, rewards, entropies

for episode in range(600):
    logps, values, rewards, entropies = run_episode()
    # Discounted returns G_t, computed backward over the trajectory.
    returns, G = [], 0.0
    for r in reversed(rewards):
        G = r + gamma * G
        returns.insert(0, G)
    returns = torch.tensor(returns)
    values = torch.stack(values)
    # Advantage = return - critic baseline (detached so it only weights the actor).
    adv = returns - values.detach()
    adv = (adv - adv.mean()) / (adv.std() + 1e-8)   # normalize for stable gradients
    logps = torch.stack(logps); ent = torch.stack(entropies)
    actor_loss = -(logps * adv).mean()              # policy gradient with advantage
    critic_loss = F.mse_loss(values, returns)       # fit V(s) to the returns
    loss = actor_loss + 0.5 * critic_loss - 0.01 * ent.mean()   # entropy bonus
    opt.zero_grad(); loss.backward(); opt.step()
    if episode % 100 == 0:
        print(f"episode {episode:4d}  return {sum(rewards):6.1f}")

episode    0  return   18.0
episode  100  return   94.0
episode  200  return  201.0
episode  300  return  500.0
episode  400  return  500.0
episode  500  return  500.0

The hand-written version above is roughly 50 lines of careful bookkeeping: the network, the rollout loop, the discounted-return computation, the advantage, and the combined loss. A production library collapses all of that, and gives you the more robust PPO clip and GAE for free. Code 25.3.2 solves the identical task with stable-baselines3.

from stable_baselines3 import PPO

# The entire actor-critic: network, rollout buffer, GAE, clipped PPO update.
model = PPO("MlpPolicy", "CartPole-v1", verbose=0,
            n_steps=2048, gae_lambda=0.95, clip_range=0.2)
model.learn(total_timesteps=100_000)

# Evaluate the trained policy over a few episodes.
eval_env = gym.make("CartPole-v1")
for _ in range(3):
    s, _ = eval_env.reset(); done = False; total = 0.0
    while not done:
        a, _ = model.predict(s, deterministic=True)
        s, r, term, trunc, _ = eval_env.step(int(a))
        done = term or trunc; total += r
    print(f"eval return {total:.1f}")

eval return 500.0
eval return 500.0
eval return 500.0

Read the two code blocks together. Code 25.3.1 built advantage actor-critic by hand so every piece (the actor head, the critic baseline, the TD-style advantage, the entropy bonus) is visible and editable. Code 25.3.2 reproduced and improved on the result with two lines, trading that visibility for the robustness of PPO's clip and GAE. The progression mirrors the whole section: understand the actor-critic skeleton from scratch, then reach for the library when you want PPO's stability without reimplementing it.

from stable_baselines3 import A2C, PPO, SAC

a2c = A2C("MlpPolicy", "CartPole-v1").learn(100_000)        # on-policy, simple baseline
ppo = PPO("MlpPolicy", "CartPole-v1").learn(100_000)        # on-policy, clipped, robust default
sac = SAC("MlpPolicy", "Pendulum-v1").learn(50_000)         # off-policy, continuous, sample-efficient

6. GRPO: Group Relative Policy Optimization Advanced

Proximal policy optimization works well for reinforcement learning from human feedback, but it carries a cost that becomes significant at large model scales: a separate value critic network, typically as large as the policy itself, that must be stored, forwarded, and backpropagated through. For a 70-billion-parameter language model, doubling the memory footprint is not a minor inconvenience. Group Relative Policy Optimization (GRPO; Shao et al., DeepSeekMath, 2024) eliminates the critic entirely and replaces it with a statistical baseline computed from a group of responses sampled for the same prompt. It was central to the training of DeepSeek-R1 (2025) and has become the dominant alternative to PPO for LLM post-training on reasoning tasks.

The key idea is both conceptually clean and computationally cheap. For each prompt $x$, sample a group of $G$ completions $y_1, y_2, \ldots, y_G$ from the current policy, and obtain a scalar reward $r_i$ for each (from a rule-based verifier, a reward model, or human raters). The group mean and standard deviation play the role of the critic's baseline:

This is an advantage estimate, but one derived entirely from within the group rather than from a learned value function. The completion that earned the highest reward in the group gets a positive advantage; below-average completions get negative advantages; exactly average completions get zero. No bootstrap target, no temporal-difference calculation, no separate network. The GRPO objective clips these advantages with the same PPO-style ratio and adds a KL penalty against a frozen reference policy $\pi_{\text{ref}}$:

where $r_t(\theta) = \pi_\theta(y_i \mid x) / \pi_{\theta_{\text{old}}}(y_i \mid x)$ is the token-level probability ratio aggregated over the completion. The KL term with coefficient $\beta$ prevents the policy from drifting too far from the reference, playing the same stabilizing role as the clip does in PPO. In practice the KL is estimated via a per-token sample approximation rather than an analytic computation.

The savings relative to PPO are substantial. A PPO run for LLM fine-tuning requires four models in memory: the active policy, the frozen reference, the value critic, and the critic's target network. GRPO requires two: the active policy and the frozen reference. Eliminating the critic removes roughly half the parameters, half the optimizer states, and all of the bootstrap target computation and GAE accumulation. The trade-off is that the group-relative advantage is noisier than a well-trained critic's estimate, which typically requires larger $G$ (4 to 8 in practice) and benefits from verifiable reward signals where the reward is binary or near-binary, as in math and code generation where correctness can be checked automatically.

import torch
import torch.nn.functional as F

def grpo_loss(log_probs_new, log_probs_old, log_probs_ref,
              rewards, eps=0.2, beta=0.01):
    """
    GRPO loss for one batch of (prompt, group-of-completions) pairs.

    Args:
        log_probs_new : (B, G) -- sum of token log-probs under current policy
        log_probs_old : (B, G) -- same under old policy (before this epoch's updates)
        log_probs_ref : (B, G) -- same under frozen reference policy
        rewards       : (B, G) -- scalar reward for each completion
        eps           : clip radius (same as PPO epsilon)
        beta          : KL penalty coefficient

    Returns:
        scalar loss (negate to maximize the objective)
    """
    # --- Group-relative advantages ---
    mean_r = rewards.mean(dim=1, keepdim=True)          # (B, 1)
    std_r  = rewards.std(dim=1, keepdim=True) + 1e-8    # (B, 1)
    advantages = (rewards - mean_r) / std_r             # (B, G)

    # --- Clipped PPO surrogate ---
    ratio = torch.exp(log_probs_new - log_probs_old)    # (B, G)
    clipped = torch.clamp(ratio, 1.0 - eps, 1.0 + eps)
    surrogate = torch.min(ratio * advantages,
                          clipped * advantages).mean()

    # --- Per-sample KL from reference (sample estimate) ---
    kl = (log_probs_new - log_probs_ref).mean()

    return -(surrogate - beta * kl)                     # negate: we minimize

from trl import GRPOConfig, GRPOTrainer

config = GRPOConfig(
    num_generations=4,        # G: completions per prompt
    max_new_tokens=512,
    learning_rate=1e-6,
    kl_coeff=0.01,            # beta
    clip_range=0.2,           # epsilon
)

trainer = GRPOTrainer(
    model=policy_model,
    ref_model=reference_model,
    reward_funcs=[verifier_reward],   # callable: (prompts, completions) -> List[float]
    args=config,
    train_dataset=math_dataset,
)
trainer.train()

Comparing GRPO to PPO along the dimensions that matter for LLM fine-tuning: GRPO uses no critic (simpler, roughly half the memory), operates on-policy like PPO (requires fresh samples each step), works on discrete token sequences where a group average is a natural baseline, and targets tasks with verifiable or dense reward signals where group sampling produces informative advantages. PPO is preferable when the reward signal is sparse or binary for most completions and a learned critic can smooth the landscape, or when you are doing continuous-control rather than language generation. For math reasoning and code generation, where a verifier can score every completion cleanly, GRPO's simplicity and memory savings give it a practical edge, as the DeepSeek-R1 results demonstrated at scale. The discussion continues in Section 25.4, which covers continuous-control methods where the critic remains essential.

7. DPO: Direct Preference Optimization Advanced

Reinforcement learning from human feedback (RLHF) as practiced with PPO requires a pipeline with at least four stages: collect preference data, train a reward model, use that reward model as the signal for a PPO fine-tuning run, and maintain the frozen reference policy throughout. Each stage has its own hyperparameters, its own failure modes, and its own memory footprint. Direct Preference Optimization (DPO; Rafailov et al., 2023) collapses the reward model and the RL training loop into a single binary classification loss over preference pairs, eliminating the explicit reward model and the RL optimizer entirely.

The setup is a dataset of triples $(x, y_w, y_l)$: a prompt $x$, a preferred (chosen) completion $y_w$, and a dispreferred (rejected) completion $y_l$, for example from human annotators who compared two model outputs and picked the better one. DPO derives its loss from the observation that the optimal policy under a KL-constrained RLHF objective has a closed form:

where $r^*$ is the optimal reward and $Z(x)$ is the partition function. Inverting this relationship expresses the optimal reward in terms of the optimal policy and the reference: $r^*(x, y) = \beta \log(\pi^*(y \mid x) / \pi_{\text{ref}}(y \mid x)) + \beta \log Z(x)$. Substituting this implicit reward into the Bradley-Terry preference model (which gives the probability that $y_w$ is preferred to $y_l$) and noting that $Z(x)$ cancels, one arrives at the DPO loss:

where $\sigma$ is the sigmoid function and $\beta > 0$ controls how far the trained policy may stray from the reference. The loss is a standard binary cross-entropy: the argument to $\sigma$ is the difference between the log-likelihood ratio for the preferred completion and the log-likelihood ratio for the rejected completion, both measured relative to the reference policy. Minimizing the loss pushes the policy to assign higher relative probability to $y_w$ than to $y_l$, calibrated against what the reference already does.

The practical advantages of DPO over PPO-based RLHF are significant. PPO-based RLHF requires four models in memory: the active policy, the frozen reference, the reward model, and the value critic. DPO requires two: the active policy and the frozen reference. There is no RL training loop, no rollout collection, no reward model inference at every step; the entire training procedure is a supervised fine-tuning pass over preference pairs, which means it fits into standard fine-tuning infrastructure without modification. The cost of this simplicity is that DPO has no mechanism for online exploration: it trains only on the fixed preference dataset, and if the reference policy's distribution does not cover the cases where preferences are most informative, the trained policy may not improve on those cases. PPO with an active reward model can in principle discover better completions that lie outside the reference distribution; DPO cannot unless those completions appear in the offline preference data.

import torch
import torch.nn.functional as F

def dpo_loss(log_probs_chosen_policy,
             log_probs_rejected_policy,
             log_probs_chosen_ref,
             log_probs_rejected_ref,
             beta=0.1):
    """
    DPO loss for a batch of preference pairs.

    Args:
        log_probs_chosen_policy   : (B,) sum of token log-probs for y_w under pi_theta
        log_probs_rejected_policy : (B,) sum of token log-probs for y_l under pi_theta
        log_probs_chosen_ref      : (B,) same under frozen reference policy
        log_probs_rejected_ref    : (B,) same under frozen reference policy
        beta                      : regularization strength (distance from reference)

    Returns:
        scalar loss
    """
    # Log-ratio for chosen completion: how much more (or less) likely than reference
    log_ratio_chosen   = log_probs_chosen_policy   - log_probs_chosen_ref    # (B,)
    # Log-ratio for rejected completion
    log_ratio_rejected = log_probs_rejected_policy - log_probs_rejected_ref  # (B,)

    # DPO objective: sigmoid binary cross-entropy on the scaled difference
    logits = beta * (log_ratio_chosen - log_ratio_rejected)
    loss = -F.logsigmoid(logits).mean()
    return loss

from trl import DPOConfig, DPOTrainer

config = DPOConfig(
    beta=0.1,
    learning_rate=5e-7,
    per_device_train_batch_size=4,
    max_length=1024,
)

trainer = DPOTrainer(
    model=policy_model,
    ref_model=reference_model,   # frozen; pass None to use implicit reference via PEFT
    args=config,
    train_dataset=preference_dataset,  # must have "prompt", "chosen", "rejected" columns
)
trainer.train()

DPO and GRPO occupy complementary niches in the post-PPO landscape. DPO is the choice when you have an offline dataset of human preference pairs and want alignment without RL machinery. GRPO is the choice when you have a verifiable reward (a math checker, a code executor) and can afford to sample multiple completions per prompt at training time. PPO remains relevant when the reward signal is a trained model (as opposed to a verifier), the task requires online exploration, or you are doing continuous control where a learned critic is genuinely informative. Together the three methods span most of the practical space of policy optimization for language models, and the right choice depends on the shape of the reward signal available, the memory budget, and whether online sampling is feasible. These trade-offs connect directly to the advanced reward-learning methods of Chapter 26.

Exercises

1. Conceptual. Explain why subtracting a state-dependent baseline $b(s_t)$ from the return in the policy gradient leaves the gradient's expectation unchanged but reduces its variance. Then argue why the state value $V^\pi(s_t)$ is a particularly good choice of baseline, and state what the advantage $G_t - V^\pi(s_t)$ measures in one sentence.
2. Implementation. Modify the from-scratch A2C of Code 25.3.1 to use the generalized advantage estimator instead of the plain return-minus-baseline advantage: compute per-step TD errors $\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$ and accumulate $\hat{A}_t = \delta_t + \gamma\lambda\,\hat{A}_{t+1}$ backward over the trajectory with $\lambda = 0.95$. Train on CartPole and compare the learning curve to the original; report whether GAE reaches return 500 in fewer episodes.
3. Open-ended. Take the PPO clip of subsection two and investigate the role of $\epsilon$. Train stable-baselines3 PPO on CartPole with $\epsilon \in \{0.05, 0.2, 0.5\}$ (the clip_range argument) and a fixed seed, and characterize how the clip width affects learning speed and stability. Relate your finding to the trust-region interpretation: what failure mode would you expect as $\epsilon \to \infty$, and why?