Preference-Based RL and RLHF

"Nobody could tell me what good behavior was, and nobody could show me either. So I stopped asking. I just put two of my own attempts side by side and watched which one the human reached for. A thousand quiet choices later, I knew their taste better than they could have spelled it out. I am the score they never wrote down."

A Reward Model Learning What Humans Like by Watching Them Choose

In Section 27.4 we closed the imitation-learning arc with inverse reinforcement learning: recover a reward function from expert demonstrations, then plan against it. That program assumes the expert can act well enough to demonstrate. This section relaxes that assumption to its breaking point. Suppose no one can reliably demonstrate the behavior you want, and no one can write its reward function either, but anyone can look at two attempts and point to the better one. This is the common case for the most valuable and least formalizable objectives, and it is precisely the regime where preference-based RL lives. The shift in supervision is the whole story: from "do this" (demonstrations) to "this is better than that" (comparisons), a strictly weaker and strictly cheaper signal that turns out to be enough.

The reason comparisons are enough connects directly to a thread we opened in Part V. The human-in-the-loop adaptive systems of Chapter 21 already treated human feedback as a first-class signal a system queries and adapts to over time; there the feedback corrected a forecast or a decision online. Here we make that feedback the source of the reward itself, learning the objective rather than just the policy. And the payoff is enormous in scope: the same machinery that aligns a robot's gait to "looks natural" aligns a large language model to "is a good assistant". We use the unified notation of Appendix A throughout: $\pi_\theta$ the policy, $r_\phi$ the learned reward model, $\pi_{\text{ref}}$ the reference policy, and $(y_w, y_l)$ a preferred-versus-rejected pair.

The four competencies this section installs are: to recognize when comparison is the right supervision and frame preference learning as reward modeling; to write and fit the Bradley-Terry preference likelihood that turns pairwise choices into a scalar reward; to state the RLHF objective, a learned reward minus a KL leash to a reference, and explain why every term is there; and to read the 2024 to 2026 shift to DPO and RLAIF, knowing what each stage buys and what dropping it costs. These are the load-bearing skills for the sequence-as-decision view of Chapter 28, where the policy we align here becomes a sequence model conditioned on outcomes.

1. When Comparison Beats Demonstration Beginner

Every method so far in this chapter has needed an expert who could produce good behavior. Behavioral cloning needed state-action pairs from a competent demonstrator; DAgger needed that demonstrator on call to label the states the learner visited; inverse RL needed demonstrations clean enough that "the expert was near-optimal" was a defensible assumption. The quiet precondition under all of them is that someone, somewhere, can act well. Relax that and the methods have nothing to learn from.

Yet many of the behaviors we most want are exactly the ones nobody can demonstrate to optimality. Consider summarizing a document "faithfully and concisely", or writing a "helpful" answer, or controlling a prosthetic limb so the motion "feels natural" to its wearer. A person cannot reliably produce the best summary on demand, and certainly cannot assign it a calibrated numerical score, "this summary is a 7.3 out of 10" is a meaningless request. But ask that same person to look at two summaries and say which is better, and they answer instantly and consistently. Comparison is psychologically cheap and reliable precisely where absolute scoring and demonstration are expensive and noisy. Humans are excellent relative judges and poor absolute ones.

Preference-based RL is built on that asymmetry. We never ask a human to act and never ask for a number; we only ask them to choose. Formally, we collect a dataset of comparisons $\mathcal{D} = \{(\sigma_w^{(i)}, \sigma_l^{(i)})\}$, where each pair is two behavior segments (two trajectories, two summaries, two model responses) and $\sigma_w$ is the one the human preferred (won) over $\sigma_l$ (lost). From this dataset of choices, with no rewards and no demonstrations anywhere in it, we will recover a scalar reward function that explains the choices, and then optimize a policy against it. The bet is that a single consistent reward landscape underlies the human's many local comparisons, and that recovering that landscape recovers the objective.

One historical anchor is worth carrying forward. The modern recipe was crystallized by Christiano and colleagues in 2017, who trained simulated robots and Atari agents from human preferences over short video clips, with the human never seeing the reward and never demonstrating, and reached or exceeded hand-engineered-reward performance using feedback on under one percent of the agent's interactions. That result, learn a reward model from clip comparisons, optimize a policy against it, refresh the comparisons as the policy improves, is the template the rest of this section formalizes and that the language-model alignment of 2022 onward scaled up.

2. The Bradley-Terry Model and Training a Reward Model Intermediate

To turn a pile of "$A$ beat $B$" judgments into a scalar reward, we need a model that maps each behavior to a number and predicts comparisons from those numbers. The Bradley-Terry model, a century-old model of paired comparisons (think chess Elo, which is the same idea), is exactly that link. Assign each item $\sigma$ a latent score $r_\phi(\sigma)$. The model says the probability that $\sigma_w$ is preferred to $\sigma_l$ is a logistic function of the score difference:

where $\sigma(z) = 1/(1 + e^{-z})$ is the logistic sigmoid. The structure is the entire point: preference probability depends only on the difference of scores, not their absolute level. A reward that is uniformly shifted by a constant predicts the identical preferences, so the reward model is identified only up to an additive constant, a fact that returns in subsection three as the reason a baseline or KL anchor is needed. When the two scores are equal the probability is one half (a coin flip, the human is indifferent); as the winner's score pulls ahead the probability saturates toward one.

We fit $r_\phi$ by maximum likelihood: choose parameters that make the observed human choices as probable as possible. The negative log-likelihood over the comparison dataset is the reward-model training loss,

which is precisely a binary-cross-entropy loss on the event "the winner outscores the loser", with the score difference as the logit. Minimizing it pushes the score of each preferred item above the score of its rejected partner by a margin that grows with how confidently the model should predict the human's choice. In the language-model setting $\sigma$ is a prompt-response pair and $r_\phi$ is typically the base language model with its output head replaced by a single scalar value head, so the reward model is "the policy network, scored". Figure 27.5.1 lays out the data flow from comparisons to a trained reward.

Two practical properties of this loss deserve emphasis before we use it. First, because only score differences are supervised, the reward model's absolute scale and offset are arbitrary; you cannot read $r_\phi(\sigma) = 4.2$ as "good" in any absolute sense, only $r_\phi(\sigma_w) > r_\phi(\sigma_l)$ is meaningful. Second, the reward model is a learned approximation of human taste and is therefore exploitable: a policy optimized hard enough against it will find inputs where the model's score is high but a real human would disagree, the phenomenon of reward hacking that motivates the KL leash of subsection three. The reward model is a proxy, accurate near the data it was trained on and untrustworthy far from it.

3. RLHF: Optimizing a Policy Against the Learned Reward Intermediate

With a reward model $r_\phi$ in hand we have an objective; now we optimize a policy against it. The naive move, "maximize expected learned reward", $\max_\theta \mathbb{E}_{y\sim\pi_\theta}[r_\phi(y)]$, fails for the reason subsection two warned about: the reward model is a proxy, and a policy free to maximize it will gallop off to regions where the proxy is high but wrong, producing degenerate outputs that score well and read badly. The reward model was never trained on those regions and cannot be trusted there. The standard fix is to add a leash. The RLHF objective optimizes reward minus a penalty for straying too far from a trusted reference policy $\pi_{\text{ref}}$ (usually the supervised-fine-tuned model the run started from):

Read each term. The first, $r_\phi(x, y)$, is the learned reward: pull the policy toward outputs the reward model scores highly. The second, $\beta\,\mathrm{KL}(\pi_\theta \| \pi_{\text{ref}})$, is the leash: penalize the policy for putting probability mass where the reference would not, with coefficient $\beta$ setting the leash length. A large $\beta$ keeps the policy close to the trusted reference (safe, but little improvement); a small $\beta$ lets it chase reward harder (more improvement, more risk of reward hacking and of forgetting the fluency the reference had). The KL term also resolves the additive-constant ambiguity of subsection two: it anchors the policy to a concrete reference distribution, so the arbitrary offset of $r_\phi$ no longer matters.

In practice this objective is optimized with proximal policy optimization (PPO), the policy-gradient algorithm of Chapter 25, treating each generated token as an action and the per-prompt reward (the reward-model score, with the KL penalty folded in as a per-token shaping term) as the return. This is where the temporal framing becomes literal: generating a response is a sequential decision process over tokens. The state is the prompt plus the tokens emitted so far, each action is the next token, and the episode ends at the end-of-sequence token, at which point the reward model scores the whole completion. RLHF is reinforcement learning over a token-level MDP, and the policy being aligned is a language model. The connection to the large language models of Chapter 15 is not an analogy; it is the same object. The assistants you interact with daily were tuned by exactly this loop: pretrain, supervised fine-tune to get $\pi_{\text{ref}}$, train a reward model from human comparisons, then PPO against it with a KL leash. Figure 27.5.2 names the three stages.

The KL leash earns its place by preventing two distinct failure modes at once. It blocks reward hacking, where the policy discovers adversarial outputs the reward model overrates (repetitive boilerplate that happens to score high, say), because those outputs are far from the reference and the KL term taxes the distance. And it blocks catastrophic forgetting of the fluency and knowledge baked into the reference during pretraining and supervised fine-tuning, because staying near $\pi_{\text{ref}}$ in distribution means retaining its competence. Tuning $\beta$ is the central practical lever of an RLHF run: too tight and the policy barely improves, too loose and it games the reward and degrades. Much of the engineering folklore of RLHF is really folklore about keeping that one balance.

4. DPO and the 2024 to 2026 Shift to Direct Preference Optimization Advanced

The classic RLHF pipeline is powerful but heavy: three stages, a separate reward model to train and serve, and an on-policy RL loop (PPO) that is notoriously finicky to stabilize, needing careful reward normalization, value-function fitting, and $\beta$ tuning. A natural question is whether all that machinery is necessary, or whether the same preference signal can be optimized more directly. The answer, from Rafailov and colleagues in 2023 and the wave of work that followed, is direct preference optimization (DPO), which collapses reward modeling and policy optimization into a single supervised loss on the preference pairs, with no explicit reward model and no RL loop.

The derivation rests on a clean observation: the KL-constrained objective of subsection three has a closed-form optimal policy, $\pi^*(y\mid x) \propto \pi_{\text{ref}}(y\mid x)\exp(r_\phi(x, y)/\beta)$. Invert that relation to write the reward in terms of the optimal policy and the reference, $r_\phi(x, y) = \beta\log\frac{\pi^*(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta\log Z(x)$, and substitute into the Bradley-Terry likelihood of subsection two. The partition function $Z(x)$ is shared between winner and loser and cancels in the difference, leaving a loss that depends only on the policy, the reference, and the preference pairs:

This is the same Bradley-Terry binary-cross-entropy as the reward-model loss, but with the implicit reward $\hat{r}_\theta(x, y) = \beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ substituted in. The policy is the reward model now. Minimizing $\mathcal{L}_{\text{DPO}}$ raises the policy's log-probability of preferred responses and lowers it for rejected ones, each measured relative to the reference, which is precisely what the three-stage pipeline accomplished, but in one offline supervised pass with no reward model to train, no samples to draw on-policy, and no value function to fit. The KL leash of subsection three has not vanished; it survives, exactly, inside the $\log(\pi_\theta/\pi_{\text{ref}})$ ratio and the coefficient $\beta$, which is why DPO needs the reference model at training time.

The DPO paper opened a fast-moving design space, and the 2024 to 2026 literature is full of variants worth naming. IPO adds a margin term to curb the overfitting DPO can show when preferences are near-deterministic; KTO drops the pairwise requirement entirely and learns from individual thumbs-up or thumbs-down labels using a prospect-theory-inspired loss, which matters because unpaired feedback is even cheaper to collect than pairs; ORPO folds preference optimization into the supervised fine-tuning stage so a single run produces an aligned model without a separate reference pass. Running through all of them is the same retreat from the heavy RL loop toward direct, supervised-style objectives on preference data.

A second axis of the same period removes the human from the loop. Reinforcement learning from AI feedback (RLAIF), and the Constitutional AI approach that popularized it, replaces the human comparison labeler with a capable language model prompted to judge which of two responses better follows a written set of principles (a "constitution"). The AI judge produces the preference labels, which then feed exactly the reward-model-plus-RL or DPO machinery of this section. RLAIF trades some judgment quality for enormous gains in scale and cost, because an AI labeler is effectively unlimited, and it has become a standard way to bootstrap and scale alignment when human comparisons are the bottleneck. The frontier of 2025 to 2026 increasingly blends the two: a thin layer of trusted human comparisons to anchor quality, amplified by a much larger volume of AI-generated preferences.

5. Worked Example: A Bradley-Terry Reward Model From Scratch, Then trl Advanced

We now make the pipeline executable on a tiny synthetic problem where we know the ground-truth taste and can check that preference learning recovers it. The setup: each "behavior" is a 1-D feature $\sigma \in \mathbb{R}$, and the hidden human reward is $r^*(\sigma) = -(\sigma - 1.0)^2$, a taste that prefers behaviors near $\sigma = 1.0$ and dislikes those far from it. We never reveal $r^*$ to the learner. Instead we generate pairwise preferences: sample two behaviors, and let the human prefer the one with higher $r^*$, with Bradley-Terry label noise so the comparisons are realistic. Code 27.5.1 generates the preference data and trains a small Bradley-Terry reward model on it from scratch, then uses that learned reward to take one tiny preference-driven policy step, a from-scratch microcosm of the whole RLHF idea.

import torch, torch.nn as nn

torch.manual_seed(0)
def true_reward(s):                       # the HIDDEN human taste; learner never sees it
    return -(s - 1.0) ** 2                 # prefers behaviors near s = 1.0

# --- 1. Generate pairwise preferences from the hidden taste (Bradley-Terry labels) ---
N = 4000
a = torch.rand(N, 1) * 6 - 3                # candidate behavior A in [-3, 3]
b = torch.rand(N, 1) * 6 - 3                # candidate behavior B in [-3, 3]
p_a_wins = torch.sigmoid(true_reward(a) - true_reward(b))   # Bradley-Terry probability
a_wins = (torch.rand(N, 1) < p_a_wins).float()              # noisy human choice
win = torch.where(a_wins.bool(), a, b)     # the preferred (won) behavior
los = torch.where(a_wins.bool(), b, a)     # the rejected (lost) behavior

# --- 2. Train a reward model r_phi by the Bradley-Terry / BCE loss of subsection 2 ---
reward_model = nn.Sequential(nn.Linear(1, 32), nn.Tanh(), nn.Linear(32, 1))
opt = torch.optim.Adam(reward_model.parameters(), lr=1e-2)
for step in range(800):
    rw, rl = reward_model(win), reward_model(los)            # scores of winner, loser
    loss = -torch.log(torch.sigmoid(rw - rl) + 1e-8).mean()  # -log sigma(r_w - r_l)
    opt.zero_grad(); loss.backward(); opt.step()
print("final reward-model loss = %.4f" % loss.item())

# --- 3. Use the LEARNED reward to take one preference-driven policy step ---
# A "policy" here is a single mean parameter mu we generate behaviors around.
mu = torch.zeros(1, requires_grad=True)    # start far from the (unknown) optimum 1.0
pol_opt = torch.optim.SGD([mu], lr=0.2)
ref_mu = torch.tensor(0.0)                  # reference policy (the KL anchor)
for step in range(60):
    samples = mu + 0.3 * torch.randn(256, 1)               # behaviors from the policy
    learned_r = reward_model(samples).mean()                # maximize LEARNED reward...
    kl = 0.5 * (mu - ref_mu) ** 2                           # ...minus a KL leash to ref
    obj = -(learned_r - 0.1 * kl)                           # negate: SGD minimizes
    pol_opt.zero_grad(); obj.backward(); pol_opt.step()
print("policy mean moved to mu = %.3f (true optimum is 1.0)" % mu.item())

final reward-model loss = 0.5126
policy mean moved to mu = 0.873 (true optimum is 1.0)

Code 27.5.1 ran about 35 lines to express data generation, reward-model training, and a policy step. For a real language-model alignment run, the production library trl (HuggingFace Transformer Reinforcement Learning) collapses each of those stages to a configured trainer. Code 27.5.2 shows the reward-model and DPO calls; the same preference dataset that took dozens of lines of manual loss and optimization above becomes a trainer object and a single .train().

from datasets import Dataset
from transformers import AutoModelForSequenceClassification, AutoModelForCausalLM, AutoTokenizer
from trl import RewardTrainer, RewardConfig, DPOTrainer, DPOConfig

# Preference data: each row has a prompt, a chosen response, and a rejected response.
prefs = Dataset.from_dict({
    "prompt":   ["Summarize the report.", "Explain gradient descent."],
    "chosen":   ["A concise, faithful summary.", "A clear, correct explanation."],
    "rejected": ["A rambling, off-topic reply.", "A confidently wrong explanation."],
})

# --- Option A: train an explicit Bradley-Terry reward model (subsection 2) ---
rm = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=1)
tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
RewardTrainer(model=rm, args=RewardConfig(output_dir="rm"),
              train_dataset=prefs, processing_class=tok).train()   # BT loss, internally

# --- Option B: skip the reward model and RL loop entirely with DPO (subsection 4) ---
policy = AutoModelForCausalLM.from_pretrained("gpt2")
DPOTrainer(model=policy, args=DPOConfig(output_dir="dpo", beta=0.1),
           train_dataset=prefs, processing_class=tok).train()      # DPO loss, internally

# RewardTrainer:  Bradley-Terry loss minimized over the (chosen, rejected) pairs
# DPOTrainer:     implicit-reward DPO loss; no separate reward model, no PPO loop
# (both stream loss/accuracy logs per step; the chosen-vs-rejected margin grows)

Read the two code blocks together. Code 27.5.1 exposed the gears: a Bradley-Terry loss is binary cross-entropy on score differences, and "RLHF" is maximizing a learned reward under a KL leash, both of which fit in a screenful of PyTorch. Code 27.5.2 showed that production tooling hides those gears behind a trainer, and, more importantly, that the modern default (DPOTrainer) discards the reward model and the RL loop altogether, exactly the simplification subsection four derived. The pedagogical payoff is that preference alignment is not a black box: you can build its reward model and policy step by hand, a library wraps the same math, and DPO is a principled shortcut through it, not a different idea.