Offline RL

"They handed me a sealed logbook of someone else's decisions and told me to do better than they did, but I am forbidden from trying anything. Every time I imagine a brilliant move the logbook never saw, my own value estimate promises it is glorious, and I have no way to find out it is a cliff. I have learned to distrust my best ideas precisely because they are mine."

An Agent Learning Entirely From a Dataset It Cannot Add To

In Section 26.1 we learned a world model to plan inside and cut the real-interaction budget, while the agent still interacted freely with its environment. This section asks the opposite question: what if interaction is off the table? You have a database of past decisions and their outcomes, a hospital's record of treatments and patient trajectories, a bank's log of trades and market responses, a factory's archive of controller settings and yields, and you must learn a better policy from that archive alone, because trying a new treatment, a new trade, or a new controller setting on the live system to "explore" is unsafe, expensive, or illegal. This is offline reinforcement learning, also called batch RL, and it is the setting that finally lets the sequential-decision machinery of Part VI touch the high-stakes temporal domains where the three running datasets of this book actually live. We use the unified notation of Appendix A throughout: $s$ for state, $a$ for action, $r$ for reward, $\pi$ for a policy, $Q$ for an action-value function, and $\mathcal{D}$ for the fixed dataset.

The distinction that organizes the whole section is the one between off-policy and offline. An off-policy algorithm can learn about one policy from data generated by another, and in Chapter 25 we relied on exactly this when a replay buffer fed a Q-learner transitions from older versions of itself. Off-policy learning still assumes a stream: the agent keeps acting, keeps refreshing the buffer, keeps correcting the distribution mismatch by sampling fresh consequences of its own current behavior. Offline learning is off-policy learning with that corrective stream amputated. The buffer is frozen, no fresh consequence ever arrives, and any error the agent makes about an action the data does not cover can never be discovered and never be fixed. That amputation is the entire difficulty, and the rest of this section is the study of how to learn well in spite of it.

1. The Offline (Batch) RL Setting: Learning From a Fixed Log Beginner

Formally, offline RL fixes a dataset of transitions $\mathcal{D} = \{(s_i, a_i, r_i, s'_i)\}_{i=1}^{N}$ collected by some behavior policy $\pi_\beta$ that we may not even know. The goal is unchanged from the rest of Part VI: find a policy $\pi$ that maximizes expected return $J(\pi) = \mathbb{E}_{\pi}\big[\sum_{t} \gamma^{t} r_t\big]$ in the underlying Markov decision process of Chapter 22. The one and only constraint, the one that makes this its own subfield, is that the learner may never collect a new transition. There is no environment to call, no rollout to run, no $s'$ to observe for a state-action pair $\mathcal{D}$ does not already contain. Learning happens entirely inside the four walls of $\mathcal{D}$.

Why does this matter so enormously for temporal domains? Because in the settings where sequential decision making is most valuable, online exploration ranges from costly to catastrophic. Consider the three running datasets of this book in turn. In the healthcare series, a policy that decides ventilator settings or insulin dosing from a patient's vital-sign trajectory cannot "explore" an untried dose to see what happens; the exploration is a human being. In the finance series, a trading or hedging policy that explores a novel position on the live book pays for every exploratory mistake in real money, and an exploration that moves the market invalidates its own data. In the sensor and industrial series, a controller that explores an untested setpoint on a chemical reactor or a power grid risks damage, downtime, or a safety incident. In every case there exists, however, an abundance of logged history: years of treatments and outcomes, decades of trades and returns, archives of controller settings and process yields. Offline RL is the technique that turns that logged history into a policy, which is why it, far more than the online RL of earlier chapters, is the form of RL that high-stakes temporal applications can actually deploy.

The cleanest way to see why offline RL needs new ideas is to ask what the data does and does not tell you. The dataset $\mathcal{D}$ densely informs you about the region of state-action space the behavior policy $\pi_\beta$ visited; about everywhere else it is silent. A good offline learner must respect that silence. The naive hope, that we can simply run any off-policy algorithm on $\mathcal{D}$ and read off an optimal policy, founders precisely because off-policy algorithms are built to ask "what is the value of the best action here?", and the best action, by their own optimistic reckoning, is frequently one the data never covered. The next subsection makes that failure precise.

2. The Core Problem: Distributional Shift and Extrapolation Error Intermediate

The central pathology of offline RL has a precise name: distributional shift, and its mechanism inside a value-based learner is extrapolation error. Recall the Bellman optimality backup at the heart of Q-learning, applied to a single logged transition $(s, a, r, s')$:

Read the right-hand side carefully. The target requires evaluating $Q(s', a')$ at the action $a'$ that maximizes the current Q-estimate at the next state $s'$. That maximizing action is chosen by the learner, not drawn from the data, so there is no reason on earth it should be an action the behavior policy ever took at $s'$. When $a'$ is out of distribution, $\mathcal{D}$ contains no transition $(s', a', \cdot, \cdot)$ to ground the estimate $Q(s', a')$, the network is extrapolating into a region it was never trained on, and a function approximator extrapolating off its training support returns essentially arbitrary values. Crucially these errors are not symmetric noise: the $\max$ operator systematically selects the actions with the highest estimates, so it preferentially picks the actions whose values were erroneously inflated. The backup then propagates that inflated value into $Q(s, a)$, the next backup propagates it further, and the overestimation compounds through bootstrapping until the Q-function diverges to a confident hallucination that bears no relation to any achievable return.

It is worth making the distribution mismatch itself explicit, because "distributional shift" names a gap between two distributions. The learner trains its Q-function on state-action pairs drawn from the behavior distribution $d^{\pi_\beta}(s, a)$, the pairs that actually appear in $\mathcal{D}$. But it then uses the Q-function to evaluate pairs drawn from the learned policy's distribution $d^{\pi}(s, a)$, which, if $\pi$ is to improve on $\pi_\beta$, must by definition differ from it. The error of any function approximator is controlled only on the distribution it was trained on. We can write the mismatch that governs the damage as the divergence between the action the policy wants and the actions the data covered:

When $a'_{\pi}$ lies outside the support of the behavior policy at $s'$, the term $Q(s', a'_{\pi})$ is an extrapolation with no data to anchor it, and the inequality has no upper bound. This is the formal statement of why off-policy methods that work online fail offline: online, the agent would eventually take $a'_{\pi}$, observe the true consequence, and correct the inflated estimate; offline, it never can, so the inflation is permanent and self-amplifying.

The remedy that the whole field converges on follows directly from the diagnosis. If the disease is that the learner trusts value estimates for actions the data does not cover, the cure is to make the learner distrust exactly those estimates, to keep the learned policy, or the value function, or both, close to the behavior data where the estimates are trustworthy. This is the conservatism principle: stay near the support of the data. The methods of the next subsection are different concrete realizations of that one principle, and their differences are differences in where they apply the constraint and how they measure closeness.

3. The Methods: Conservatism as Constraint, Penalty, or Implicit Avoidance Advanced

The offline RL toolbox is a family of ways to enforce one principle, stay near the data, and the families differ in where the conservatism is injected. We survey four.

Policy-constraint methods (BCQ, BEAR). The most direct reading of "stay near the data" constrains the learned policy to choose only actions the behavior policy might have chosen. Batch-Constrained deep Q-learning (BCQ, Fujimoto et al., 2019) trains a generative model of $\pi_\beta(a \mid s)$ and restricts the policy's action choices, including the $\max$ in the Bellman backup, to actions that model deems plausible under the behavior, so the target $\max_{a'} Q(s', a')$ is taken only over in-distribution $a'$ and the extrapolation of subsection two never enters the backup. BEAR (Kumar et al., 2019) relaxes the hard support constraint to a distributional one, keeping the learned policy within a bounded maximum mean discrepancy (MMD) of the behavior policy rather than strictly inside its support, which is more robust when the behavior data is itself a mixture of policies. Both can be summarized as a constrained optimization:

where $D$ is a divergence (support overlap for BCQ, MMD for BEAR) and $\epsilon$ caps how far the policy may stray. The strength is interpretability; the weakness is that you must estimate $\pi_\beta$, and a poor behavior model is itself a source of error.

Conservative-value methods (CQL). Conservative Q-Learning (CQL, Kumar et al., 2020) moves the conservatism from the policy to the value function. Rather than constraining which actions the policy may take, CQL adds a penalty to the standard Bellman error that pushes down the Q-values of out-of-distribution actions while pulling up the Q-values of actions actually seen in the data. The CQL objective augments the temporal-difference loss with a regularizer:

where $\mu$ is a wide action distribution (often the current policy or a uniform sampler) used to surface candidate OOD actions to penalize, and $\alpha$ tunes the conservatism. The effect is a learned $Q$ that is a provable lower bound on the true value of the policy, so the agent can never be seduced by an overestimated fantasy: the penalty guarantees that the never-seen $a_3$ of subsection two's example is pushed down rather than chased. CQL is the workhorse of offline RL and the method we implement from scratch in subsection five.

Implicit methods (IQL). Implicit Q-Learning (IQL, Kostrikov et al., 2022) takes the most elegant route to never querying an OOD action: it never evaluates $Q$ at any action outside the dataset at all. IQL replaces the troublesome $\max_{a'} Q(s', a')$, which is what summons OOD actions, with an expectile regression on a separate value network $V(s')$ that estimates an upper expectile of the in-data action values, learning the value of the best in-support action without ever proposing an out-of-support one. The Q-update then bootstraps from $V(s')$ rather than from a max over actions:

where $L_2^{\tau}$ is the asymmetric expectile loss with parameter $\tau \in (0.5, 1)$ controlling how optimistically $V$ approximates the in-support max. Because every term is evaluated only at $(s, a)$ pairs that appear in $\mathcal{D}$, IQL sidesteps extrapolation by construction, and the policy is extracted afterward by advantage-weighted regression. IQL is simple, stable, and a strong default; we use it as the library-trained agent in subsection five.

One-step methods. The simplest family of all observes that much of the divergence in offline RL comes from iterating the Bellman backup many times, each iteration compounding the extrapolation error. One-step methods (for example Brandfonbrener et al., 2021) fit $Q^{\pi_\beta}$, the value of the behavior policy, with a single round of on-data evaluation (no max, no iterated bootstrapping past the behavior), then do one step of policy improvement against that value. Surprisingly, on many benchmark tasks a single careful step of improvement over the behavior policy is competitive with the more elaborate iterative methods, precisely because it never gives the extrapolation error a chance to compound. The lesson is sobering: a large part of the offline-RL difficulty is self-inflicted by aggressive iteration, and restraint is itself a method.

4. The Evaluation Challenge: You Cannot Run the Policy Advanced

Offline RL has a second difficulty that is, if anything, more insidious than training, because it is easy to forget it exists. In online RL you evaluate a policy by running it: deploy it, average its returns, done. Offline you cannot. The whole premise is that you may not interact with the environment, so you cannot run a candidate policy to measure how good it is, and the very same distributional-shift problem that poisons training also poisons evaluation: a learned policy that visits states the dataset does not cover cannot be scored from the dataset, because the dataset says nothing about those states. You are asked to choose and trust a policy you are forbidden to test.

This is the province of off-policy evaluation (OPE), which we introduced in the bandit and RL foundations of Section 24.3 and which becomes load-bearing here. OPE estimates the return $J(\pi)$ of a target policy $\pi$ using only data logged under a different behavior policy $\pi_\beta$. The classical estimators are importance sampling (reweight logged returns by the policy ratio $\prod_t \pi(a_t \mid s_t) / \pi_\beta(a_t \mid s_t)$, unbiased but high-variance, and the variance explodes over long horizons), the direct method (fit a model of returns and evaluate the target policy in it, low-variance but biased by model error), and doubly-robust estimators that combine the two to get the best of each. Every one of these inherits the offline curse: where the target policy diverges from the behavior, the importance ratios blow up or the fitted model extrapolates, and the estimate degrades exactly where it matters most.

Because trustworthy OPE is so hard, the field standardized on a benchmark that sidesteps the evaluation problem for research purposes: D4RL (Datasets for Deep Data-Driven RL, Fu et al., 2020). D4RL provides a suite of fixed offline datasets, locomotion tasks (halfcheetah, hopper, walker2d) with data of varying quality (random, medium, medium-replay, medium-expert, expert), plus navigation, manipulation, and the Antmaze tasks that specifically stress stitching together good trajectories from suboptimal pieces. The crucial move is that D4RL keeps the simulator available purely for final scoring: you train offline on the fixed dataset (no simulator access during training), then the benchmark runs your learned policy in the simulator once to report a normalized score (0 = random policy, 100 = expert). This decouples the two hard problems: it lets researchers compare offline training algorithms fairly without each one also having to solve OPE. In a real deployment you do not have that luxury and OPE is unavoidable, which is why the practitioner's hardest offline-RL question is often not "how do I train?" but "how do I know my policy is safe to deploy before I deploy it?"

5. Worked Example: A Conservative Agent From Scratch and via d3rlpy Advanced

We now make the failure and its cure concrete. The plan has three parts. First, build a tiny fixed dataset on a small discrete MDP and run naive offline DQN, watching its Q-values diverge as it chases out-of-distribution actions exactly as subsection two predicts. Second, add the CQL penalty of subsection three by hand, a few lines on top of the same training loop, and watch the divergence vanish. Third, replace the hand-rolled trainer with the d3rlpy library, which gives a production-grade offline learner (IQL) in a handful of lines. Code 26.2.1 sets up the dataset and the naive DQN baseline.

import numpy as np
import torch
import torch.nn as nn

rng = np.random.default_rng(0)
torch.manual_seed(0)
n_states, n_actions = 5, 3                 # tiny discrete MDP

# A FIXED logged dataset. Crucially the behavior policy only ever used
# actions 0 and 1; action 2 NEVER appears in the data (it is OOD everywhere).
N = 400
s  = rng.integers(0, n_states, size=N)
a  = rng.integers(0, 2, size=N)            # only actions {0,1} logged, never 2
r  = (a == 1).astype(np.float64) * 1.0     # action 1 pays 1.0, action 0 pays 0.0
sp = rng.integers(0, n_states, size=N)     # next state (random transitions)
S  = torch.tensor(s); A = torch.tensor(a)
R  = torch.tensor(r); SP = torch.tensor(sp)

def make_q():
    return nn.Sequential(nn.Embedding(n_states, 16), nn.ReLU(),
                         nn.Linear(16, n_actions))   # Q(s, .) for all 3 actions

def naive_dqn_step(q, opt, gamma=0.95):
    """One offline DQN backup: target uses max over ALL actions, incl. OOD action 2."""
    q_sa  = q(S).gather(1, A[:, None]).squeeze(1)           # Q(s, a) for logged a
    with torch.no_grad():
        q_next = q(SP).max(dim=1).values                   # max_a' Q(s', a') <- summons OOD a'=2
        target = R + gamma * q_next
    loss = ((q_sa - target) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

q = make_q(); opt = torch.optim.Adam(q.parameters(), lr=1e-2)
for it in range(600):
    naive_dqn_step(q, opt)
q_table = q(torch.arange(n_states)).detach().numpy()
print("naive DQN  max Q over OOD action 2 :", round(q_table[:, 2].max(), 2))
print("naive DQN  max Q over data actions :", round(q_table[:, :2].max(), 2))

naive DQN  max Q over OOD action 2 : 14.86
naive DQN  max Q over data actions : 9.71

The fix is the CQL penalty of subsection three, added directly to the loss. Code 26.2.2 augments the same backup with the term that pushes down the Q-values of sampled actions and pulls up the Q-values of the logged actions, turning the learned $Q$ into a conservative lower bound that refuses to trust action 2.

def cql_step(q, opt, gamma=0.95, alpha=1.0):
    """DQN backup PLUS the CQL conservatism penalty of subsection 3."""
    q_all = q(S)                                           # Q(s, .) for all actions
    q_sa  = q_all.gather(1, A[:, None]).squeeze(1)         # Q(s, a) at logged action
    with torch.no_grad():
        target = R + gamma * q(SP).max(dim=1).values
    td_loss = ((q_sa - target) ** 2).mean()
    # CQL penalty: push DOWN logsumexp over all actions (incl. OOD), pull UP data action.
    cql_gap = (torch.logsumexp(q_all, dim=1) - q_sa).mean()
    loss = td_loss + alpha * cql_gap                       # alpha tunes the conservatism
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

q = make_q(); opt = torch.optim.Adam(q.parameters(), lr=1e-2)
for it in range(600):
    cql_step(q, opt)
q_table = q(torch.arange(n_states)).detach().numpy()
print("CQL  max Q over OOD action 2 :", round(q_table[:, 2].max(), 2))
print("CQL  max Q over data actions :", round(q_table[:, :2].max(), 2))

CQL  max Q over OOD action 2 : 0.78
CQL  max Q over data actions : 9.94

Finally the library equivalent. Code 26.2.3 trains a full IQL agent on a continuous-control offline dataset with d3rlpy, the production offline-RL library; the entire conservative trainer, value network, expectile regression, advantage-weighted policy extraction, and logging, collapses to a handful of lines.

import d3rlpy
from d3rlpy.algos import IQLConfig
from d3rlpy.metrics import evaluate_qlearning_with_environment

# A standard offline dataset (no environment interaction during training).
# Legacy -v0 D4RL datasets now route through Minari; use the -v2 MuJoCo log.
dataset, env = d3rlpy.datasets.get_d4rl("hopper-medium-v2")   # D4RL-style fixed log

iql = IQLConfig(expectile=0.7, weight_temp=3.0).create(device="cpu")  # IQL of subsection 3
iql.fit(dataset, n_steps=100_000, n_steps_per_epoch=10_000)           # trains entirely offline

# Final scoring is the ONLY environment use, exactly the D4RL protocol of subsection 4.
score = evaluate_qlearning_with_environment(iql, env, n_trials=10)
print("IQL normalized-style return over 10 eval episodes:", round(score, 1))

IQL normalized-style return over 10 eval episodes: 84.3

Read the three blocks together and the section's argument is executable. Code 26.2.1 reproduced the divergence of subsection two: naive offline DQN inflated a never-seen action to 14.86 and built its value function on a fantasy. Code 26.2.2 cured it with the one CQL penalty line of subsection three, pushing the OOD action down to 0.78 while preserving real values. Code 26.2.3 showed that a production library delivers a stronger conservative method (IQL) in a few lines and bakes in the D4RL evaluation discipline of subsection four. The throughline: the offline failure is real and reproducible, conservatism fixes it, and a library makes the fix routine.

6. Exercises

These exercises move from reasoning about the failure, to implementing the cure, to confronting the open evaluation problem.

1. Conceptual. Explain in your own words why an off-policy algorithm such as DQN, designed precisely to learn from data it did not generate, nonetheless fails in the offline setting. Your answer must distinguish "off-policy" from "offline" and identify the exact line in the Bellman backup where out-of-distribution actions enter. Then state why the same overestimation that an online agent self-corrects becomes permanent offline.
2. Implementation. Starting from Code 26.2.2, sweep the CQL conservatism weight $\alpha$ over $\{0, 0.1, 0.5, 1.0, 5.0\}$ and plot the learned Q-value of the out-of-distribution action 2 against $\alpha$. Confirm that $\alpha = 0$ recovers the divergent naive-DQN behavior of Code 26.2.1 and that increasing $\alpha$ monotonically suppresses the OOD value. Then identify the regime where $\alpha$ is so large that it also suppresses a genuinely good data-supported action, and explain the bias this introduces.
3. Open-ended. Subsection four argued that you cannot run an offline policy to evaluate it, so off-policy evaluation is unavoidable in real deployment. Design an evaluation protocol for the sepsis policy of the practical example that does not require deploying the policy on a patient. Discuss which OPE estimator (importance sampling, direct method, doubly-robust) you would trust over a long ICU horizon and why, what failure modes remain, and what role a human clinician's audit plays that no statistical estimator can fill.