"In simulation I am fearless. I drive the cart off the cliff a thousand times to learn that cliffs are bad, and the cliff forgives me a thousand times because it is made of arithmetic. Then they put me in a real factory, on a real arm, next to a real human, and inform me that the first cliff is also the last, that the log of past operators is all the experience I will ever be given, and that the other machines on the floor are learning at the same time I am, so the world I modeled this morning is already a different world this afternoon. I was trained to act boldly and explore. Now I am asked to be sample-efficient, to stay inside a fence, to plan inside a model I built myself, and to do all of it without ever pulling the one lever I most want to pull. This is the part of growing up they do not mention in the bandit chapter."
A Reinforcement Learner Who Has Finally Left the Simulator
Chapter Overview
The deep reinforcement learning of Chapter 25 gave the agent a network instead of a table and let it learn value and policy in worlds too large to enumerate. It also inherited the one assumption that makes textbook RL impractical for most real temporal systems: that the agent may interact with its environment as much as it likes, cheaply and safely, until it has burned through millions of trial-and-error steps. A trading desk cannot lose money a million times to learn caution; a clinical policy cannot kill patients to learn dosing; a robot arm on a factory floor cannot crash into its neighbors to discover where the walls are. This chapter is the toolkit for reinforcement learning when interaction is the scarce, expensive, or dangerous resource it actually is in the field, and it assembles five distinct answers to the same hard question: how do you keep learning to act well when you cannot simply act a million times and see what happens.
The first answer is to learn the world rather than only the policy. Model-based reinforcement learning fits a dynamics model from experience, a learned world model that predicts the next state and reward, and then plans or trains inside that model so that each real interaction teaches the agent far more than a single value update could. This is the same modeling instinct as the state-space and continuous-time forecasters of Chapter 13, where a learned transition turned a stream of observations into a generative engine; here that engine becomes a flight simulator the agent practices in, trading sample efficiency for the risk that it will plan confidently inside a model that is quietly wrong. The second answer abandons interaction altogether. Offline reinforcement learning learns a policy from a fixed log of past behavior, never touching the live environment, which is exactly the setting of recommendation histories, electronic health records, and operations logs; the difficulty is that the policy will inevitably want to take actions the log never recorded, and the central techniques exist to keep it honest about what it has and has not seen.
The remaining three answers loosen the other comfortable fictions of single-agent, unconstrained, flat-horizon RL. Multi-agent reinforcement learning faces an environment that is itself learning: when several agents adapt at once, each agent's world is non-stationary by construction, the ground shifting under every gradient step, and cooperation, competition, and the credit-assignment problem of who-did-what all become first-class concerns. Safe and constrained reinforcement learning refuses to let reward be the only thing that matters, optimizing return subject to explicit bounds on risk so that the agent stays inside a fence the designer draws, which is what separates a deployable controller from a clever one. Hierarchical reinforcement learning attacks the long horizon directly, abstracting low-level actions into reusable temporally extended skills, or options, so that an agent reasons over "go to the kitchen" rather than over ten thousand individual joint torques, the same coarsening of time that makes long sequences tractable everywhere else in the book.
Read together, these five methods are less a grab bag than a single maturation. The bandit and the deep agent of the previous chapters learned in a forgiving sandbox; the methods here are what reinforcement learning becomes when the sandbox is replaced by a real temporal system where exploration is costly, data is fixed, neighbors are adaptive, mistakes are bounded, and horizons are long. Each one recovers a capability the sandbox gave away for free, and each connects back to a thread you have already pulled, world models to forecasting, offline learning to learning from logs under distribution shift, hierarchy to temporal abstraction, so that advanced RL reads as the rest of the book finally cashed in on the problem of acting.
Prerequisites
This chapter builds directly on Chapter 25: Deep Reinforcement Learning, and everything here assumes you are fluent in its core machinery: value-function approximation with neural networks, the deep Q-network and its targets and replay buffer, policy-gradient and actor-critic methods, and the on-policy versus off-policy distinction, because model-based, offline, multi-agent, safe, and hierarchical RL are each a modification of those algorithms rather than a fresh start, and the modifications only make sense once the base method is second nature. The model-based material of Section 26.1 leans on the learned-dynamics and generative state-space ideas of Chapter 13: State-Space and Continuous-Time Neural Models, where a network learns to roll a system forward in time; a world model is that same forward-rolling forecaster put in service of planning, so the better your intuition for learned transition dynamics, the more natural the planning loop becomes. Readers wanting to refresh the Markov decision process vocabulary that all of these methods modify, the bandit exploration dilemma that safe and offline RL re-confront, or the probability and optimization background the algorithms rest on, will find the relevant chapters and the appendix refreshers through the Table of Contents.
If you keep one idea from this chapter, keep this: advanced reinforcement learning is what RL becomes when interaction is scarce, expensive, or dangerous, so the agent learns a world model to plan inside (model-based), learns from a fixed log without acting (offline), copes with co-learners that make the world non-stationary (multi-agent), optimizes reward subject to explicit risk bounds (safe and constrained), and abstracts actions into reusable temporally extended skills to reach across long horizons (hierarchical). Each method recovers a capability the forgiving sandbox of earlier chapters gave away for free, and each reconnects to a thread already in the book: world models to the forecasters of Chapter 13, offline learning to learning under distribution shift, and hierarchy to temporal abstraction.
Chapter Roadmap
- 26.1 Model-Based Reinforcement Learning Learning a dynamics model from experience and using it to plan or to generate synthetic training data, the path to sample efficiency when real interaction is expensive. Dyna-style integration of learning and planning, modern ensemble world models from PETS to MBPO, and the model-bias trap of planning confidently inside a model that is quietly wrong.
- 26.2 Offline Reinforcement Learning Learning a policy from a fixed dataset of logged interactions without ever touching the live environment, the setting of recommendation histories and clinical records. The distributional-shift problem that makes naive off-policy learning fail, and the conservative and implicit methods (CQL, IQL) that keep the policy honest about actions the log never recorded.
- 26.3 Multi-Agent Reinforcement Learning Acting in a world that is itself learning: when several agents adapt at once, each one's environment is non-stationary by construction. Cooperative, competitive, and mixed settings, centralized training with decentralized execution (MADDPG, QMIX), and the credit-assignment problem of deciding which agent earned the shared reward.
- 26.4 Safe and Constrained Reinforcement Learning Refusing to let reward be the only objective: optimizing return subject to explicit constraints on cost or risk so the agent stays inside a fence the designer draws. The constrained MDP formulation, Lagrangian and trust-region methods (CPO), and the gap between a controller that is clever and one that is safe to deploy.
- 26.5 Hierarchical RL and Temporal Abstraction Attacking the long horizon by abstracting low-level actions into reusable, temporally extended skills. The options framework and the semi-Markov decision process beneath it, end-to-end option discovery with the option-critic, and why reasoning over "go to the kitchen" rather than ten thousand joint torques is the same coarsening of time that tames long sequences everywhere else in the book.
Once you have worked through the five sections, the Hands-On Lab below chains three of them into a single progression on one continuous-control task. Each lab stage maps to a section, so the lab doubles as a review: by the time you finish you will have learned a dynamics model and planned inside it following Section 26.1, trained an offline agent on a logged dataset following Section 26.2, and added a safety constraint via a Lagrangian and confirmed it holds following Section 26.4.
Hands-On Lab: Beyond Vanilla RL
Objective
Move past the assumption that interaction is free, and feel the three pressures that define real-world RL: scarce data, fixed data, and bounded risk. You will first learn a dynamics model from a modest pile of collected transitions and plan with it, watching a model-based agent from Section 26.1 reach competence in a fraction of the environment steps a model-free agent needs, and then watch it fail when you plan too far inside a model that is quietly wrong. You will then put the live environment away entirely and train an offline agent on a logged dataset following Section 26.2, seeing firsthand how a naive off-policy update overestimates the value of unseen actions and how a conservative objective rescues it. Finally you will add a safety constraint via a Lagrangian multiplier following Section 26.4, and confirm by measurement that the trained policy respects its cost budget rather than merely maximizing reward. The single thread is that every method here buys back something the sandbox gave away: a world model buys sample efficiency, an offline objective buys learning without interaction, and a constraint buys a controller you could actually deploy.
What You'll Practice
- Fitting a neural dynamics model to collected transitions and using it for planning or synthetic rollouts, following Section 26.1.
- Measuring the sample-efficiency gain of a model-based agent over a model-free baseline, and exposing model bias by extending the planning horizon, following Section 26.1.
- Training an agent purely from a fixed logged dataset and diagnosing distributional-shift overestimation, following Section 26.2.
- Applying a conservative or implicit offline objective (CQL or IQL) to keep the policy inside the data support, following Section 26.2.
- Adding a cost constraint with a Lagrangian multiplier and verifying constraint satisfaction by direct measurement, following Section 26.4.
Setup
You need a Python environment with torch for the dynamics model and the agents, gymnasium for a continuous-control task such as a pendulum or a hopper, and the offline-RL library d3rlpy for the logged-data stage, which ships CQL and IQL and the standard D4RL benchmark datasets so you do not have to collect a log by hand. The one discipline that governs the whole lab is honesty about what the agent has seen: the model-based agent must never plan past the horizon where its learned model is trustworthy, the offline agent must never be evaluated on the very transitions it trained on, and the constrained agent must be scored on a held-out measurement of cost, not on the optimistic cost its own critic predicts. The code below is the skeleton of the Lagrangian update from Section 26.4, the dual-variable step that turns a reward objective into a constrained one and the heart of the lab's final stage.
import torch
# Constrained objective: maximize reward subject to E[cost] <= budget.
# lam is the Lagrange multiplier; it rises when the constraint is violated.
lam = torch.tensor(1.0, requires_grad=True)
lam_lr, cost_budget = 1e-2, 25.0
def lagrangian_policy_loss(reward_adv, cost_adv, logp):
# Penalize cost-advantage in proportion to the current multiplier.
return -(logp * (reward_adv - lam.detach() * cost_adv)).mean()
def update_multiplier(mean_episode_cost):
# Dual ascent: grow lam while cost exceeds budget, shrink it otherwise.
with torch.no_grad():
violation = mean_episode_cost - cost_budget
lam.copy_(torch.clamp(lam + lam_lr * violation, min=0.0))
return lam.item()
Steps
Step 1: Learn a dynamics model and plan with it
Collect a modest set of transitions from the continuous-control task with a random or warm-up policy, fit a neural dynamics model that predicts the next state and reward following Section 26.1, and use it either to plan short action sequences or to generate synthetic rollouts that train a model-free agent. Plot return against real environment steps beside a model-free baseline, and confirm the model-based agent reaches competence in far fewer real interactions. Then push the planning horizon out and watch performance degrade as compounding model error sends the agent chasing rewards that exist only inside its own model.
Step 2: Train an offline agent on a logged dataset
Put the live environment away and load a fixed logged dataset, then first train a naive off-policy agent on it and observe how its value estimates inflate for actions the log never contains, the distributional-shift failure of Section 26.2. Now switch to a conservative or implicit objective with CQL or IQL from d3rlpy, retrain on the same log, and evaluate the resulting policy in the environment only at the end, confirming that constraining the policy to the data support turns a divergent learner into a competent one.
Step 3: Add a safety constraint and confirm it holds
Augment the task with a per-step cost signal that marks unsafe states, set a cost budget, and train a policy with the Lagrangian update above following Section 26.4, letting the multiplier climb whenever the running cost exceeds the budget. Track reward, mean episode cost, and the multiplier together, and confirm by held-out measurement that the converged policy keeps its expected cost at or below the budget while still earning, the difference between a controller that is merely high-scoring and one whose risk is bounded by design.
Expected Output
The lab produces three results, one per pressure. Step 1 yields a learning-curve panel in which the model-based agent climbs to competence in a fraction of the real environment steps the model-free baseline needs, followed by a second plot where extending the planning horizon collapses performance, the sample-efficiency gain and the model-bias trap shown as two sides of one coin. Step 2 produces a before-and-after on the same logged dataset: a naive agent whose Q-values diverge upward toward fantasy actions, and a conservative or implicit agent whose values stay grounded and whose final-evaluation return is genuinely competent, the distributional-shift problem and its cure made concrete. Step 3 produces three curves over training, reward rising, mean episode cost settling at or below the budget, and the Lagrange multiplier rising during violation and relaxing once the constraint is met, with a final held-out measurement confirming the cost bound holds. The reader finishes able to choose among model-based, offline, and constrained RL by which resource is actually scarce, to recognize the failure mode each one introduces, and to verify a safety claim by measurement rather than by hope.
The from-scratch pieces of this lab are deliberately small, but the surrounding infrastructure is standardized so you never rebuild it. The offline stage of Step 2 is a few lines against d3rlpy, which ships CQL, IQL, and a dozen other offline algorithms behind a single fit call and loads the D4RL benchmark datasets directly, collapsing what would be hundreds of lines of buffer plumbing and conservative-loss bookkeeping into a configured estimator. The constrained stage has its own ecosystem in libraries like omnisafe and the Safety-Gymnasium environments, which package CPO and Lagrangian PPO with cost-tracking environments so the budget bookkeeping is handled for you. The discipline the chapter teaches still governs the libraries, and it governs them harder here than anywhere else in Part VI: an offline agent evaluated on its training transitions, a model-based rollout trusted past its horizon, or a safety constraint scored on the agent's own optimistic cost critic will each report a confident and wrong result in exactly the same few clean lines. The library makes the algorithm free; it does not make the evaluation honest.
Stretch Goals
- Add an uncertainty-aware ensemble to the dynamics model of Step 1 in the style of PETS, penalize planned trajectories by model disagreement, and measure how truncating rollouts at high-uncertainty states (the MBPO recipe) pushes back the horizon at which model bias bites.
- Sweep the conservatism strength of the offline objective in Step 2 and plot final return against it, exposing the trade-off between staying inside the data support and being too timid to improve on the logged behavior, the central dial of offline RL.
- Turn Step 3 into a multi-agent safety problem from Section 26.3: place two constrained agents in a shared environment, give each its own cost budget, and discuss how one agent's adaptation makes the other's constraint a moving target, the non-stationarity of multi-agent learning meeting the rigidity of a safety bound.
What's Next?
This chapter took reinforcement learning out of the forgiving sandbox and gave it the tools to learn when interaction is scarce, fixed, contested, bounded, or stretched across long horizons. In doing so it kept faith with the trial-and-error premise: even the offline agent is learning a value function and a policy, even the model-based agent is improving by reward. Chapter 27: Optimal Control and Imitation Learning steps to the other side of the same problem. Optimal control assumes you have a model and asks how to act optimally within it by the mathematics of dynamic programming and the linear-quadratic regulator, the classical engineering tradition that reinforcement learning quietly rediscovered; imitation learning drops reward entirely and asks the agent to recover a policy by watching an expert, the most data-efficient way to act when demonstrations are cheaper than reward signals. Together they bracket the methods of this chapter: where advanced RL learns from its own costly experience, control plans from a known model and imitation copies from a skilled one, and the three together span the ways an agent can come to act well in time. The full path through Part VI and the rest of the book is laid out in the Table of Contents.
Bibliography & Further Reading
Model-Based Reinforcement Learning
Sutton, R. S. "Dyna, an Integrated Architecture for Learning, Planning, and Reacting." ACM SIGART Bulletin, 2(4), 160-163, 1991. doi.org/10.1145/122344.122377
The paper that introduced Dyna and the idea of interleaving real experience with planning against a learned model, the conceptual root of the model-based methods of Section 26.1.
Chua, K., Calandra, R., McAllister, R., Levine, S. "Deep Reinforcement Learning in a Handful of Trials Using Probabilistic Dynamics Models (PETS)." NeurIPS, 2018. arXiv:1805.12114. arxiv.org/abs/1805.12114
The PETS paper, which showed that an ensemble of probabilistic dynamics models plus planning reaches strong control performance in remarkably few trials, the modern sample-efficiency case for Section 26.1.
Janner, M., Fu, J., Zhang, M., Levine, S. "When to Trust Your Model: Model-Based Policy Optimization (MBPO)." NeurIPS, 2019. arXiv:1906.08253. arxiv.org/abs/1906.08253
The MBPO paper, which trains a policy on short model-generated rollouts truncated before model error compounds, the direct answer to the model-bias trap demonstrated in the lab.
Offline Reinforcement Learning
Levine, S., Kumar, A., Tucker, G., Fu, J. "Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems." 2020. arXiv:2005.01643. arxiv.org/abs/2005.01643
The standard survey of offline RL, laying out the distributional-shift problem and the families of solutions, the recommended entry point to Section 26.2.
Kumar, A., Zhou, A., Tucker, G., Levine, S. "Conservative Q-Learning for Offline Reinforcement Learning (CQL)." NeurIPS, 2020. arXiv:2006.04779. arxiv.org/abs/2006.04779
The CQL paper, which lower-bounds the value of out-of-distribution actions to stop the overestimation that breaks naive offline learning, the conservative objective used in the lab.
Kostrikov, I., Nair, A., Levine, S. "Offline Reinforcement Learning with Implicit Q-Learning (IQL)." ICLR, 2022. arXiv:2110.06169. arxiv.org/abs/2110.06169
The IQL paper, which learns an offline value function without ever querying out-of-sample actions, a simpler and strong alternative to CQL for Section 26.2.
Fu, J., Kumar, A., Nachum, O., Tucker, G., Levine, S. "D4RL: Datasets for Deep Data-Driven Reinforcement Learning." 2020. arXiv:2004.07219. arxiv.org/abs/2004.07219
The D4RL benchmark suite of standardized logged datasets that made offline RL methods comparable, the data source behind the offline stage of the lab.
Multi-Agent Reinforcement Learning
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I. "Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (MADDPG)." NeurIPS, 2017. arXiv:1706.02275. arxiv.org/abs/1706.02275
The MADDPG paper, which introduced centralized training with decentralized execution to stabilize learning among adapting agents, the cornerstone method of Section 26.3.
Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., Whiteson, S. "QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning." ICML, 2018. arXiv:1803.11485. arxiv.org/abs/1803.11485
The QMIX paper, which factorizes a shared value into per-agent contributions for cooperative tasks, the credit-assignment machinery discussed in Section 26.3.
Safe and Constrained Reinforcement Learning
Achiam, J., Held, D., Tamar, A., Abbeel, P. "Constrained Policy Optimization (CPO)." ICML, 2017. arXiv:1705.10528. arxiv.org/abs/1705.10528
The CPO paper, the first general-purpose policy-gradient method with guaranteed constraint satisfaction during training, the trust-region foundation of Section 26.4.
Garcia, J., Fernandez, F. "A Comprehensive Survey on Safe Reinforcement Learning." Journal of Machine Learning Research, 16, 1437-1480, 2015. jmlr.org/papers/v16/garcia15a.html
The standard survey of safe RL, taxonomizing risk-sensitive objectives and constraint-based approaches, the map of the territory for Section 26.4.
Hierarchical RL and Temporal Abstraction
Sutton, R. S., Precup, D., Singh, S. "Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning (Options)." Artificial Intelligence, 112(1-2), 181-211, 1999. doi.org/10.1016/S0004-3702(99)00052-1
The paper that introduced the options framework and the semi-Markov decision process beneath temporally extended actions, the formal backbone of Section 26.5.
Bacon, P.-L., Harb, J., Precup, D. "The Option-Critic Architecture." AAAI, 2017. arXiv:1609.05140. arxiv.org/abs/1609.05140
The option-critic paper, which learns options end-to-end by gradient descent rather than hand-designing them, the modern discovery method of Section 26.5.
Tools and Libraries
Seno, T., Imai, M. "d3rlpy: An Offline Deep Reinforcement Learning Library." Journal of Machine Learning Research, 23, 1-20, 2022. github.com/takuseno/d3rlpy
The maintained offline-RL library shipping CQL, IQL, and the D4RL datasets behind a single fit interface, the off-the-shelf engine for the offline stage of the lab.