"For years they told me to maximize a reward, and I obliged, climbing whatever scalar they handed me with the single-mindedness of a creature that has never doubted its objective. Then one day a control engineer looked at my flailing and said, quietly, that she already knew how her system moved, that she had written the equations down years ago, and that she could simply solve for the best action without my thousand reckless rollouts. Beside her sat a demonstrator who had never written an equation at all but could show me, again and again, exactly what good looked like, and asked only that I copy him and ask when I drifted. I had spent my whole life inventing rewards to chase, and here were two people who needed none: one had the map, the other had the example. I have been quieter ever since, and considerably more useful."
A Reward Function Quietly Made Redundant
Chapter Overview
The reinforcement learning of the last few chapters made one demanding assumption: that you have a scalar reward signal and the patience to learn from it by trial and error, accumulating experience through rollouts that earn nothing while they teach. That assumption is often false, and this chapter is about the two situations where it is. Sometimes you already know how your system behaves, because an engineer wrote the dynamics down years ago, and the right move is not to relearn them by flailing but to solve directly for the best action. Other times you cannot write a reward at all, but you can show what good looks like, because an expert is willing to demonstrate, or a human is willing to compare two behaviors and say which is better. This chapter develops both practical alternatives to reward-driven reinforcement learning: classical optimal control when you have a model, and imitation and preference learning when you have demonstrations or human feedback.
The first half returns to the oldest and most rigorous corner of sequential decision making. When the dynamics are linear and the cost is quadratic, the linear-quadratic regulator (LQR) gives the optimal controller in closed form, no rollouts required, just a backward recursion that any laptop solves in milliseconds. Add Gaussian process and measurement noise and you cannot observe the state directly, and the optimal controller separates cleanly into a familiar pair: a Kalman filter from Chapter 7 that estimates the hidden state, and an LQR gain that acts on the estimate. That combination, the linear-quadratic-Gaussian (LQG) controller, is the Kalman filter you already know turned from forecasting toward control. When the system is nonlinear or constrained, model predictive control (MPC) keeps the same idea but re-solves a short-horizon optimization at every step, replanning as the world moves, which is exactly the loop a learned world model from Chapter 25 and later world-model chapters plug into when the dynamics are learned rather than derived.
The second half drops the model and picks up the teacher. Behavioral cloning treats imitation as plain supervised learning, mapping observed states to expert actions, and fails in a characteristic way: the moment the agent drifts off the expert's distribution it sees states the expert never demonstrated, and small errors compound into catastrophe. DAgger fixes the drift by interleaving execution and expert correction so the training distribution matches the agent's own. Inverse reinforcement learning asks the deeper question, recovering the reward the expert must have been optimizing rather than copying actions, and generative adversarial imitation learning (GAIL) makes that practical by casting imitation as a two-player game, the very generative-adversarial machinery of the GANs you met earlier turned toward matching expert behavior rather than expert images. Finally, when there is no expert trajectory but a human can compare outcomes, preference-based reinforcement learning fits a reward model to pairwise comparisons, the recipe that became reinforcement learning from human feedback (RLHF) and aligned the large language models of Part III.
Read together, the chapter is the bridge from prediction into control. The state estimators of Part II and the world models of Part VI stop being ends in themselves and become the model that LQR and MPC act on; the sequence models that forecast a future become the simulators a controller plans through. Whether the model is derived or learned, whether the objective is written or demonstrated or merely preferred, the unifying move is the same: turn what you already have, equations, examples, or judgments, into a policy, without ever needing the trial-and-error reward signal that reinforcement learning demands.
Prerequisites
This chapter assumes you can formulate a sequential decision problem and reason about value, and it leans on three earlier chapters in particular. From Chapter 22: Markov Decision Processes you need dynamic programming and the Bellman recursion, because LQR is dynamic programming solved in closed form for the linear-quadratic case and MPC is dynamic programming truncated to a short receding horizon, so the backward recursions in the first half are the planning you already know specialized to continuous states and costs. From Chapter 7: State-Space Models and Filtering you need the Kalman filter and its predict-update loop, because the linear-quadratic-Gaussian controller is exactly a Kalman state estimate fed into an LQR gain, and the separation principle that justifies this is the conceptual hinge of Section 27.1. From Chapter 25: Deep Reinforcement Learning you need policy and value networks and the adversarial training of generative models, since GAIL in Section 27.4 reuses the generator-versus-discriminator game and RLHF in Section 27.5 fine-tunes a policy network against a learned reward model with the same policy-gradient machinery. Readers wanting to refresh linear algebra, the calculus of optimization, or the probability behind Gaussian noise and Bayesian updates will find those refreshers through the Table of Contents.
If you keep one idea from this chapter, keep this: when you have a model you can solve directly for the best action with optimal control, and when you have demonstrations or human judgments you can learn a policy by imitation or preference, so reward-driven trial and error is only one of three ways to get a controller. LQR gives the optimal linear-quadratic controller in closed form and LQG adds a Kalman filter to handle hidden state; MPC re-solves a short-horizon plan at every step for nonlinear and constrained systems; behavioral cloning is supervised imitation that drifts, and DAgger fixes the drift; inverse RL and GAIL recover or match the expert's intent rather than copy actions, with GAIL reusing the GAN's adversarial game; and preference-based RL fits a reward model to human comparisons, the RLHF recipe that aligned modern language models.
Chapter Roadmap
- 27.1 LQR and the Linear-Quadratic-Gaussian Setting Optimal control in closed form: the linear-quadratic regulator as dynamic programming solved exactly via the Riccati recursion, and the linear-quadratic-Gaussian controller that pairs an LQR gain with a Kalman filter from Chapter 7 under noise, the separation principle made concrete.
- 27.2 Model Predictive Control (MPC) Replanning as the world moves: solving a short receding-horizon optimization at every step to handle nonlinear dynamics and hard constraints, the sampling-based MPPI variant, and the loop a learned world model plugs into when the dynamics are learned rather than derived.
- 27.3 Behavioral Cloning and DAgger Imitation as supervised learning: mapping states to expert actions, the compounding-error failure when the agent drifts off the expert's distribution, and DAgger's fix of interleaving execution and expert correction so the training distribution matches the agent's own.
- 27.4 Inverse Reinforcement Learning and GAIL Recovering intent rather than copying actions: inferring the reward an expert must have optimized through maximum-entropy inverse RL, and generative adversarial imitation learning that casts behavior matching as a two-player game, the GAN's adversarial machinery turned toward expert trajectories.
- 27.5 Preference-Based RL and RLHF Learning from comparisons when no expert trajectory exists: fitting a Bradley-Terry reward model to pairwise human preferences, optimizing a policy against it, and the reinforcement-learning-from-human-feedback recipe that aligned the large language models of Part III, with direct preference optimization as the simpler successor.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single progression. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have stabilized a system with the LQR of Section 27.1, tracked a reference under disturbance with the MPC of Section 27.2, cloned an expert and then repaired its drift with the behavioral cloning and DAgger of Section 27.3, and fit a reward model from preferences following the RLHF recipe of Section 27.5.
Hands-On Lab: Control by Model and by Demonstration
Objective
Build a controller four ways, once from a model and three times from demonstrations or judgments, and watch each approach succeed where reward-driven trial and error would have struggled. You will first stabilize a classic unstable system with the linear-quadratic regulator of Section 27.1, solving the Riccati equation once and feeling the optimal feedback gain hold the system upright with no rollouts at all. You will then make the task harder, asking the system to track a moving reference while a disturbance pushes against it under a hard actuation limit, and meet it with the receding-horizon replanning of the model predictive control of Section 27.2. Switching from model to teacher, you will clone an expert controller with behavioral cloning from Section 27.3, watch it drift off the expert's distribution and fail, and repair it with DAgger by letting the expert correct the states the clone actually visits. Finally, with no expert trajectory at all, you will collect pairwise preferences over trajectory pairs and fit a Bradley-Terry reward model following the RLHF recipe of Section 27.5 and the inverse-reward idea of Section 27.4, recovering an objective from judgments alone. The single thread is that a policy can come from equations, from examples, or from comparisons, and reward-driven reinforcement learning is only one road among several.
What You'll Practice
- Solving the discrete-time Riccati equation and applying the optimal LQR feedback gain to stabilize a system, following Section 27.1.
- Implementing a receding-horizon MPC controller that tracks a reference under disturbance and actuation constraints, following Section 27.2.
- Training a behavioral-cloning policy, diagnosing its compounding-error drift, and fixing it with DAgger, following Section 27.3.
- Fitting a Bradley-Terry reward model to pairwise preferences and reading it as a recovered objective, following Section 27.4 and Section 27.5.
Setup
You need a Python environment with numpy and scipy for the control algorithms and a small neural-network framework such as torch for the cloning and reward-model stages, plus gymnasium to supply the cart-pole and pendulum systems that are the standard stages for control labs. The discipline that governs the model-based half of the lab is that LQR and MPC act on a model of the dynamics, so keep the linearization explicit and honest: an LQR gain computed from a wrong linear model will confidently drive the true system into the floor, and the failure looks exactly like a code bug. The code below solves the discrete-time algebraic Riccati equation and forms the LQR gain, the closed-form optimal controller that every later stage of the lab is measured against.
import numpy as np
from scipy.linalg import solve_discrete_are
def lqr_gain(A, B, Q, R):
"""Optimal feedback gain K for u = -K x, minimizing the
infinite-horizon quadratic cost sum(x'Qx + u'Ru)."""
P = solve_discrete_are(A, B, Q, R) # solve the Riccati equation once
K = np.linalg.solve(R + B.T @ P @ B, B.T @ P @ A)
return K # no rollouts: closed-form optimal
Steps
Step 1: Stabilize a system with LQR
Linearize the cart-pole around its upright equilibrium, choose state and action cost matrices, compute the LQR gain with the function above, and apply the feedback law to hold the pole upright from a small perturbation, the closed-form control of Section 27.1. Vary the cost weights and watch the controller trade actuation effort against settling time, the quadratic cost made tangible.
Step 2: Track a reference under disturbance with MPC
Ask the system to track a moving reference while a disturbance pushes against it and the actuator saturates at a hard limit, and meet it with the receding-horizon MPC of Section 27.2. Solve the short-horizon optimization at every step, apply only the first action, and replan, and confirm that MPC respects the constraint that LQR alone would violate.
Step 3: Clone an expert with behavioral cloning
Use the LQR or MPC controller from Steps 1 and 2 as an expert, collect its state-action pairs, and train a behavioral-cloning policy to imitate it by supervised learning, following Section 27.3. Roll the clone out from a fresh start and watch it drift off the expert's distribution into states the expert never visited, where small errors compound into failure.
Step 4: Fix the drift with DAgger
Apply DAgger from Section 27.3: roll out the current clone, query the expert for the correct action on the states the clone actually visited, add those corrections to the dataset, and retrain. Repeat for a few iterations and confirm that aligning the training distribution with the agent's own distribution closes the compounding-error gap that plain cloning could not.
Step 5: Train a reward model from preferences
Drop the expert entirely and present pairs of short trajectories, labeling each pair by which behavior is preferred, then fit a Bradley-Terry reward model that assigns higher reward to the preferred trajectory, the RLHF recipe of Section 27.5 and the recovered-objective idea of Section 27.4. Inspect the learned reward over states and confirm it ranks behaviors the way your preferences did, an objective reconstructed from judgments alone.
Expected Output
The lab produces four controllers and one clear lesson about where each comes from. Step 1 yields an LQR controller that stabilizes the system instantly from the Riccati solution, with cost-weight sweeps showing the effort-versus-speed trade-off as a family of trajectories. Step 2 yields an MPC controller that tracks the moving reference and honors the actuation limit, replanning visibly at every step where the unconstrained LQR would have saturated and failed. Steps 3 and 4 produce the chapter's signature imitation contrast: a behavioral-cloning policy that tracks the expert briefly and then diverges into compounding failure, beside a DAgger policy that, after a few rounds of expert correction on its own visited states, holds the task as well as the expert it learned from. Step 5 produces a learned reward model whose ranking of trajectories matches the preferences you supplied, demonstrating that an objective can be reconstructed from comparisons with no demonstration and no hand-written reward at all. The reader finishes able to build a controller from a model, from demonstrations, and from preferences, and to recognize from the failure mode alone which ingredient a given problem actually supplies.
The from-scratch controllers of this lab are small on purpose, but the surrounding machinery is standardized so you never rebuild it. The LQR gain of Step 1 is a single scipy.linalg.solve_discrete_are call wrapped by the python-control library, whose lqr and dlqr functions collapse the whole controller into one line and add analysis tools for stability and frequency response. The imitation half is covered by the imitation library, which ships tested implementations of behavioral cloning, DAgger, GAIL, AIRL, and preference-based reward learning that wrap a Stable-Baselines3 policy, turning Steps 3 through 5 into a few configuration lines. The discipline the chapter teaches still governs the libraries: an LQR gain computed from a wrong linearization, a DAgger run whose expert is queried on the wrong states, or a reward model fit on inconsistent preferences will each report a confident and wrong controller in exactly the same few lines. The library makes the optimization free; it does not make the model correct or the demonstrations honest.
Stretch Goals
- Replace the linear LQR of Step 1 with the iterative LQR (iLQR) algorithm on the full nonlinear pendulum, relinearizing along the trajectory, and observe how the closed-form local solver extends to nonlinear swing-up where a single LQR gain fails.
- Swap the behavioral cloning of Step 3 for GAIL from Section 27.4, training a discriminator to distinguish expert from agent trajectories, and confirm that adversarial imitation matches the expert from fewer demonstrations than cloning by reusing the GAN game.
- Extend the reward model of Step 5 into a full preference-optimization loop, fine-tuning the policy against the learned reward and then comparing the result to direct preference optimization (DPO), which skips the explicit reward model entirely, the simpler RLHF successor of Section 27.5.
What's Next?
This chapter built controllers without the reward-driven trial and error of reinforcement learning, either by solving a known model or by learning from a teacher. Both halves still treated the policy as a function from state to action, fit by optimization or imitation one step at a time. Chapter 28: Sequence Models for Decision Making takes the opposite view: it treats the entire problem of acting, the stream of states, actions, and returns, as a sequence to be modeled directly, bringing the Transformers and sequence architectures of Part III to bear on decision making and recasting the Markov decision process of Chapter 22 as something a sequence model can predict its way through. The behavioral cloning of this chapter reappears there as the simplest sequence model of action, and the preference learning of RLHF reappears as the alignment step that fine-tunes those sequence models. The full path through Part VI and the rest of the book is laid out in the Table of Contents.
Bibliography & Further Reading
Optimal Control Foundations
Kalman, R. E. "Contributions to the Theory of Optimal Control." Boletin de la Sociedad Matematica Mexicana, 5, 102-119, 1960. liberzon.csl.illinois.edu/.../kalman_bucy.pdf
The founding paper of the linear-quadratic regulator, deriving the optimal feedback controller and the Riccati equation that underpin Section 27.1.
Bertsekas, D. P. "Dynamic Programming and Optimal Control." 4th ed., Athena Scientific, 2017. athenasc.com/dpbook.html
The standard graduate reference on dynamic programming for control, covering LQR, LQG, and the separation principle in the framework that Section 27.1 specializes from Chapter 22.
Model Predictive Control
Rawlings, J. B., Mayne, D. Q., Diehl, M. M. "Model Predictive Control: Theory, Computation, and Design." 2nd ed., Nob Hill Publishing, 2017. sites.engineering.ucsb.edu/~jbraw/mpc
The definitive modern textbook on model predictive control, covering the receding-horizon formulation, stability, and constraint handling that Section 27.2 builds on.
Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., Theodorou, E. A. "Information Theoretic MPC for Model-Based Reinforcement Learning (MPPI)." ICRA, 2017. ieeexplore.ieee.org/document/7989202
The paper that introduced model predictive path integral control, the sampling-based MPC variant of Section 27.2 that scales to nonlinear dynamics and learned models.
Imitation Learning
Pomerleau, D. A. "ALVINN: An Autonomous Land Vehicle in a Neural Network." NeurIPS, 1988. proceedings.neurips.cc/paper/1988
The classic behavioral-cloning result that trained a neural network to drive by imitating a human, the original supervised-imitation approach of Section 27.3.
Ross, S., Gordon, G. J., Bagnell, J. A. "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger)." AISTATS, 2011. arXiv:1011.0686. arxiv.org/abs/1011.0686
The paper that introduced DAgger and explained the compounding-error failure of behavioral cloning, the distribution-correction fix at the heart of Section 27.3.
Inverse Reinforcement Learning and GAIL
Ng, A. Y., Russell, S. "Algorithms for Inverse Reinforcement Learning." ICML, 2000. ai.stanford.edu/~ang/papers/icml00-irl.pdf
The paper that founded inverse reinforcement learning, framing the recovery of an expert's reward function as the inverse of the control problem, the premise of Section 27.4.
Ziebart, B. D., Maas, A., Bagnell, J. A., Dey, A. K. "Maximum Entropy Inverse Reinforcement Learning." AAAI, 2008. cdn.aaai.org/AAAI/2008/AAAI08-227.pdf
The maximum-entropy formulation that made inverse RL well-posed and probabilistic, the principled reward-recovery method of Section 27.4.
Ho, J., Ermon, S. "Generative Adversarial Imitation Learning (GAIL)." NeurIPS, 2016. arXiv:1606.03476. arxiv.org/abs/1606.03476
The paper that cast imitation as a two-player adversarial game, reusing the GAN machinery to match expert behavior without recovering an explicit reward, the centerpiece of Section 27.4.
Fu, J., Luo, K., Levine, S. "Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (AIRL)." ICLR, 2018. arXiv:1710.11248. arxiv.org/abs/1710.11248
The adversarial inverse-RL method that recovers a transferable reward function rather than only a policy, extending GAIL toward the reward-recovery goal of Section 27.4.
Preference-Based RL and RLHF
Christiano, P., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D. "Deep Reinforcement Learning from Human Preferences." NeurIPS, 2017. arXiv:1706.03741. arxiv.org/abs/1706.03741
The paper that learned a reward model from pairwise human comparisons and optimized a policy against it, the founding preference-based RL recipe of Section 27.5.
Ouyang, L., et al. "Training Language Models to Follow Instructions with Human Feedback (InstructGPT)." NeurIPS, 2022. arXiv:2203.02155. arxiv.org/abs/2203.02155
The paper that scaled RLHF to align large language models, the application of the Section 27.5 recipe that connects this chapter to the foundation models of Part III.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model (DPO)." NeurIPS, 2023. arXiv:2305.18290. arxiv.org/abs/2305.18290
The method that optimizes a policy directly from preferences without fitting an explicit reward model, the simpler successor to RLHF discussed in Section 27.5.
Tools and Libraries
Fuller, S., et al. "python-control: The Python Control Systems Library." python-control.readthedocs.io
The standard Python library for control-systems analysis and design, supplying the LQR, LQG, and state-space tools behind the model-based half of the lab.
Gleave, A., et al. "imitation: Clean Implementations of Imitation and Reward Learning Algorithms." imitation.readthedocs.io
A tested library of behavioral cloning, DAgger, GAIL, AIRL, and preference-based reward learning that wraps Stable-Baselines3 policies, covering Sections 27.3 to 27.5 in a few lines.