Part Overview
The book's first five parts taught models to understand and predict time. Part VI, its largest after Part III, teaches them to act in it. Across eight chapters it develops the full theory and practice of sequential decision making: Markov decision processes and dynamic programming, partial observability, bandits and the foundations of reinforcement learning, deep and advanced RL, optimal control and imitation, the recent reframing of RL as sequence modeling, and world models that plan in a learned latent space. This is where prediction becomes policy, and where the temporal models of earlier parts find their most demanding application.
The progression is carefully staged. Chapter 22 establishes MDPs, value functions, and Bellman equations. Chapter 23 confronts partial observability with belief states and recurrent and SSM-based agents, closing a temporal-thread arc that runs from the Kalman filter through filtering to belief tracking. Chapter 24 builds up from bandits and regret to temporal-difference learning and Q-learning. Chapters 25 and 26 cover deep RL (DQNs, policy gradients, actor-critic) and its advanced frontiers (model-based, offline, multi-agent, safe, and hierarchical RL). Chapter 27 brings in the control and imitation traditions: LQR, MPC, behavioral cloning, inverse RL, and RLHF. Chapter 28 reframes decision making as sequence modeling with the Decision Transformer and diffusion planners, explicitly reusing the architectures of Part III as policies. Chapter 29 closes with world models and latent-space planning.
Two temporal-thread arcs converge here: the MDP of Chapter 22 returns as a sequence model in Chapter 28, and the filtering and belief-tracking line resurfaces in the recurrent agents of Chapter 23. The autonomous-systems and robotics applications previewed throughout point toward Chapter 35.5. Part VI is the conceptual summit of the book, the part where understanding time and acting over time become one problem.
The Markov property, value functions, Bellman equations, and the dynamic programming that solves them exactly.
Belief states, approximate solvers, and recurrent and SSM-based agents that remember what they cannot directly see.
Multi-armed and contextual bandits, regret, and the temporal-difference and Q-learning roots of RL.
DQNs, policy gradients, actor-critic methods, continuous control, and the realities of training deep RL.
Model-based, offline, multi-agent, safe, and hierarchical RL with temporal abstraction over options.
LQR and MPC, behavioral cloning and DAgger, inverse RL, and preference-based RL with RLHF.
RL recast as sequence modeling: Decision Transformer, return conditioning, and diffusion planners reusing Part III as policies.
Latent environment models, Dreamer, and planning in latent space for long-horizon, model-based control.
Where This Part Leads
With prediction and action unified, the book turns to assembling complete, reasoning systems. Part VII: Building Intelligent Temporal Systems builds on the policies and planners developed here, adding temporal reasoning and causality, memory-equipped temporal agents, and spatio-temporal intelligence that ties the decision-making machinery into the wider world.