"They replaced my table with a network and called it progress. For a thousand episodes I climbed, my reward curve rising like a promise, and on the final seed I was a champion. Then they changed the seed, and I was a fool again, walking into the same wall I had mastered the day before, because nothing I had learned was quite as stable as the curve had let everyone believe. I had not solved the world; I had memorized one lucky run of it. They show the good seed in the paper and keep the other four in a folder named diagnostics. I have learned that a single triumphant curve proves only that triumph was possible once, and that the truest number an agent can report is not its best return but how widely its returns disagree with themselves."
A Deep Q-Network Confident on Exactly One Seed
Chapter Overview
Chapter 24 taught the agent to learn value and control from raw experience, but it learned into a table, one entry per state or state-action pair. That table is also the wall the previous chapter ran into: the moment the state becomes an image, a sensor stream, or a continuous vector, the table explodes past any memory and never sees the same state twice, so the careful Monte Carlo and temporal-difference estimates have nothing to average over. This chapter replaces the table with a neural network. The function approximator generalizes value or policy across states the agent has never visited, and that single change of representation turns the small tabular ideas of Chapter 24 into the methods that play Atari from pixels, control simulated robots, and underlie much of modern decision-making AI. The idea does not change; only what stores it does.
The chapter follows the two great families that the move to neural networks opens, beginning with value-based methods. The deep Q-network (DQN) is Q-learning with a network standing in for the table, and the chapter shows why that substitution is not free: bootstrapping off a network that is itself changing creates instability that tabular Q-learning never faced, and the celebrated DQN result depended on two stabilizers, experience replay and a slowly updated target network, that made the unstable combination converge. From there the chapter walks the variants that each fix a specific pathology: Double DQN corrects the maximization bias that makes Q-learning overestimate values, Dueling DQN separates state value from action advantage, prioritized replay samples the transitions that teach most, and Rainbow combines the lot into one strong baseline.
The second family optimizes the policy directly rather than reading it off a value function. Policy-gradient methods push the parameters of a stochastic policy up the gradient of expected return, which makes them natural for continuous and high-dimensional action spaces where taking a max over actions is infeasible, but the raw gradient estimate has punishing variance. Actor-critic methods cut that variance by learning a value function (the critic) to score the policy's actions (the actor), and the chapter builds the modern workhorses on that frame: A2C and A3C, proximal policy optimization (PPO) with its clipped objective that has become the default on-policy method, and soft actor-critic (SAC), the off-policy maximum-entropy method that dominates continuous control. Continuous control gets its own treatment through DDPG and TD3, the deterministic methods that extend Q-learning into unbounded action spaces.
The chapter ends where deep reinforcement learning is most often misreported: its notorious brittleness. Results swing wildly across random seeds, hyperparameters, and even undocumented implementation details, so a single triumphant learning curve proves almost nothing. The final section is therefore a methodology section, not an algorithm section: how to evaluate across many seeds with confidence intervals rather than a best run, why reproducibility is a first-class engineering problem here, and the honest guidance the field has earned the hard way, namely that deep reinforcement learning is powerful where the simulator is cheap and the reward is dense, and frequently the wrong tool where data is expensive or a simpler controller would do. Knowing when not to reach for it is part of mastering it.
Prerequisites
This chapter is the neural sequel to the tabular foundations, so it assumes you have worked through Chapter 24: Bandits and Foundations of Reinforcement Learning and carry its core machinery intact: the exploration-exploitation dilemma, the temporal-difference update, the off-policy versus on-policy distinction, and Q-learning and SARSA as the tabular algorithms that the deep methods here re-express with a network in place of the table. Every deep method in this chapter is one of those ideas with a function approximator standing in, so the table is the thing you must already understand. From Part III, beginning with Chapter 9: Neural Sequence Modeling Fundamentals, you need fluency with neural networks and PyTorch: forward and backward passes, stochastic-gradient optimization, loss functions, and the training loop, because the deep Q-network, the policy network, and the actor and critic are all ordinary networks trained by ordinary gradient descent, only with targets that the agent generates for itself from its own experience. That self-generated, moving target is exactly what makes the training unstable, which is why the network fluency matters here in a way it did not for supervised forecasting. Readers wanting to refresh the probability behind expectations and variance, the optimization behind gradient descent, or the reinforcement-learning vocabulary these methods build on will find the refreshers through the Table of Contents.
If you keep one idea from this chapter, keep this: deep reinforcement learning replaces the tabular value function or policy with a neural network that generalizes across unseen states, scaling the ideas of Chapter 24 to high-dimensional and continuous worlds, but the moving, self-generated targets that make this possible also make it brittle, so a result is only as trustworthy as its evaluation across many seeds. Value-based methods run from the deep Q-network and its stabilizers (experience replay, target networks) through Double, Dueling, prioritized-replay, and Rainbow; policy-gradient and actor-critic methods (A2C/A3C, PPO, SAC) optimize the policy directly and own continuous control alongside DDPG and TD3; and the discipline that separates a real result from a lucky seed, evaluation with confidence intervals, reproducibility, and knowing when deep RL is the wrong tool, is not an afterthought but the point.
Chapter Roadmap
- 25.1 Deep Q-Networks and Variants Q-learning with a network in place of the table: why bootstrapping off a moving network is unstable, the experience replay and target network that tamed it, and the variant family that each fixes one pathology, Double DQN for maximization bias, Dueling DQN for value-advantage separation, prioritized replay, and Rainbow as the combined baseline.
- 25.2 Policy Gradient Methods Optimizing the policy directly by ascending the gradient of expected return: the REINFORCE estimator and the policy-gradient theorem, why this approach is natural for continuous and high-dimensional action spaces, and the punishing variance of the raw gradient that the next section is built to reduce.
- 25.3 Actor-Critic Methods Cutting policy-gradient variance with a learned critic that scores the actor: advantage estimation and GAE, the A2C and A3C workhorses, proximal policy optimization (PPO) with its clipped objective that became the on-policy default, and soft actor-critic (SAC), the off-policy maximum-entropy method that dominates continuous control.
- 25.4 Continuous Control Reinforcement learning where actions are real-valued vectors and the max over actions is infeasible: the deterministic policy gradient and DDPG, the overestimation and instability fixes of TD3, and the benchmarks (locomotion, manipulation) on which continuous-control methods are measured.
- 25.5 Practical Deep RL The methodology that separates a result from a lucky run: why deep RL swings across seeds, hyperparameters, and implementation details, evaluation across many seeds with confidence intervals rather than a best curve, reproducibility as a first-class engineering problem, and honest guidance on when deep RL is worth it and when a simpler method wins.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single progression on a control task. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have trained the deep Q-network of Section 25.1, the PPO agent of Section 25.3, and a soft actor-critic continuous-control run from Section 25.4, then evaluated all of them across multiple seeds with confidence intervals exactly as Section 25.5 demands, so that "it works" becomes a number with an interval around it rather than a single lucky curve.
Hands-On Lab: Train and Trust a Deep RL Agent
Objective
Train three deep reinforcement-learning agents on standard control tasks and then, crucially, evaluate them the way a result deserves to be evaluated. You will first train the deep Q-network of Section 25.1 on a discrete-action control task, watching experience replay and the target network turn an otherwise unstable bootstrap into a rising learning curve. You will then train a proximal-policy-optimization agent from Section 25.3 on the same task and contrast the value-based and policy-gradient families on a level field. Next you will move to a continuous-control task and train a soft actor-critic agent from Section 25.4, where the action is a real-valued vector and the max-over-actions of DQN is no longer available. Finally, and this is the heart of the lab, you will rerun each agent across several random seeds and report mean performance with confidence intervals following Section 25.5, so that the deliverable is not a single triumphant curve but an honest statement of how widely the agent's outcomes disagree with themselves. The single thread is that in deep reinforcement learning the evaluation protocol is part of the result, and an agent you cannot reproduce across seeds is an agent you have not actually trained.
What You'll Practice
- Implementing a deep Q-network with experience replay and a target network, the stabilizers of Section 25.1.
- Training a PPO agent with a clipped objective and advantage estimation, following Section 25.2 and Section 25.3.
- Training a soft actor-critic agent on a continuous-control task, following Section 25.4.
- Running each agent across multiple seeds and reporting mean return with confidence intervals, the core discipline of Section 25.5.
- Reading a seed-spread as a measure of trust rather than a single best curve as a measure of success.
Setup
You need a Python environment with torch for the networks, gymnasium for the environments, and numpy for the seed aggregation. The discrete-action stages use a classic control task such as CartPole-v1, and the continuous-control stage uses a task with a real-valued action vector such as Pendulum-v1 or a locomotion environment. The one discipline that governs the whole lab is that every reported number must be aggregated over multiple seeds: a single training run tells you only that an outcome was possible once, not that it is reliable, so you will wrap each agent in a loop over seeds and carry the spread through to the final plot. The code below is the experience-replay buffer of Section 25.1, the small piece of bookkeeping that breaks the temporal correlation between consecutive transitions and makes the deep Q-network's bootstrap stable enough to train.
import numpy as np
from collections import deque
import random
class ReplayBuffer:
def __init__(self, capacity=100_000):
self.buffer = deque(maxlen=capacity) # old transitions drop out automatically
def push(self, state, action, reward, next_state, done):
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size):
batch = random.sample(self.buffer, batch_size) # decorrelate consecutive steps
s, a, r, s2, d = map(np.array, zip(*batch))
return s, a, r, s2, d
def __len__(self):
return len(self.buffer)
Steps
Step 1: Train a deep Q-network on a discrete-action task
Build a deep Q-network with the replay buffer above and a target network, train it on the discrete-action control task, and plot its learning curve, the value-based baseline of Section 25.1. Then ablate the stabilizers: remove the target network or the replay buffer and watch the curve become unstable or diverge, making concrete why the original DQN result depended on both.
Step 2: Train a PPO agent on the same task
Implement or configure proximal policy optimization with the clipped objective and advantage estimation of Section 25.3, train it on the same discrete-action task, and overlay its learning curve on the deep Q-network's. Note where the on-policy policy-gradient method and the off-policy value-based method differ in sample efficiency and stability on a level field.
Step 3: Train soft actor-critic on a continuous-control task
Switch to a continuous-control task whose action is a real-valued vector, where the max-over-actions of the deep Q-network is no longer available, and train the soft actor-critic agent of Section 25.4. Observe how the maximum-entropy objective keeps the policy exploring without an explicit epsilon schedule, and confirm it learns where a discrete-action method could not even be applied.
Step 4: Rerun every agent across multiple seeds
Wrap each of the three agents in a loop over at least five random seeds, holding all hyperparameters fixed, and collect the final performance of every run, the protocol of Section 25.5. Resist the temptation to report only the best seed: the spread across seeds is the result, not noise to be hidden.
Step 5: Report mean performance with confidence intervals
Aggregate the seed runs into a mean learning curve with a shaded confidence interval for each agent, following Section 25.5, and write the final comparison as means with intervals rather than single numbers. Discuss what the width of each interval says about how much you can trust the agent, and which comparisons between agents survive once the intervals overlap.
Expected Output
The lab produces three trained agents and one honest scoreboard. Steps 1 and 2 yield overlaid learning curves on the discrete-action task showing the deep Q-network and PPO both solving it, with the DQN ablations of Step 1 visibly destabilizing once a stabilizer is removed, the practical proof that experience replay and the target network are load-bearing rather than decorative. Step 3 shows soft actor-critic learning a continuous-control task that the value-based method cannot address at all, demonstrating why the policy-based and continuous-control families exist. The decisive output is the multi-seed aggregation of Steps 4 and 5: a plot in which each agent is a mean curve wrapped in a confidence band, and a final table reporting mean return plus or minus an interval rather than a single best number. The reader finishes able to implement and train agents from all three families, and, more importantly, able to read a deep reinforcement-learning result skeptically: to ask how many seeds, how wide the interval, and whether a claimed improvement survives the overlap, which is the difference between a method that works and a curve that worked once.
The from-scratch deep Q-network of Step 1 is worth building once so the bootstrap, the replay buffer, and the target network stop being abstractions, but you should rarely reimplement these agents in production. Stable-Baselines3 ships DQN, PPO, SAC, DDPG, and TD3 as tuned, tested implementations behind a uniform model.learn(total_timesteps) call, collapsing Steps 1 through 3 into a few lines each against the same gymnasium environment API, and CleanRL offers single-file reference implementations whose every line is visible when you need to understand exactly what a tuned agent does. For the evaluation half, the rliable library implements precisely the multi-seed protocol of Section 25.5: stratified bootstrap confidence intervals, interquartile mean, and performance profiles that replace the misleading single-seed bar chart. The discipline the chapter teaches still governs the libraries: a tuned agent run on three seeds and reported by its best is exactly as misleading in two lines as in two hundred. The library makes the agent free; it does not make the evaluation honest.
Stretch Goals
- Add Double DQN and Dueling DQN from Section 25.1 to the Step 1 agent and measure, across seeds, whether each variant's claimed improvement survives the confidence intervals of Section 25.5 on your task, the same skepticism the chapter applies to the literature.
- Sweep one sensitive hyperparameter (learning rate or entropy coefficient) for the soft actor-critic agent of Section 25.4 and plot performance against it with seed-spread bands, exposing how much of deep reinforcement learning's reported success is hyperparameter tuning rather than algorithmic advance.
- Take one agent that solved its task and evaluate it under a small distribution shift (a changed mass, a changed reward scale) without retraining, and discuss how the brittleness of Section 25.5 shows up not only across training seeds but across deployment conditions, the real-world face of the reproducibility problem.
What's Next?
This chapter scaled reinforcement learning from a table to a network, and with that scale came both the reach to control high-dimensional worlds and the brittleness that makes those results hard to trust. The deep methods here share an expensive assumption: a cheap simulator and a dense reward signal, so the agent can take the millions of online steps that experience replay and policy gradients devour. Chapter 26: Advanced Reinforcement Learning confronts the settings where that assumption breaks. It turns to learning from fixed logged data when fresh interaction is impossible or unsafe (offline reinforcement learning), to shaping behavior when the reward is sparse or must be inferred rather than given, to coordinating many agents at once, and to the model-based and hierarchical methods that squeeze far more out of each interaction than the model-free agents of this chapter. The value-based, policy-gradient, and actor-critic machinery you built here is the foundation every advanced method extends, and the evaluation discipline of Section 25.5 only matters more as the problems get harder. The full path through Part VI and the rest of the book is laid out in the Table of Contents.
Bibliography & Further Reading
Value-Based Deep RL
Mnih, V., et al. "Human-Level Control Through Deep Reinforcement Learning." Nature, 518, 529-533, 2015. arXiv:1312.5602. arxiv.org/abs/1312.5602
The deep Q-network paper that learned to play Atari from pixels, introducing the experience replay and target network that stabilized Q-learning with a neural function approximator, the foundation of Section 25.1.
van Hasselt, H., Guez, A., Silver, D. "Deep Reinforcement Learning with Double Q-learning." AAAI, 2016. arXiv:1509.06461. arxiv.org/abs/1509.06461
The paper that brought Double Q-learning to the deep setting, correcting the maximization bias that makes the deep Q-network overestimate action values, the first variant of Section 25.1.
Wang, Z., et al. "Dueling Network Architectures for Deep Reinforcement Learning." ICML, 2016. arXiv:1511.06581. arxiv.org/abs/1511.06581
The paper that split the Q-network into separate state-value and action-advantage streams, the Dueling DQN architecture of Section 25.1 that learns which states are valuable independent of the action taken.
Hessel, M., et al. "Rainbow: Combining Improvements in Deep Reinforcement Learning." AAAI, 2018. arXiv:1710.02298. arxiv.org/abs/1710.02298
The paper that combined six DQN improvements, including Double, Dueling, and prioritized replay, into one strong baseline and ablated each contribution, the integrated value-based agent of Section 25.1.
Policy Gradient and Actor-Critic
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. "Proximal Policy Optimization Algorithms." 2017. arXiv:1707.06347. arxiv.org/abs/1707.06347
The paper introducing PPO and its clipped surrogate objective, the on-policy actor-critic method that became the default for its simplicity and reliability, central to Section 25.3 and the lab.
Schulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P. "High-Dimensional Continuous Control Using Generalized Advantage Estimation." ICLR, 2016. arXiv:1506.02438. arxiv.org/abs/1506.02438
The paper introducing generalized advantage estimation (GAE), the bias-variance trade-off for advantage signals that underpins the variance reduction of the actor-critic methods in Section 25.3.
Haarnoja, T., Zhou, A., Abbeel, P., Levine, S. "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor." ICML, 2018. arXiv:1801.01290. arxiv.org/abs/1801.01290
The paper introducing soft actor-critic (SAC), the off-policy maximum-entropy method whose sample efficiency and stability made it the dominant continuous-control algorithm of Sections 25.3 and 25.4.
Continuous Control
Lillicrap, T. P., et al. "Continuous Control with Deep Reinforcement Learning." ICLR, 2016. arXiv:1509.02971. arxiv.org/abs/1509.02971
The paper introducing DDPG, the deterministic policy gradient that extended Q-learning into continuous action spaces by learning an actor alongside the critic, the starting point of Section 25.4.
Fujimoto, S., van Hoof, H., Meger, D. "Addressing Function Approximation Error in Actor-Critic Methods (TD3)." ICML, 2018. arXiv:1802.09477. arxiv.org/abs/1802.09477
The paper introducing TD3, which fixed DDPG's overestimation and instability with twin critics, delayed policy updates, and target smoothing, the robust continuous-control method of Section 25.4.
Reproducibility and Evaluation
Henderson, P., et al. "Deep Reinforcement Learning that Matters." AAAI, 2018. arXiv:1709.06560. arxiv.org/abs/1709.06560
The paper that documented how wildly deep reinforcement-learning results swing across seeds, hyperparameters, and implementations, the empirical case for the evaluation discipline of Section 25.5.
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., Bellemare, M. G. "Deep Reinforcement Learning at the Edge of the Statistical Precipice (rliable)." NeurIPS, 2021. arXiv:2108.13264. arxiv.org/abs/2108.13264
The paper introducing the rliable protocol, stratified bootstrap confidence intervals, interquartile mean, and performance profiles, the rigorous multi-seed reporting standard the lab adopts for Section 25.5.
Tools and Libraries
Raffin, A., et al. "Stable-Baselines3: Reliable Reinforcement Learning Implementations." Journal of Machine Learning Research, 22(268), 2021. jmlr.org/papers/v22/20-1364.html
The library of tuned, tested implementations of DQN, PPO, SAC, DDPG, and TD3 behind a uniform API, the off-the-shelf agents that collapse the training steps of the lab into a few lines each.
Huang, S., et al. "CleanRL: High-Quality Single-File Implementations of Deep Reinforcement Learning Algorithms." Journal of Machine Learning Research, 23(274), 2022. arXiv:2111.08819. arxiv.org/abs/2111.08819
The library of single-file reference implementations whose every line is visible, the readable counterpart to Stable-Baselines3 for understanding exactly what a tuned deep reinforcement-learning agent does.