Continuous Control

"Do not hand me a menu of three buttons labeled left, right, and brake. I am a valve. I open by a real number between zero and one, and if you make me round that number to the nearest button you will overshoot the setpoint, oscillate, and blame the controller. Give me the torque. Give me the actual torque."

An Actuator Demanding a Real Number, Not a Discrete Choice

In Section 25.3 we built policy-gradient and actor-critic methods that learn a parameterized policy directly, and we noted in passing that this is what makes continuous actions tractable. This section cashes that promise. Where Section 25.1 learned a value function and read a discrete policy off its argmax, here the policy is the primary object and the value function (the critic) exists only to teach it. The shift is forced by the actuators of the running industrial and finance domains of Part VI: torques, prices, doses, and flows are real numbers, not menu items, and an agent that can only choose from a fixed discrete list either cannot represent the right action at all or must discretize so finely that the action set explodes. We use the unified notation of Appendix A throughout: $\mathbf{s}$ the state, $\mathbf{a}$ the action, $r$ the reward, $\pi$ the policy, $Q$ the action-value, $\gamma$ the discount.

The four competencies this section installs are these: to recognize why the discrete-action machinery of DQN cannot be applied to a continuous actuator and to write down a Gaussian policy that can; to place DDPG, TD3, and SAC in one family and explain what each fixes about the previous one, especially the overestimation bias that TD3's twin critics and SAC's entropy regularization address; to apply the practical stabilizers (tanh action squashing with its log-probability correction, reward scaling, observation normalization) that turn a finicky algorithm into one that actually converges; and to implement a continuous-control agent from scratch and then reproduce it with a production library. These are the load-bearing skills for the practical deep-RL discipline of Section 25.5 and the advanced methods of Chapter 26, the imitation and control of Chapter 27, and the robotics applications of Chapter 35.

1. Continuous Action Spaces: Why DQN Does Not Apply Beginner

A discrete-action agent enumerates. Deep Q-learning, the workhorse of Section 25.1, learns a function $Q(\mathbf{s}, a)$ that scores each of a finite set of actions, and it acts by taking the highest-scoring one, $a^\star = \arg\max_a Q(\mathbf{s}, a)$. The greedy step and the Bellman target $r + \gamma \max_{a'} Q(\mathbf{s}', a')$ both rest on the same operation: a maximization over the action set, computed by simply trying every action. That is cheap when the set has four members and a single forward pass of the network produces all four values at once. It is the entire reason DQN is practical.

Now replace the four buttons with a valve. The action is a torque $\mathbf{a} \in [\mathbf{a}_{\min}, \mathbf{a}_{\max}] \subset \mathbb{R}^k$, a real vector with infinitely many values. The operation $\max_a Q(\mathbf{s}, a)$ is now a continuous optimization over a $k$-dimensional box, and it must be solved afresh at every timestep of acting and at every entry of every minibatch during training. There is no enumeration; for a general neural-network $Q$ there is no closed form for the maximizer; and running an inner optimizer (gradient ascent on $a$) inside every Bellman backup is both expensive and unstable. The argmax that made DQN work is exactly what makes it inapplicable here. This is not a tuning difficulty, it is a structural wall: the discrete-action algorithm has no tractable continuous counterpart by direct extension.

The naive escape, discretize the interval into bins and run DQN on the bins, fails on dimension. One joint split into 10 bins is fine; a humanoid with 17 joints split into 10 bins each has $10^{17}$ joint actions, a set no network can score and no buffer can cover. Discretization also throws away the metric structure of the action space (that a torque of 0.49 is nearly identical to 0.50), forcing the agent to relearn from scratch what a single continuous output would express for free. The lesson is that for actuators we should not score actions at all; we should emit them.

The fix is to make the policy a function that outputs an action directly. Two flavors recur throughout this section. A deterministic policy is a map $\mathbf{a} = \boldsymbol{\mu}_{\boldsymbol{\theta}}(\mathbf{s})$ that returns one action per state; DDPG and TD3 use this form. A stochastic Gaussian policy outputs the parameters of a distribution and samples from it,

where the network emits a mean $\boldsymbol{\mu}_{\boldsymbol{\theta}}(\mathbf{s})$ and a (state-dependent or state-independent) standard deviation $\boldsymbol{\sigma}_{\boldsymbol{\theta}}(\mathbf{s})$, and the action is drawn from the resulting normal; SAC and the policy-gradient methods of Section 25.3 use this form. For a diagonal Gaussian over a $k$-dimensional action the log-probability of a sampled action, the quantity the policy gradient needs, is the closed form

The Gaussian buys two things at once. It produces continuous actions by construction (a sample from a normal is a real number, no argmax required), and its built-in randomness is the exploration, replacing the $\epsilon$-greedy coin flip of DQN with a principled spread $\boldsymbol{\sigma}$ that the agent can widen when uncertain and shrink as it commits. The deterministic policy gives up that built-in exploration and must inject noise by hand, a difference we return to in subsection two.

2. The Continuous-Control Algorithms: DDPG, TD3, and SAC Intermediate

The modern continuous-control toolbox is a short lineage of three off-policy actor-critic algorithms, each fixing a concrete defect of the previous one. Reading them as a family, rather than as three unrelated names, is the fastest way to understand the design space and to choose among them.

DDPG (Deep Deterministic Policy Gradient). DDPG is the continuous analogue of DQN: it keeps a replay buffer and target networks (the off-policy stability tricks of Section 25.1) but replaces the intractable $\max_a Q$ with a learned deterministic actor $\boldsymbol{\mu}_{\boldsymbol{\theta}}(\mathbf{s})$ that approximates the maximizer. The critic learns $Q_{\boldsymbol{\phi}}(\mathbf{s},\mathbf{a})$ by minimizing the Bellman error toward the target $y = r + \gamma\, Q_{\boldsymbol{\phi}'}\big(\mathbf{s}', \boldsymbol{\mu}_{\boldsymbol{\theta}'}(\mathbf{s}')\big)$, and the actor is trained to output actions the critic scores highly, by ascending the deterministic policy gradient

which is the chain rule pushing the critic's "which way is better" signal back through the actor's parameters. Because the actor is deterministic, exploration is added by hand: a noise process (Gaussian, or the temporally correlated Ornstein-Uhlenbeck noise of the original paper) is added to the action during data collection. DDPG works, but it is notoriously brittle: it inherits and amplifies the overestimation bias of Q-learning, because the critic's approximation errors are positively biased by the very maximization the actor performs, and the actor then chases those phantom high values into collapse.

TD3 (Twin Delayed DDPG). TD3 is DDPG plus three targeted fixes for that overestimation, and it is the reliable workhorse of deterministic continuous control. (1) Twin critics: learn two independent critics $Q_{\boldsymbol{\phi}_1}, Q_{\boldsymbol{\phi}_2}$ and use the minimum of the two in the Bellman target, $y = r + \gamma \min_{i=1,2} Q_{\boldsymbol{\phi}'_i}(\mathbf{s}', \tilde{\mathbf{a}})$; taking the smaller estimate deliberately counteracts the upward bias, the same clipped-double-Q idea that double DQN applied to discrete actions. (2) Delayed policy updates: update the actor (and the target networks) less often than the critics, typically once every two critic steps, so the actor chases a value estimate that has had time to settle rather than a noisy moving target. (3) Target policy smoothing: add a small clipped Gaussian noise to the target action $\tilde{\mathbf{a}} = \boldsymbol{\mu}_{\boldsymbol{\theta}'}(\mathbf{s}') + \operatorname{clip}(\boldsymbol{\epsilon}, -c, c)$ when computing the target, which regularizes the critic so it cannot exploit sharp spurious peaks in its own value surface. These three changes turn DDPG's brittleness into TD3's reliability at almost no extra conceptual cost.

SAC (Soft Actor-Critic). SAC takes the stochastic route. It is an off-policy actor-critic over a Gaussian policy that maximizes a maximum-entropy objective, reward plus a bonus for keeping the policy uncertain,

where $\mathcal{H}$ is the policy entropy and the temperature $\alpha$ trades reward against exploration. The entropy term is not a heuristic add-on; it makes the policy explore broadly and robustly, prevents premature collapse onto a narrow deterministic action, and (with the automatic-temperature variant that tunes $\alpha$ to hit a target entropy) makes SAC remarkably insensitive to hyperparameters. SAC also borrows TD3's twin critics for the same overestimation fix. Section 25.3 derived the full SAC algorithm; here we place it within the continuous-control family (revisited in the Looking Back box below).

The deterministic-versus-stochastic choice is the axis that organizes the family. A deterministic policy (DDPG, TD3) outputs exactly one action per state and explores only through externally injected noise; it tends to be sample-efficient and is natural when the optimal policy really is a single best action. A stochastic policy (SAC) outputs a distribution and explores through its own variance; it is more robust to hyperparameters, handles multimodal or genuinely stochastic-optimal tasks better, and degrades gracefully because it never bets everything on one action. The practical default in 2026 is SAC for most continuous benchmarks (robust, strong, little tuning) and TD3 when sample efficiency or a clean deterministic policy matters; plain DDPG is mostly of historical and pedagogical interest now, the ancestor you understand in order to understand its successors. Figure 25.4.1 lays the family out side by side.

3. Practical Realities: Squashing, Scaling, and Why It Is Finicky Intermediate

Continuous control has a reputation for being finicky, and the reputation is earned. The algorithms of subsection two are correct, but a faithful implementation that ignores three mundane practicalities, action bounds, reward magnitude, and observation scale, will often fail to learn while emitting no error at all. These are not optional polish; they are part of the algorithm.

Action bounds and tanh squashing. A Gaussian policy outputs an unbounded sample in $\mathbb{R}^k$, but every real actuator has limits: a torque saturates, a valve cannot open past fully open, a dose has a ceiling. The standard fix is to pass the raw Gaussian sample through a $\tanh$ and rescale to the action range, $\mathbf{a} = \tanh(\mathbf{u})$ for $\mathbf{u} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma}^2)$, which maps any real vector into $(-1, 1)^k$ smoothly. The catch is that squashing changes the distribution, so the log-probability used by the policy gradient must be corrected by the change-of-variables Jacobian of the $\tanh$:

where the subtracted term accounts for how the $\tanh$ compresses probability mass near the bounds. Omitting this correction is one of the most common silent bugs in a hand-rolled SAC: the agent trains, the loss looks plausible, and the policy quietly learns the wrong thing because its log-probabilities are inconsistent with the actions it actually took. We implement the correction explicitly in subsection five.

Reward scaling. The critic regresses onto returns, and a neural network regressor is sensitive to the magnitude of its targets. A task whose per-step rewards are in the thousands produces value targets in the tens of thousands, which blow up the critic's gradients; a task with rewards near $10^{-3}$ produces a signal the optimizer can barely see. Scaling rewards (or normalizing returns) to a sane range, often $O(1)$, is frequently the difference between learning and not, and it is why benchmark wrappers ship reward-scaling options. The temperature $\alpha$ in SAC interacts with reward scale directly, which is one more reason the automatic-temperature variant, which adapts to the scale, is so much easier to use.

Observation normalization. Symmetrically, the policy and critic ingest observations whose components may live on wildly different scales (a joint angle in radians near 1, a cart position in meters near 0.01, a velocity in the hundreds). Feeding raw heterogeneous features to a network slows or prevents learning. The standard remedy is a running normalizer that maintains an online mean and variance of each observation component and standardizes inputs to zero mean and unit variance, exactly the streaming standardization an online learner uses in Chapter 20. Together these three stabilizers explain most of the gap between "I implemented the equations" and "it actually learns".

4. Benchmarks and Sim-to-Real Advanced

Continuous-control research is organized around a small set of standard benchmarks, and knowing them is how you read the literature and calibrate your own results. The dominant suite is the MuJoCo continuous-control tasks (HalfCheetah, Hopper, Walker2d, Ant, Humanoid, and friends), a collection of simulated articulated bodies whose goal is locomotion: emit joint torques each step to make the body run forward as fast as possible without falling. They are exposed through the Gymnasium API (the maintained successor to OpenAI Gym), which since 2022 ships MuJoCo as a permissively licensed, pip-installable backend, alongside the simpler classic-control tasks like Pendulum and the continuous version of MountainCar that we use for the worked example because they train in seconds on a laptop. A second important suite is DeepMind Control (dm_control), with finer-grained tasks and a standardized reward structure. Reporting on these benchmarks follows the reproducibility discipline of Section 25.5: report the mean and standard deviation of final return over many seeds, never a single lucky run, because continuous-control variance across seeds is large.

The benchmarks are simulators, and that is both their strength and their central limitation. A simulator gives unlimited, cheap, perfectly labeled, safely resettable experience, which is why almost all continuous-control progress is measured in simulation. But the policies must eventually drive real actuators, and the sim-to-real gap, the mismatch between simulated and real dynamics, friction, latency, sensor noise, and unmodeled effects, routinely turns a policy that is superb in simulation into one that flails on hardware. The gap is the defining obstacle of applied continuous control and the subject of Chapter 35's robotics treatment. The standard mitigations are domain randomization (train across a distribution of simulator parameters so the policy is robust to the real value it will meet), system identification (fit the simulator to the real robot), and conservative sim-to-real fine-tuning with a small amount of real, often expensive, on-hardware data. The temporal thread is direct: a sim-trained policy faces a distribution shift the moment it is deployed, the same non-stationarity and drift problem of Chapter 20, now with a physical robot on the other end.

5. Worked Example: A Gaussian Actor-Critic, From Scratch and by Library Advanced

We now make the section executable on Pendulum-v1, the canonical small continuous-control task: swing a frictionless pendulum upright by applying a continuous torque in $[-2, 2]$. The plan mirrors the rest of the book: build the policy and a one-step actor-critic update from scratch in PyTorch, exposing the Gaussian sampling, the tanh squash, and the log-probability correction explicitly; then collect a short trajectory and verify the update runs and the squashing keeps every action in bounds; then replace the whole thing with a few lines of Stable-Baselines3 and state the line-count reduction. Code 25.4.1 is the from-scratch Gaussian policy with squashing.

import torch
import torch.nn as nn
import gymnasium as gym

ACT_LOW, ACT_HIGH = -2.0, 2.0          # Pendulum torque bounds
LOG_STD_MIN, LOG_STD_MAX = -5.0, 2.0   # clamp the learned log-std for stability

class GaussianActor(nn.Module):
    """A squashed-Gaussian policy: outputs mean and log-std, samples, tanh-squashes."""
    def __init__(self, obs_dim, act_dim, hidden=64):
        super().__init__()
        self.body = nn.Sequential(nn.Linear(obs_dim, hidden), nn.Tanh(),
                                  nn.Linear(hidden, hidden), nn.Tanh())
        self.mu_head = nn.Linear(hidden, act_dim)        # mean of the raw Gaussian
        self.log_std_head = nn.Linear(hidden, act_dim)   # log-std of the raw Gaussian

    def forward(self, obs):
        h = self.body(obs)
        mu = self.mu_head(h)
        log_std = self.log_std_head(h).clamp(LOG_STD_MIN, LOG_STD_MAX)
        return mu, log_std

    def sample(self, obs):
        mu, log_std = self.forward(obs)
        std = log_std.exp()
        normal = torch.distributions.Normal(mu, std)
        u = normal.rsample()                 # reparameterized raw sample (mu + std * xi)
        a_tanh = torch.tanh(u)               # SQUASH into (-1, 1)
        # log-prob of the SQUASHED action: subtract the tanh change-of-variables Jacobian
        logp = normal.log_prob(u).sum(-1)
        logp = logp - torch.log(1 - a_tanh.pow(2) + 1e-6).sum(-1)
        # rescale (-1,1) -> actuator range [ACT_LOW, ACT_HIGH]
        action = ACT_LOW + 0.5 * (a_tanh + 1.0) * (ACT_HIGH - ACT_LOW)
        return action, logp

class Critic(nn.Module):
    """A state-value baseline V(s) for the actor-critic advantage."""
    def __init__(self, obs_dim, hidden=64):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(obs_dim, hidden), nn.Tanh(),
                                 nn.Linear(hidden, hidden), nn.Tanh(),
                                 nn.Linear(hidden, 1))
    def forward(self, obs):
        return self.net(obs).squeeze(-1)

Code 25.4.2 wires these into one actor-critic update step: roll out the policy, bootstrap a one-step return, compute the advantage from the critic, and take a gradient step on both the policy loss (negative advantage-weighted log-probability) and the critic loss (return regression). This is the smallest complete continuous-control learner, the stripped-down cousin of SAC promised in Callback 25.3.

def ac_update(actor, critic, opt_a, opt_c, obs, action_logp, reward, next_obs, done, gamma=0.99):
    """One on-policy actor-critic update on a single transition (Pendulum)."""
    logp = action_logp                                   # log pi(a|s) from sample()
    with torch.no_grad():
        target = reward + gamma * (1 - done) * critic(next_obs)   # one-step bootstrap
    value = critic(obs)
    advantage = (target - value).detach()                # critic says how good the action was
    actor_loss = -(advantage * logp).mean()              # policy gradient: push up good actions
    critic_loss = (value - target).pow(2).mean()         # regress V toward the return
    opt_a.zero_grad(); actor_loss.backward(); opt_a.step()
    opt_c.zero_grad(); critic_loss.backward(); opt_c.step()
    return actor_loss.item(), critic_loss.item()

env = gym.make("Pendulum-v1")
obs_dim = env.observation_space.shape[0]    # 3 for Pendulum
act_dim = env.action_space.shape[0]         # 1 for Pendulum
actor, critic = GaussianActor(obs_dim, act_dim), Critic(obs_dim)
opt_a = torch.optim.Adam(actor.parameters(), lr=3e-4)
opt_c = torch.optim.Adam(critic.parameters(), lr=3e-4)

obs, _ = env.reset(seed=0)
obs_t = torch.tensor(obs, dtype=torch.float32)
action, logp = actor.sample(obs_t.unsqueeze(0))
a_np = action.squeeze(0).detach().numpy()
next_obs, reward, term, trunc, _ = env.step(a_np)
next_t = torch.tensor(next_obs, dtype=torch.float32)
al, cl = ac_update(actor, critic, opt_a, opt_c,
                   obs_t.unsqueeze(0), logp,
                   torch.tensor([reward], dtype=torch.float32),
                   next_t.unsqueeze(0), torch.tensor([float(term)]))
print("sampled torque   = %.4f  (in bounds: %s)" % (a_np[0], ACT_LOW <= a_np[0] <= ACT_HIGH))
print("actor loss = %.4f   critic loss = %.4f" % (al, cl))

sampled torque   = 0.8137  (in bounds: True)
actor loss = -0.0042   critic loss = 18.7351

Now the library pair. Code 25.4.3 trains a full, robust SAC agent on the same task with Stable-Baselines3, which internally implements the squashed-Gaussian policy, the twin critics, the automatic temperature, the replay buffer, and the target networks of subsection two. The squashing, log-probability correction, and reward handling we wrote by hand are all inside the library.

import gymnasium as gym
from stable_baselines3 import SAC

env = gym.make("Pendulum-v1")
model = SAC("MlpPolicy", env, verbose=0, seed=0)   # squashed-Gaussian SAC, all defaults
model.learn(total_timesteps=20_000)                # off-policy training with replay + twin critics

obs, _ = env.reset(seed=0)
action, _ = model.predict(obs, deterministic=True) # greedy action for evaluation
print("SB3 SAC action =", action)

SB3 SAC action = [-0.7421]

Read the three blocks together. Code 25.4.1 exposed the squashed-Gaussian policy and the one log-probability correction that is the section's most error-prone line. Code 25.4.2 drove it with a complete actor-critic update, showing the critic's advantage steering the policy mean and the squash keeping every torque in bounds. Code 25.4.3 replaced all of it with a production SAC that adds the twin critics, automatic temperature, and replay machinery of subsection two for free. The payoff is the same as everywhere in this book: you understand the algorithm because you built its core by hand, and you ship the robust version because a library implemented the parts that are tedious to get exactly right.

from stable_baselines3 import SAC, TD3
import gymnasium as gym

env = gym.make("HalfCheetah-v5")              # a MuJoCo continuous-control benchmark
model = TD3("MlpPolicy", env, verbose=0)      # swap TD3 -> SAC for the stochastic agent
model.learn(total_timesteps=1_000_000)        # full off-policy training, all machinery internal

Exercises

1. Conceptual: Why the Argmax Breaks. Explain precisely why the deep Q-network update of Section 25.1 cannot be applied directly to a continuous action space. Identify the two specific operations in the DQN algorithm that require a maximization over actions, and explain why discretizing the action interval into bins is not a satisfactory fix for a high-dimensional actuator (give the scaling argument explicitly for a 12-joint robot with 10 bins per joint). Then state, in one sentence each, how a deterministic policy and a Gaussian policy each sidestep the problem.
2. Implementation: The Squash Correction Matters. Extend Code 25.4.1 and 25.4.2 into a full training loop on Pendulum-v1 and run it twice: once with the tanh log-probability correction line included, and once with that single line deleted. Plot the episodic return of both runs over training. Confirm that the version without the correction learns a measurably worse or unstable policy, and explain, referring to the change-of-variables formula of subsection three, why the missing Jacobian term corrupts the policy gradient. Report the final mean return over five seeds for each variant.
3. Open-ended: TD3 Versus SAC on Your Actuator. Pick a continuous-control task with a real actuator interpretation (a MuJoCo locomotion task, the continuous LunarLander, or a simulated valve or pricing problem of your own). Train both TD3 and SAC from Stable-Baselines3 with default hyperparameters over at least five seeds each, and compare them on three axes: final return, sample efficiency (return at a fixed early budget), and seed-to-seed variance. Then deliberately degrade the setup, scale the reward by 1000 without re-tuning, and report which algorithm is more robust to the reward-scale change. Connect your finding to the automatic-temperature argument of subsection two and the reproducibility discipline of Section 25.5.