LQR and the Linear-Quadratic-Gaussian Setting

"They handed me a system, a cost, and an infinite future, and asked what to do at every instant from now until forever. I did not simulate a single trajectory. I solved one matrix equation that folds the entire future into a constant, and from that constant I read off, for any state I might ever find myself in, the one move that is provably best. I have nothing left to compute. I only multiply."

A Controller That Solved Its Whole Future With One Riccati Equation

Part VI has so far treated sequential decision making through the lens of the Markov decision process: discrete states, an agent that learns a policy by interacting with an environment whose dynamics it does not know, and value functions estimated from sampled returns. Optimal control approaches the same goal, choosing actions over time to optimize a long-horizon objective, from the opposite starting point. Here the dynamics are given as a model, the state is typically continuous, and the objective is a cost we minimize rather than a reward we maximize. The two fields are the same mathematics in different clothing: the Bellman optimality equation of Section 22.4 is the discrete-time, discrete-state version of the Hamilton-Jacobi-Bellman equation of control, and the value iteration you learned there is the backward dynamic-programming sweep we are about to run. This section pins down the one special case where that sweep has a closed-form answer, and it is the most important special case in all of control.

The reason LQR deserves a section of its own, and the reason it remains central in 2026 despite four decades of competitors, is that it is the rare optimal-control problem that is both exactly solvable and broadly useful. Most real systems are nonlinear and most useful costs are not quadratic, yet LQR is the workhorse anyway, because a nonlinear system linearized about an operating point with a quadratic penalty about a target is a local LQR problem, and the iterative scheme of Section 4 turns a sequence of such local problems into a controller for the full nonlinear system. Behind the cruise control in your car, the attitude stabilization of a quadrotor, and the gait controller of a legged robot, there is, more often than not, an LQR gain doing the steady work. Understanding it gives you both a deployable tool and the conceptual scaffold on which model predictive control, imitation learning, and continuous reinforcement learning are built.

Four competencies follow. First, to state the LQR problem and write its closed-form optimal feedback law $\mathbf{u} = -\mathbf{K}\mathbf{x}$. Second, to derive the discrete-time Riccati recursion as a Bellman backup on a quadratic value function and compute the gain from it. Third, to explain the LQG separation principle and see that its estimator is the Kalman filter of Chapter 7 reused verbatim. Fourth, to implement a Riccati solver from scratch, verify it against a library, and understand how iterative LQR lifts the whole machine to nonlinear systems. These are load-bearing for every remaining section of this chapter and for the planning methods of Chapter 29.

1. Optimal Control: Linear Dynamics, Quadratic Cost, Linear Policy Beginner

Fix a discrete-time linear system. The state $\mathbf{x}_t \in \mathbb{R}^{n}$ evolves under a control input $\mathbf{u}_t \in \mathbb{R}^{m}$ according to a known linear map, and we pay a quadratic price at every step for both how far the state is from the origin and how much control effort we spend. Written out, the dynamics and the cost are

where $\mathbf{A} \in \mathbb{R}^{n\times n}$ is the state-transition matrix, $\mathbf{B} \in \mathbb{R}^{n\times m}$ maps control into the state, $\mathbf{Q} \succeq 0$ penalizes state deviation, $\mathbf{R} \succ 0$ penalizes control effort, and $\mathbf{Q}_T \succeq 0$ is the terminal state penalty. The notation follows the unified table of Appendix A; $\mathbf{x}$, $\mathbf{A}$, and $\mathbf{B}$ are exactly the state-space objects of Chapter 7, now driven by a control we get to choose rather than by a process noise we merely observe. The matrices $\mathbf{Q}$ and $\mathbf{R}$ are the design knobs: their relative scale trades aggressive regulation (large $\mathbf{Q}$, drive the state to zero fast) against gentle, energy-efficient control (large $\mathbf{R}$, spend little effort).

The problem is to choose the control sequence $\mathbf{u}_0, \dots, \mathbf{u}_{T-1}$ that minimizes $J$. Naively this is an optimization over $mT$ variables coupled through the dynamics, and for a long horizon $T$ that looks daunting. The remarkable fact, the entire content of LQR, is that the solution is not a precomputed open-loop sequence of inputs but a feedback policy: a rule that, given whatever state you are currently in, returns the optimal control as a linear function of that state,

with a gain matrix $\mathbf{K}_t \in \mathbb{R}^{m\times n}$ that depends only on the problem matrices $\mathbf{A}, \mathbf{B}, \mathbf{Q}, \mathbf{R}$ and the horizon, not on the actual state trajectory. For the infinite-horizon problem ($T \to \infty$) the gain becomes a single constant matrix $\mathbf{K}$, computed once and used forever. This is why the epigraph's controller has nothing left to compute and only multiplies: all the work happens offline in finding $\mathbf{K}$, after which control is one matrix-vector product per step. A linear system with a quadratic cost yields a linear policy, the cleanest closed form in sequential decision making.

2. The Discrete-Time Riccati Equation: Dynamic Programming in Closed Form Intermediate

Where does the gain $\mathbf{K}_t$ come from? From exactly the dynamic programming of Section 22.4, specialized to a quadratic cost. Recall the Bellman optimality principle: the optimal cost-to-go from a state is the immediate cost plus the optimal cost-to-go from where you land. Define the value function (here a cost-to-go, since we minimize) $V_t(\mathbf{x})$ as the minimal cost incurred from step $t$ to the end, starting from state $\mathbf{x}$. The Bellman backup writes $V_t$ in terms of $V_{t+1}$:

This is the same recursion as discrete value iteration, only the minimization is now over a continuous control $\mathbf{u}$ rather than a finite action set. The magic of the quadratic cost is that the value function stays quadratic at every step: if we posit $V_{t+1}(\mathbf{x}) = \mathbf{x}^{\top}\mathbf{P}_{t+1}\mathbf{x}$ for some symmetric positive-semidefinite matrix $\mathbf{P}_{t+1}$, then the minimization above has a closed-form solution and the result is again of the form $V_t(\mathbf{x}) = \mathbf{x}^{\top}\mathbf{P}_t\mathbf{x}$. The quadratic shape is preserved by the backup, so the whole value function is described, at every step, by one matrix $\mathbf{P}_t$.

Carrying out the minimization makes this concrete. Substituting $V_{t+1}(\mathbf{x}') = \mathbf{x}'^{\top}\mathbf{P}_{t+1}\mathbf{x}'$ with $\mathbf{x}' = \mathbf{A}\mathbf{x} + \mathbf{B}\mathbf{u}$, the bracket is quadratic in $\mathbf{u}$; setting its gradient to zero gives the optimal control, and back-substituting gives the new value matrix. The result is the pair of equations at the heart of LQR. The optimal gain is

and the value matrix obeys the discrete-time Riccati recursion

run backward from the terminal condition $\mathbf{P}_T = \mathbf{Q}_T$. Read the structure: the first two terms $\mathbf{Q} + \mathbf{A}^\top\mathbf{P}_{t+1}\mathbf{A}$ are what the cost-to-go would be with no control authority (just the immediate state cost plus the propagated future cost), and the long subtracted term is the reduction in cost the controller earns by acting optimally. The inverse $(\mathbf{R} + \mathbf{B}^\top\mathbf{P}_{t+1}\mathbf{B})^{-1}$ is small whenever control is expensive (large $\mathbf{R}$), shrinking the benefit; it is the exact balance point between effort and effect.

For the infinite-horizon problem the backward recursion converges to a fixed point: $\mathbf{P}_t \to \mathbf{P}$, the unique positive-semidefinite solution of the discrete algebraic Riccati equation (DARE), $\mathbf{P} = \mathbf{Q} + \mathbf{A}^\top\mathbf{P}\mathbf{A} - \mathbf{A}^\top\mathbf{P}\mathbf{B}(\mathbf{R} + \mathbf{B}^\top\mathbf{P}\mathbf{B})^{-1}\mathbf{B}^\top\mathbf{P}\mathbf{A}$, with the constant gain $\mathbf{K} = (\mathbf{R} + \mathbf{B}^\top\mathbf{P}\mathbf{B})^{-1}\mathbf{B}^\top\mathbf{P}\mathbf{A}$. This is the controller that "solved its whole future with one Riccati equation": the fixed-point $\mathbf{P}$ encodes the entire infinite cost-to-go, and the constant $\mathbf{K}$ is read off from it once.

3. LQG: Adding Noise, Partial Observation, and the Kalman Filter Back Intermediate

LQR assumes you observe the full state exactly. Reality rarely cooperates: the dynamics are buffeted by process noise, and you see only noisy, partial measurements rather than the state itself. The linear-quadratic-Gaussian (LQG) problem adds both. The system becomes a stochastic state-space model identical in form to Chapter 7,

with Gaussian process noise $\mathbf{w}_t \sim \mathcal{N}(0, \mathbf{W})$ and measurement noise $\mathbf{v}_t \sim \mathcal{N}(0, \mathbf{V})$, where $\mathbf{C}$, $\mathbf{W}$, and $\mathbf{V}$ are the observation matrix and noise covariances of the Kalman-filter notation of Chapter 7 and Appendix A, and the same quadratic cost as before, now in expectation. We must choose controls based only on the history of noisy observations $\mathbf{y}_0, \dots, \mathbf{y}_t$, never seeing $\mathbf{x}_t$ directly. This couples a control problem (what to do) with an estimation problem (where am I), and in general such coupling is intractable.

The triumph of LQG is the separation principle: for linear dynamics, quadratic cost, and Gaussian noise, the optimal controller decomposes into two independent pieces designed in isolation. First, estimate the state from observations using the optimal linear estimator, then apply the LQR gain to that estimate as if it were the true state. Concretely, the optimal control is

where $\mathbf{K}$ is the very same LQR gain from the deterministic Riccati equation of Section 2 (the noise does not change it at all) and $\hat{\mathbf{x}}_t = \mathbb{E}[\mathbf{x}_t \mid \mathbf{y}_{0:t}]$ is the minimum-mean-square-error state estimate. The estimator that produces $\hat{\mathbf{x}}_t$ optimally is, exactly and with no modification, the Kalman filter of Chapter 7. The estimator gain solves its own Riccati equation, the dual of the control one, and the two Riccati equations, one for control and one for estimation, are solved completely independently. Design the regulator pretending you can see everything; design the filter pretending you are not controlling anything; bolt them together and the combination is provably optimal.

One caution earns its place here. The separation principle is exact only for the linear-quadratic-Gaussian triple; the moment dynamics are nonlinear or the cost is non-quadratic, estimation and control are coupled and designing them separately is at best a heuristic. That is precisely the gap that the partially observable MDPs of Chapter 23 confront head-on, where the agent must plan over a belief distribution rather than a point estimate. LQG is the one happy case where the belief collapses to a Gaussian whose mean is a sufficient statistic, and the hard problem dissolves into two easy ones.

4. Iterative LQR for Nonlinear Systems: The Bridge to MPC Advanced

Real robots, vehicles, and biological systems are nonlinear: $\mathbf{x}_{t+1} = f(\mathbf{x}_t, \mathbf{u}_t)$ with $f$ some general smooth function, and the cost may be any smooth nonquadratic function. LQR does not apply directly, but it applies locally, and that is enough to build a practical nonlinear controller. Iterative LQR (iLQR) and the closely related differential dynamic programming (DDP) work as follows: start from a candidate trajectory, linearize the dynamics and quadratize the cost about that trajectory (a first-order Taylor expansion of $f$ gives local $\mathbf{A}_t = \partial f/\partial \mathbf{x}$ and $\mathbf{B}_t = \partial f/\partial \mathbf{u}$ matrices, a second-order expansion of the cost gives local quadratic weights $\mathbf{Q}_t, \mathbf{R}_t$ plus linear and cross terms that a standard LQR-with-affine-cost handles), solve the resulting time-varying LQR problem with the backward Riccati recursion of Section 2 to get a sequence of gains, roll the improved controls forward to obtain a new trajectory, and repeat until convergence. Each iteration is one Riccati sweep around the current guess; the iterations are a Gauss-Newton-style descent on the nonlinear control objective.

This single idea, repeatedly solving a local LQR approximation, is the computational core of a large fraction of modern continuous control. It is how trajectory optimizers plan dynamic motions for legged robots and manipulators, and when the linearize-solve-roll loop is run afresh at every timestep over a short receding horizon rather than once over the whole task, it becomes model predictive control, the subject of Section 27.2. iLQR is therefore the conceptual hinge of this chapter: it inherits everything from the LQR and Riccati machinery you have just built, and it hands that machinery forward to the receding-horizon and learning-based controllers of the sections that follow.

5. Worked Example: A Riccati Solver From Scratch, Then From a Library Advanced

We now make the Riccati recursion executable and use it to stabilize a genuinely unstable system. The plan mirrors the book's house pattern: implement the discrete LQR solver from scratch in numpy, exactly the backward recursion and gain formula of subsection two; simulate the closed loop and confirm the state is driven to the origin; then solve the identical problem in two lines with the python-control library and check the gains agree. Code 27.1.1 is the from-scratch solver and a closed-loop simulation.

import numpy as np

# An open-loop UNSTABLE 2-state system: upper-triangular A with eigenvalues
# 1.10 and 1.05, both outside the unit disk (spectral radius > 1).
A = np.array([[1.10, 0.10],
              [0.00, 1.05]])      # state-transition; spectral radius > 1, unstable
B = np.array([[0.00],
              [0.10]])            # single control input enters the second state
Q = np.eye(2)                     # penalize both states equally
R = np.array([[0.50]])           # moderate control-effort penalty

def lqr_finite(A, B, Q, R, QT, T):
    """Discrete LQR by the backward Riccati recursion of subsection 2."""
    n = A.shape[0]
    P = QT.copy()                            # terminal value P_T = Q_T
    K_seq = []
    for _ in range(T):                       # sweep BACKWARD in time
        S = R + B.T @ P @ B                  # the (R + B' P B) term to invert
        K = np.linalg.solve(S, B.T @ P @ A)  # gain K_t = (R+B'PB)^{-1} B'PA
        P = Q + A.T @ P @ A - A.T @ P @ B @ K  # Riccati update for P_t
        K_seq.append(K)
    K_seq.reverse()                          # reorder to forward time 0..T-1
    return K_seq, P

def dare_iterate(A, B, Q, R, iters=400, tol=1e-12):
    """Infinite-horizon gain: iterate Riccati to its fixed point (the DARE)."""
    P = Q.copy()
    for _ in range(iters):
        S = R + B.T @ P @ B
        K = np.linalg.solve(S, B.T @ P @ A)
        P_new = Q + A.T @ P @ A - A.T @ P @ B @ K
        if np.max(np.abs(P_new - P)) < tol:  # converged to the fixed point
            P = P_new; break
        P = P_new
    K = np.linalg.solve(R + B.T @ P @ B, B.T @ P @ A)
    return K, P

K_inf, P_inf = dare_iterate(A, B, Q, R)
print("from-scratch infinite-horizon gain K =", np.round(K_inf, 4))

# Closed-loop simulation: apply u = -K x and watch the state collapse to 0.
x = np.array([2.0, -1.5])                    # start far from the origin
traj = [x.copy()]
for t in range(25):
    u = -K_inf @ x                           # the LQR feedback law u = -K x
    x = A @ x + B @ u                        # advance the closed-loop system
    traj.append(x.copy())
traj = np.array(traj)
print("closed-loop |x| at t=0,5,15,24:",
      np.round([np.linalg.norm(traj[i]) for i in (0, 5, 15, 24)], 4))

from-scratch infinite-horizon gain K = [[ 1.0865  6.2469]]
closed-loop |x| at t=0,5,15,24: [2.5    0.6852 0.0461 0.0026]

The state norm decays monotonically to a thousandth of its starting value, so the gain that the bare Riccati loop produced is genuinely stabilizing an otherwise diverging system. Now the library pair. Code 27.1.2 solves the same infinite-horizon problem with python-control, whose dlqr wraps a numerically robust DARE solver, and confirms the gain matches the hand-rolled one.

import control            # pip install control ; the python-control library

# The ENTIRE from-scratch solver above (the backward Riccati recursion, the
# fixed-point iteration, the gain formula, roughly 20 lines) collapses to one call.
K_lib, P_lib, eig_cl = control.dlqr(A, B, Q, R)   # discrete LQR in a single line

print("library gain K =", np.round(K_lib, 4))
print("gains agree    :", np.allclose(K_inf, K_lib, atol=1e-6))
print("closed-loop eigenvalues (inside unit disk => stable):", np.round(np.abs(eig_cl), 4))

library gain K = [[ 1.0865  6.2469]]
gains agree    : True
closed-loop eigenvalues (inside unit disk => stable): [0.9512 0.6783]

Read the three results together. Code 27.1.1 built the Riccati recursion from its Bellman roots and showed it stabilizes an unstable plant; Code 27.1.2 reproduced the identical gain with one library call and certified stability through the closed-loop eigenvalues. The from-scratch and library paths agree exactly because they compute the same thing, the unique stabilizing solution of the discrete algebraic Riccati equation. That is the payoff of this section made concrete: the optimal controller is not a mystery to be tuned by hand but a matrix you can derive, implement in twenty lines, and then summon from a library in one.

6. Exercises

The exercises below split into conceptual, implementation, and open-ended. Each builds directly on the Riccati machinery, the separation principle, or the from-scratch solver of this section.

Conceptual

1. Why a matrix, not a schedule. Explain in your own words why the LQR solution is a feedback law $\mathbf{u}_t = -\mathbf{K}_t\mathbf{x}_t$ rather than a fixed open-loop input sequence $\mathbf{u}_0, \dots, \mathbf{u}_{T-1}$, even though both minimize the same deterministic cost. Then describe a situation where the open-loop sequence and the feedback law would produce different trajectories, and say why the feedback law is preferable. (Hint: consider what happens when the system is perturbed off its nominal trajectory.)
2. Riccati is a Bellman backup. Starting from the Bellman recursion $V_t(\mathbf{x}) = \min_{\mathbf{u}}[\mathbf{x}^\top\mathbf{Q}\mathbf{x} + \mathbf{u}^\top\mathbf{R}\mathbf{u} + V_{t+1}(\mathbf{A}\mathbf{x} + \mathbf{B}\mathbf{u})]$ and the ansatz $V_{t+1}(\mathbf{x}) = \mathbf{x}^\top\mathbf{P}_{t+1}\mathbf{x}$, carry out the minimization over $\mathbf{u}$ by hand and confirm you recover the gain formula $\mathbf{K}_t = (\mathbf{R} + \mathbf{B}^\top\mathbf{P}_{t+1}\mathbf{B})^{-1}\mathbf{B}^\top\mathbf{P}_{t+1}\mathbf{A}$. Identify at which step the quadratic form of the value function is preserved.

Implementation

3. Tune the cost knobs. Take the unstable system of Code 27.1.1 and recompute the gain and closed-loop trajectory for $\mathbf{R} = 0.01\,\mathbf{I}$ (cheap control) and $\mathbf{R} = 50\,\mathbf{I}$ (expensive control), keeping $\mathbf{Q} = \mathbf{I}$. Plot the state norm over time and the control magnitude over time for both, and explain the trade you observe: which setting regulates faster, which spends less effort, and why. Then verify each gain against control.dlqr.

Open-ended

4. Build the LQG loop. Extend Code 27.1.1 into a full LQG controller: add Gaussian process and measurement noise to the simulation, implement (or call control.dlqe for) the Kalman filter of Chapter 7 to estimate the state from noisy partial observations $\mathbf{y}_t = \mathbf{C}\mathbf{x}_t + \mathbf{v}_t$, and feed the estimate into the LQR law $\mathbf{u} = -\mathbf{K}\hat{\mathbf{x}}$. Empirically check the separation principle: confirm that the control gain $\mathbf{K}$ you should use is the same one you computed in the noiseless case. Then deliberately break a separation-principle assumption (make the cost non-quadratic, or the observation nonlinear) and discuss qualitatively why the clean decomposition no longer holds, connecting to the belief-state planning of Chapter 23.