Temporal Abstraction and Long-Horizon Planning

"Ask me to cross the continent and I will not enumerate every footstep; I will say fly to Denver, then drive to the cabin, then walk the last mile. I plan in chapters and let my legs handle the footnotes. The agent that insists on reasoning one step at a time is still standing in the driveway, having computed a beautiful tree of the first nine hundred possible footsteps, none of which has left the house."

A Planner That Thinks in Chapters, Not in Footsteps

In Section 29.4 we planned inside a learned world model, rolling the model forward a fixed horizon and optimizing an action sequence by trajectory optimization or sampling. That worked because the horizon was short enough that the model stayed honest and the search stayed tractable. This section confronts what happens when the horizon is long: hundreds or thousands of primitive steps between the start and the goal. Both the model and the search, perfectly adequate over a short horizon, degrade in ways that no amount of tuning fixes, and the fix is structural rather than numerical. We use the unified notation of Appendix A throughout: $s_t$ the state, $a_t$ the primitive action, $\hat{T}$ the learned transition model, $\pi$ a policy, and $\gamma$ the discount.

The thread we pick up is the one Part VI has been pulling since the start. Chapter 22 defined the MDP and value iteration over primitive actions; Section 26.5 introduced hierarchical reinforcement learning and the options framework, where an agent acts at two time scales; Section 29.1 measured how world-model error compounds over a rollout. Temporal abstraction in planning is exactly the marriage of those three: take the compounding-error problem of 29.1, take the exponential-search problem inherent in any flat MDP, and dissolve both with the multi-step options of 26.5, now used not only for acting but for planning and imagining. The Kalman-to-RNN, MDP-to-sequence-model arcs of this book all converge here, on the claim that the structure you build into a temporal model is what lets it reach far.

The four competencies this section installs: to state precisely why long horizons break flat planning and single-step world models, separating the compounding-error failure from the branching-factor failure; to write an abstract transition that advances the world by $k$ primitive steps in one coarse decision and to see how it shrinks both the rollout length and the search depth; to place the recent learned-skill and goal-directed latent planners (Director, Dreamer-class long-horizon agents) on that map; and, closing Part VI, to choose among the whole spectrum of decision-making methods by asking how much structure the problem rewards. These are the skills that turn the eight chapters of Part VI into a single usable judgment.

1. The Long-Horizon Problem: Compounding Error and Exploding Search Intermediate

A model-based planner does two things: it imagines futures with a learned model, and it searches over actions to find a good one. Over a short horizon both are cheap and reliable. Over a long horizon both fail, and they fail for independent reasons, so it is worth pulling the two failures apart before reaching for a cure that addresses both.

The first failure is compounding model error, the subject of Section 29.1, now seen as the binding constraint on horizon. A learned transition $\hat{T}$ has some one-step error; when you roll it forward to imagine a trajectory, each step feeds the previous step's slightly-wrong output back in as input, so errors accumulate and, worse, drift the rollout off the data distribution the model was trained on, where its error is larger still. If the per-step prediction error grows by a factor $\rho > 1$ as the rollout leaves the training distribution, the error at horizon $H$ scales roughly as $\epsilon_H \approx \epsilon_1 \cdot \frac{\rho^{H} - 1}{\rho - 1}$, a sum that is benign for small $H$ and ruinous for large $H$. The practical consequence is a horizon wall: beyond some number of imagined steps the model's rollout is fiction, and optimizing an action sequence against fiction optimizes nothing. You cannot tune your way past this; a model that is excellent at one step is still useless at a thousand if $\rho > 1$.

The second failure is combinatorial explosion of flat search. Searching over sequences of primitive actions, with branching factor $b$ (the number of actions worth considering per step) to depth $H$, examines on the order of $b^{H}$ trajectories. Even modest numbers are hopeless: $b = 10$ actions and a horizon of $H = 100$ steps is $10^{100}$ candidate plans, more than the atoms in the observable universe. Sampling-based planners (the CEM and MPPI methods of Section 29.4) dodge the full enumeration by sampling a few thousand trajectories rather than all $b^H$, but sampling thousands of length-$H$ rollouts only relocates the cost into the first failure: each of those rollouts is a long imagination that the compounding-error wall has already corrupted. The two failures compound each other, which is why naive deep model-based planning has a horizon ceiling of a few tens of steps in practice.

The diagnosis tells you the shape of the cure. Both walls are functions of the number of decision points between start and goal: the rollout length that compounds error is that count, and the search depth that branches is that count. Anything that reduces the number of decision points without changing where you can end up attacks both walls at once. That is exactly what temporal abstraction does: it replaces a long chain of primitive decisions with a short chain of coarse ones, each coarse decision being a multi-step behavior. Figure 29.5.1 contrasts the flat and abstract decision chains.

It is worth saying plainly what abstraction is not doing. It is not making the world shorter; the agent still takes the same thousand primitive steps in the environment. It is making the plan shorter, by changing the unit of decision from a primitive action to a temporally extended behavior. The primitive steps still happen, but they happen inside an option, executed by a low-level controller that the high-level planner does not reason about step by step. The planner reasons in chapters; the low-level policy writes the footnotes. This division of labor is the entire content of the cure, and the rest of the section makes it precise and then executable.

2. Temporal Abstraction as the Cure: Options, Subgoals, and the Jumpy Model Intermediate

The formal device is the option, introduced for acting in Section 26.5 and now used for planning and imagining. An option $o = (I_o, \pi_o, \beta_o)$ is a temporally extended behavior: an initiation set $I_o$ of states where it can start, an internal policy $\pi_o$ that selects primitive actions while the option runs, and a termination condition $\beta_o(s)$ giving the probability the option ends in state $s$. Running an option from a state means handing control to $\pi_o$ until $\beta_o$ fires, which takes some random number of primitive steps $k$. From the high-level planner's view, the option is one decision that advances the world by $k$ steps. A subgoal is a particularly clean special case: the option is "drive the low-level policy until you reach subgoal $g$", $\pi_o$ is a goal-conditioned policy $\pi(a \mid s, g)$, and $\beta_o$ fires when $s$ is close enough to $g$.

The object that lets a planner work at this coarse scale is the abstract (jumpy) transition model. Where the flat model $\hat{T}(s_{t+1} \mid s_t, a_t)$ advances one primitive step, the abstract model advances one whole option: it predicts the state $k$ steps later, where the option lands, and the cumulative discounted reward earned along the way, without imagining the intervening primitive steps at all. For an option $o$ started in state $s$, define the abstract transition

where $k$ is the (state-dependent, random) option duration set by $\beta_o$. This is a semi-Markov model: transitions take variable amounts of time, and the discount enters as $\gamma^{k}$ rather than $\gamma$. Planning with $\hat{T}^{o}$ and $\hat{R}^{o}$ is ordinary planning (value iteration, search, trajectory optimization) over a much smaller, much shallower problem, because each abstract step covers $k$ primitive steps. The high-level Bellman backup over options is

identical in form to the primitive Bellman equation of Chapter 22 except that the action set is options, the reward is a multi-step return, and the discount is raised to the option's duration. The crucial property is that the same dynamic-programming and search machinery applies: temporal abstraction does not require new algorithms, only a coarser model to run them on. A learned jumpy model is trained exactly to predict these $k$-step landings and cumulative rewards, so the planner never has to roll the flat model forward through the option's interior, which is precisely where the compounding error of subsection one lived.

The benefit is now quantifiable in the same terms as the problem. If options have average duration $k$, then reaching a goal $H$ primitive steps away takes $H/k$ abstract decisions. The abstract rollout that compounds model error is $H/k$ steps long, not $H$, so by the error formula of subsection one the compounded error drops from its $H$-step value to its $H/k$-step value, often a dramatic reduction because that sum grows superlinearly. The abstract search tree is $b^{H/k}$ rather than $b^{H}$, an exponential reduction in the exponent. And because the jumpy model predicts option landings directly, it can be trained to be accurate at exactly the abstract scale the planner uses, rather than being a flat model strained far past its reliable horizon. The low-level policies inside the options absorb the primitive-step detail that the high level no longer reasons about.

3. Recent Directions: Learned Skills and Goal-Directed Latent Planning Advanced

The modern instantiations of subsection two share a recipe: learn the world model and the abstraction together, in a latent space, end to end, rather than hand-designing options. Three threads define the 2023 to 2026 landscape, and all three sit directly on the jumpy-model-plus-low-level-policy split.

The first is goal-directed latent planning, whose cleanest exemplar is Director (Hafner et al., 2022). Director learns a latent world model (the recurrent state-space model family this chapter has been building) and then trains a high-level "manager" policy that proposes subgoals in the model's latent space every fixed number of steps, while a low-level "worker" policy is rewarded for reaching those latent subgoals. The manager plans in the compact latent state, abstractly and infrequently; the worker fills in primitive actions. Crucially, the subgoals are points in a learned representation, not hand-specified coordinates, so the abstraction is discovered rather than designed. Director solved long-horizon sparse-reward tasks (visual mazes, ant navigation) that flat model-based agents could not, and it is the canonical proof that latent subgoals are a workable temporal abstraction for planning.

The second is long-horizon model-based agents in the Dreamer line. DreamerV3 (Hafner et al., 2023) trains a latent world model and learns behaviors by imagining rollouts inside it, and it generalized across more than 150 tasks with fixed hyperparameters, including the famously long-horizon task of collecting diamonds in Minecraft from scratch, which demands credit assignment across thousands of steps. DreamerV3 reaches long horizons less by explicit subgoals than by a high-quality latent model plus a value function that propagates reward across imagined rollouts, but the recent trend (TD-MPC2 of Hansen et al., 2024, and follow-on planning-in-latent-space agents) explicitly combines learned latent dynamics with short-horizon planning and a learned terminal value, which is itself a form of abstraction: the learned value summarizes the infinite tail beyond the short plan, so a short imagined rollout plus a value reaches an effectively long horizon. The value function is the jumpiest model of all, collapsing the entire future after the plan into one number.

The third is learned skills and skill discovery. V-JEPA 2 (Meta, 2025) extends this thread to zero-shot settings: by predicting future representations in a self-supervised feature space pretrained on internet-scale video, it serves as a drop-in forward model for goal-conditioned robotic planning without any task-specific training, demonstrating that an agent's world model need not be learned from interaction at all, only from passive observation of how the world changes around agents that act (see Section 29.2 for the full V-JEPA 2 architecture and its comparison with Dreamer). Methods that learn a library of reusable skills without reward (mutual-information skill discovery in the DIAYN line, and its successors) produce exactly the options of subsection two, discovered from interaction; a planner then composes them. The 2023 to 2026 direction couples skill discovery with a world model so that the discovered skills are the action set of a jumpy planner, and recent offline and hierarchical methods (for instance hierarchical decision transformers and goal-conditioned offline RL of Chapter 28) learn both the skills and the high-level controller from logged data. The common thread across all three is the marriage this section has argued for from the start: a world model for imagining, an abstraction for keeping the horizon short, learned jointly.

4. Synthesis of Part VI: Choosing Your Structure on the Decision-Making Spectrum Advanced

Part VI has presented sequential decision making as a spectrum of methods, and this is the moment to lay the spectrum out whole and say how to choose along it. The chapters were not a list of competing algorithms but a graded series of answers to one question: how much structure should you build into the agent, and of what kind? Each family makes a different bet, and the right bet is a property of the problem, not of fashion.

The choice rule that threads the table is a single question asked in three forms. First, how expensive is interaction? Cheap, abundant interaction (a fast simulator) favors model-free value and policy methods that simply consume samples; expensive or unsafe interaction (a real robot, a clinical system, a market) favors model-based methods that squeeze more learning from each sample, and offline sequence models that need no new interaction at all. Second, how much do you already know? Known dynamics call for the optimal-control methods of Chapter 27 rather than learning a model you already have; an available expert calls for imitation; a large logged dataset calls for the offline and sequence-model methods of Chapter 28. Third, how long is the horizon? Short horizons are served by any of the families directly; long horizons are where flat methods hit the walls of subsection one and where the temporal abstraction of this section becomes not an optimization but a prerequisite. These three questions, not a benchmark leaderboard, are how a practitioner actually chooses.

Underneath all three questions sits the lesson that has recurred in every chapter of this book, and most insistently in Part VI: the right amount of structure beats brute force. Too little structure (pure model-free search over primitive actions on a long-horizon sparse-reward task) hits the compounding-error and branching walls and never learns. Too much structure (a hand-built model and hand-designed options that the real problem violates) is brittle and wrong in exactly the situations that matter. The methods that have moved the field forward, from the Kalman filter becoming the RNN, from the MDP becoming the sequence model, from flat planning becoming hierarchical planning, all found the structure that matched the problem's real regularities and let learning fill the rest. Temporal abstraction is the purest statement of that lesson: it is structure (a hierarchy of time scales) added exactly where the problem (a long horizon) demands it, and nowhere else.

5. Worked Example: A Coarse Subgoal Planner Beats Flat Planning Advanced

We make the whole argument executable on a long-horizon navigation task: a $50 \times 50$ grid where an agent must travel from a corner to the opposite corner, a primitive horizon of about a hundred steps. We compare two planners against a learned-style jumpy model. The flat planner samples primitive action sequences and rolls a one-step model forward across the full horizon, paying both walls of subsection one. The abstract planner plans over a handful of subgoals (waypoints) using a jumpy model that lands on a subgoal in one coarse step, then a trivial low-level policy walks between waypoints. We show the abstract planner reaching the goal where the flat one does not, then collapse it to a library reference. Code 29.5.1 sets up the environment and the flat baseline.

import numpy as np

rng = np.random.default_rng(0)
N = 50                                    # 50 x 50 grid; goal is ~100 primitive steps away
GOAL = np.array([N - 1, N - 1])
START = np.array([0, 0])
ACTIONS = np.array([[1, 0], [-1, 0], [0, 1], [0, -1]])   # 4 primitive moves

def step(s, a):
    """One-step true dynamics: move and clip to the grid (the 'real' env)."""
    return np.clip(s + a, 0, N - 1)

def rollout_cost(s0, action_seq):
    """Run a primitive action sequence; cost = remaining distance to goal."""
    s = s0.copy()
    for a in action_seq:
        s = step(s, ACTIONS[a])
    return np.abs(s - GOAL).sum(), s       # Manhattan distance left

def flat_plan(s0, horizon, n_samples=4000):
    """Flat sampling planner: sample primitive sequences of full length, keep the best."""
    best_cost, best_seq = np.inf, None
    for _ in range(n_samples):
        seq = rng.integers(0, 4, size=horizon)     # a length-`horizon` primitive plan
        c, _ = rollout_cost(s0, seq)
        if c < best_cost:
            best_cost, best_seq = c, seq
    return best_cost

H = 100                                   # full primitive horizon
flat_best = flat_plan(START, H, n_samples=4000)
print("flat planner: best remaining distance = %d (0 = reached goal)" % flat_best)

flat planner: best remaining distance = 54 (0 = reached goal)

Now the abstract planner. Code 29.5.2 introduces a jumpy model that advances the world by a whole option in one coarse decision (the option is "head toward subgoal $g$", and the jumpy model predicts where it lands without simulating the interior), and plans over a short chain of subgoals instead of primitive actions. This is the cure of subsection two built from scratch.

def low_level_policy(s, g):
    """Greedy goal-conditioned policy: take the primitive move that most reduces |s - g|."""
    best_a, best_d = None, np.inf
    for a in ACTIONS:
        d = np.abs(np.clip(s + a, 0, N - 1) - g).sum()
        if d < best_d:
            best_d, best_a = d, a
    return best_a

def jumpy_transition(s, g, max_k=N):
    """Abstract k-step transition: run the option 'go to g' until arrival; return landing + k."""
    k = 0
    while not np.array_equal(s, g) and k < max_k:
        s = np.clip(s + low_level_policy(s, g), 0, N - 1)   # low level fills in primitive steps
        k += 1
    return s, k                              # the high level sees only (landing, duration)

def abstract_plan(s0, n_subgoals=4, n_samples=2000):
    """Plan over a short chain of subgoals; the jumpy model jumps between them."""
    best_cost, best_chain = np.inf, None
    for _ in range(n_samples):
        # propose a chain of subgoals (coarse decisions), each a grid waypoint
        chain = rng.integers(0, N, size=(n_subgoals, 2))
        s, total_k = s0.copy(), 0
        for g in chain:
            s, k = jumpy_transition(s, g)    # one coarse step covers k primitive steps
            total_k += k
        s, k = jumpy_transition(s, GOAL)     # final coarse step aims at the true goal
        total_k += k
        c = np.abs(s - GOAL).sum()
        if c < best_cost:
            best_cost, best_chain = c, (total_k, chain)
    return best_cost, best_chain

abs_best, (steps_used, _) = abstract_plan(START, n_subgoals=4, n_samples=2000)
print("abstract planner: best remaining distance = %d  (used %d primitive steps)"
      % (abs_best, steps_used))
print("search depth: flat = %d primitive decisions, abstract = %d coarse decisions"
      % (H, 5))

abstract planner: best remaining distance = 0  (used 98 primitive steps)
search depth: flat = 100 primitive decisions, abstract = 5 coarse decisions

The contrast is the whole section in two numbers: 54 cells short versus 0, at search depth 100 versus 5. The abstract planner did not have a better model or more samples (it used fewer); it had a coarser decision unit, and that single change moved the horizon from unreachable to solved. Code 29.5.3 collapses the hand-built planner to its library form, where a hierarchical or goal-conditioned RL toolkit supplies the option machinery and the jumpy rollout.

# Library reference: hierarchical / goal-conditioned planning with a learned world model.
# Sketch using a Dreamer-style stack (e.g. via the `dreamerv3` or `tdmpc2` reference impls)
# plus a goal-conditioned low level; the jumpy model and option loop are provided.
#
# import gymnasium as gym
# from dreamerv3 import Agent          # learns latent world model + imagination planning
#
# env   = gym.make("LongHorizonNav-v0")
# agent = Agent(env.observation_space, env.action_space,
#               hierarchical=True,      # high-level subgoal manager + low-level worker
#               subgoal_every=8)        # one coarse decision per 8 primitive steps (the option length k)
# agent.train(env, steps=1_000_000)    # learns jumpy dynamics, manager, and worker jointly
# action = agent.act(obs)              # internally: manager proposes a latent subgoal, worker executes
print("library: subgoal manager + worker + jumpy model provided by the framework")

library: subgoal manager + worker + jumpy model provided by the framework

Read the three blocks together. Code 29.5.1 hit both long-horizon walls with flat planning and fell 54 cells short. Code 29.5.2 supplied temporal abstraction from scratch, a jumpy model and a depth-5 subgoal search, and reached the goal. Code 29.5.3 collapsed that hand-built hierarchy to the few configuration lines a modern hierarchical world-model agent needs. The numeric example below puts the horizon reduction in the same terms as subsection one.

Exercises