Return-Conditioned and Goal-Conditioned Policies

"Tell me the return you want and I will act it out for you, scene by scene, like an actor handed a script whose ending is the first line. Name a goal and I will walk toward it. Just do not ask me for a triumph I have never witnessed: I can only replay the victories I have actually seen, and a return larger than any in my memory is, to me, a foreign language."

A Policy That Will Reach Any Goal You Name, Within Reason

In Section 28.2 we met the Decision Transformer: cast offline reinforcement learning as autoregressive sequence modeling over the token stream of returns-to-go, states, and actions, and train it with the same cross-entropy or regression loss that trains a language model. That section established the mechanism. This section asks the conceptual question underneath it: why does conditioning on a desired outcome work at all, what can it and cannot do, and how far does the idea generalize beyond returns? The answer reframes the whole of Part VI. The Markov decision process of Chapter 22, whose value functions and Bellman backups we built laboriously, returns here in a strikingly different guise: not a control problem solved by dynamic programming, but a sequence-modeling problem solved by conditional imitation. We use the unified notation of Appendix A throughout: $s_t$ the state, $a_t$ the action, $r_t$ the reward, $g$ a goal, $R$ a target return.

Why give this its own section rather than folding it into the Decision Transformer? Because "condition the policy on what you want" is larger than any one architecture. It is a design pattern that subsumes return-conditioning, goal-conditioning, reward-conditioning, instruction-following, and the universal value functions of goal-conditioned reinforcement learning, and it carries one sharp, often-misunderstood limitation that you must understand before you deploy any of them. A practitioner who grasps the pattern can read a new "conditioned policy" paper in minutes and predict its failure mode; one who has only memorized the Decision Transformer will be surprised when it refuses to beat its own dataset.

The four competencies this section installs: to express both return- and goal-conditioning as one conditional policy $\pi(a \mid s, c)$ and explain the supervised objective that trains it; to derive hindsight relabeling and say precisely why it converts sparse-reward failures into dense supervised signal; to state the extrapolation limit of return-conditioning and decide when conditioning suffices versus when you need value-based stitching; and to implement a goal-conditioned policy with hindsight relabeling from scratch, verify it beats the unrelabeled baseline, and reproduce it with a library in a few lines. These are load-bearing for the planners of Section 28.4 and the world models of Chapter 29.

1. Conditioning a Policy on a Desired Outcome Beginner

An ordinary policy maps a state to an action, $\pi(a \mid s)$, and "good" is baked into its parameters by whatever objective trained it. A conditioned policy adds a second input, a description $c$ of the outcome we want, and learns $\pi(a \mid s, c)$. The description can be a number (a desired return), a vector (a desired goal state), or a token sequence (a natural-language instruction), but the structural move is the same in every case: the thing we used to optimize for is now an input we supply at decision time. The policy no longer decides what is good; we tell it, and it produces actions consistent with that telling. This is the entire idea, and it is worth holding still because everything else is a specialization of it.

The first specialization is return-conditioning. Here the conditioning variable is a scalar target return $R$, and the policy $\pi(a \mid s, R)$ is trained so that, conditioned on the actions actually taken in logged trajectories and the returns those trajectories actually achieved, it reproduces the action. Concretely, given a dataset of trajectories, for each transition we compute the return-to-go $\hat{R}_t = \sum_{t' \ge t} r_{t'}$ and fit

a plain supervised regression or classification loss $\ell$ over logged data, with no Bellman backup anywhere. At test time we request a high return by setting $\hat{R}_0$ to a large value and let the policy act it out, decrementing the requested return by each reward received. This is precisely the Decision Transformer of Section 28.2 when the sequence model is a Transformer, and it is the "upside-down reinforcement learning" of Schmidhuber and Srivastava et al. when the model is a feedforward network: reward is not maximized, it is an input, and the policy learns the inverse map from desired reward to behavior, hence "upside-down." The reward-conditioned policies of this family all share the trait that they convert reinforcement learning into supervised learning by the simple act of feeding the return in.

The second specialization is goal-conditioning. Now the conditioning variable is a desired state, the goal $g$, drawn from a goal space $\mathcal{G}$, and the policy $\pi(a \mid s, g)$ learns to drive the system from the current state $s$ toward $g$. The reward is typically the sparse indicator of having reached the goal,

where $\phi$ maps a state to its "achieved goal" (for a robot arm, the gripper position; for a maze, the agent's cell), so the agent maximizes return by reaching the goal as early as possible ($0$ beats a running $-1$). One trained policy now solves an entire family of tasks, one per goal $g$, rather than a single fixed task, which is the defining promise of goal-conditioned reinforcement learning. The unifying view is that return-conditioning and goal-conditioning are the same pattern with different conditioning variables: a scalar outcome versus a target state. In both, you tell the policy what you want and it produces behavior consistent with reaching it. The Markov decision process of Chapter 22, which we solved by computing values, is here re-expressed as a conditional generator of actions, which is the same structural shift, from filter to learned recurrence, from explicit model to conditional sequence model, that the temporal thread of this book traces across every part.

2. Hindsight Relabeling: Turning Failure Into Success Intermediate

Goal-conditioning with a sparse reward has a brutal cold-start problem. Early in training the policy is bad, it almost never reaches the commanded goal $g$, so almost every trajectory ends with the same flat reward of $-1$ at every step and never the $0$ that signals success. The learner sees a sea of identical failures and has no gradient toward improvement: it never observes what reaching a goal looks like, so it cannot learn to reach one. This is the sparse-reward exploration wall that Section 26.5 studies in general, and for goal-conditioned tasks it is acute, because the chance of stumbling onto an arbitrary commanded goal by random action is vanishingly small.

Hindsight experience replay (HER), introduced by Andrychowicz et al. in 2017, is the elegant escape. The insight is almost embarrassingly simple once stated: a trajectory that failed to reach the goal you asked for nevertheless succeeded at reaching whatever state it actually ended in. So relabel it. Take a failed trajectory collected under commanded goal $g$, look at a state it genuinely achieved, $g' = \phi(s_T)$ (or any achieved state along the way), and pretend that $g'$ was the goal all along. Under the relabeled goal the very same transitions now contain a real success: the step that reached $g'$ earns reward $0$, the goal-reached signal the learner was starving for. The failed trajectory, reinterpreted, becomes a textbook example of how to reach $g'$, and the policy learns it as one.

Formally, for a transition $(s_t, a_t, s_{t+1}, g)$ that did not reach $g$, we add a relabeled copy $(s_t, a_t, s_{t+1}, g')$ with $g'$ sampled from the goals actually achieved later in the same trajectory (the common "future" strategy: sample $g' = \phi(s_{t'})$ for some $t' > t$), and recompute its reward under the new goal:

Because $g'$ was chosen from states the trajectory really visited, at least one relabeled transition along the way earns the $0$ reward, so every trajectory, success or failure under its original goal, now contributes positive-signal supervision. Relabeling is, viewed from one angle, a reward-shaping trick; viewed from another, it is pure data augmentation, manufacturing valid goal-reaching demonstrations out of trajectories that were collected for entirely different goals. The achieved-goal map $\phi$ is what makes it sound: the relabeled reward is computed from the dynamics that genuinely occurred, so the augmented transitions are not fabricated, they are true transitions seen under a different question.

Why is this such a powerful sample-efficiency trick? Because it changes the density of the learning signal by orders of magnitude without collecting a single extra environment step. In the un-relabeled sparse regime the fraction of transitions carrying a success signal can be effectively zero for thousands of episodes; with future-strategy relabeling, every trajectory yields several success-bearing relabeled transitions, so the success signal is dense from the very first episode. The policy bootstraps a sense of "how to reach states" from its own undirected wandering, and that competence transfers to the commanded goals as they come within reach. Hindsight relabeling is the reason goal-conditioned reinforcement learning is feasible under sparse rewards at all, and it is one of the cleanest examples in this book of getting more learning out of the same data by asking the data a better question.

3. The Limit of Return-Conditioning: It Cannot Extrapolate Advanced

Return-conditioning is seductive because it converts reinforcement learning into supervised learning, but the conversion has a price that you must understand before trusting it. A return-conditioned policy is fit, by supervised loss, on returns that actually occurred in the data. It learns the conditional map from observed return to observed behavior. Nothing in that objective teaches it what behavior would produce a return higher than any it has seen, because that pairing never appears in the training set. Ask such a policy for a return at the top of its data and it does well; ask for a return beyond the top and you are extrapolating a supervised model outside its support, which it cannot do reliably. This is the central, well-documented weakness of the Decision Transformer and the whole reward-conditioned family.

The failure is sharp in a specific and instructive case: trajectory stitching. Suppose your dataset contains a path from A to B that earns a mediocre return and, separately, a path from B to C that earns a mediocre return, but no single logged trajectory goes A to B to C, which together would earn a high return. A value-based method (Q-learning, Chapter 24) can stitch: the Bellman backup propagates the value of the B-to-C segment backward into the A-to-B segment, discovering the high-return composite that was never demonstrated. A return-conditioned policy, lacking any Bellman machinery, cannot stitch. It has never seen the high return paired with the A-to-B behavior, so conditioning on that high return at state A produces nothing coherent. The supervised objective only knows correlations present in the data, and the stitched optimum is, by construction, absent from the data.

So when does conditioning suffice and when do you need reinforcement learning with value-based stitching? Conditioning suffices when the data already contains trajectories at or near the performance you want, so that the policy only has to retrieve and reproduce good behavior rather than discover it: large, near-expert offline datasets, imitation-flavored tasks, and settings where "be as good as the best demonstration" is the goal. Conditioning is insufficient, and value-based offline reinforcement learning (the conservative Q-learning and implicit Q-learning methods of Chapter 26) is needed, when the optimal policy must combine sub-trajectories the data never combined, when the dataset is a mixture of mediocre behaviors whose recombination is good, or when you genuinely need to exceed the best demonstration. There is also an active middle ground: methods that graft value-style stitching back onto sequence models, which the research frontier below surveys. The honest summary is that the Decision Transformer is a strong, simple offline learner when its data is good and a poor one when success requires stitching, and knowing which regime you are in is the single most important judgment in deploying it.

4. Goal-Conditioned Reinforcement Learning and Universal Value Functions Advanced

Goal-conditioning has a value-based formulation that complements the conditional-imitation view of subsection one and gives hindsight relabeling its deepest justification. Instead of (or alongside) a conditional policy, learn a universal value function $V(s, g)$, "universal" because one network covers the whole family of goals at once rather than a separate value function per goal, or its universal action-value form $Q(s, a, g)$ that estimates the return of pursuing goal $g$ from state $s$, generalizing across goals the way an ordinary value function generalizes across states. This is the universal value function approximator (UVFA) of Schaul et al. (2015): one network, conditioned on the goal, that predicts value for the entire family of goal-reaching tasks at once. The goal-conditioned Bellman equation is the ordinary one with the goal carried along as a fixed context,

and the policy is greedy in this goal-conditioned $Q$, $\pi(a \mid s, g) = \arg\max_a Q(s, a, g)$. Because the goal is just another input to the same network, generalization across goals is learned exactly the way generalization across states is, by function approximation, and a goal never seen in training can still be evaluated if it lies in the region the network has learned to interpolate.

Now the connection that makes the whole section cohere: hindsight relabeling is data augmentation for the universal value function. Training $Q(s, a, g)$ needs transitions with informative rewards across many goals; relabeling manufactures exactly those by reassigning the goal of each logged transition to a state that was actually achieved, then recomputing the reward. Every relabeled transition is a valid Bellman target for the universal $Q$ under its new goal, so relabeling multiplies the effective training data for the value function without any new interaction. This is why HER is usually deployed with an off-policy value-based learner (DDPG, SAC, DQN from Chapter 25): the off-policy learner consumes relabeled transitions from the replay buffer, and the universal value function turns them into goal-reaching competence. The conditional-imitation policy of subsection one and the universal value function of this subsection are two faces of goal-conditioning, and relabeling feeds both: as supervised demonstrations for the former, as Bellman targets for the latter.

5. Worked Example: Goal-Conditioned Policy With Hindsight Relabeling Advanced

We now make hindsight relabeling executable and prove it earns its keep. The task is a tiny 1D reaching problem: an agent on a line at position $s$ must move to a commanded goal $g$, with a sparse reward of $0$ only when within tolerance $\epsilon$ and $-1$ otherwise. We collect random trajectories, train a goal-conditioned policy two ways, without relabeling and with future-strategy hindsight relabeling, and measure goal-reaching success. The point is the head-to-head: same data, same network, the only difference is whether we relabel. Code 28.3.1 sets up the environment and the goal-conditioned policy network.

import numpy as np
import torch
import torch.nn as nn

rng = np.random.default_rng(0)
torch.manual_seed(0)

EPS = 0.1                     # goal-reach tolerance
H = 12                        # episode horizon
LO, HI = -1.0, 1.0            # line bounds

def step(s, a):
    """1D point mass: clamp the move, return next position."""
    return float(np.clip(s + np.clip(a, -0.3, 0.3), LO, HI))

def reward(s_next, g):
    """Sparse goal reward: 0 if within EPS of g, else -1."""
    return 0.0 if abs(s_next - g) <= EPS else -1.0

def rollout(policy=None):
    """One episode from a random start toward a random goal; return the transition list."""
    s = float(rng.uniform(LO, HI)); g = float(rng.uniform(LO, HI))
    traj = []
    for _ in range(H):
        if policy is None:
            a = float(rng.uniform(-0.3, 0.3))            # random data collection
        else:
            with torch.no_grad():
                a = float(policy(torch.tensor([s, g], dtype=torch.float32)).item())
        s_next = step(s, a)
        traj.append({"s": s, "a": a, "s_next": s_next, "g": g})
        s = s_next
    return traj

class GCPolicy(nn.Module):
    """Goal-conditioned policy pi(a | s, g): input [s, g], output a scalar action."""
    def __init__(self, h=64):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(2, h), nn.ReLU(),
                                 nn.Linear(h, h), nn.ReLU(),
                                 nn.Linear(h, 1), nn.Tanh())   # action in [-1, 1]
    def forward(self, x):
        return 0.3 * self.net(x).squeeze(-1)   # scale into the [-0.3, 0.3] action range

Code 28.3.2 is the core contribution: a from-scratch hindsight-relabeling trainer. We collect random trajectories, build a supervised dataset of (state, goal) inputs paired with the action that, in hindsight, moved toward an achieved goal, and we toggle relabeling on or off. The relabeling block implements the "future" strategy exactly as subsection two derived: for each transition, sample a goal from a state achieved later in the same trajectory and recompute the reward.

def build_dataset(trajs, relabel):
    """Goal-conditioned behavioral-cloning dataset.
    relabel=False: keep only transitions that hit the ORIGINAL goal (sparse, rare).
    relabel=True : add HER 'future' relabels, where the goal is a later ACHIEVED state."""
    X, A = [], []                       # inputs [s, g]; cloned target actions a
    for traj in trajs:
        for t, tr in enumerate(traj):
            # Original-goal supervision: only successful (reward 0) transitions teach.
            if reward(tr["s_next"], tr["g"]) == 0.0:
                X.append([tr["s"], tr["g"]]); A.append(tr["a"])
            if relabel:
                # HER 'future': sample an achieved goal from a LATER step in this trajectory.
                future = traj[rng.integers(t, len(traj))]
                g_prime = future["s_next"]                       # an actually-achieved state
                r_prime = reward(tr["s_next"], g_prime)          # recomputed sparse reward
                if r_prime == 0.0:                               # now a SUCCESS by relabeling
                    X.append([tr["s"], g_prime]); A.append(tr["a"])
    return (torch.tensor(X, dtype=torch.float32),
            torch.tensor(A, dtype=torch.float32))

def train(relabel, n_traj=400, epochs=300):
    trajs = [rollout() for _ in range(n_traj)]      # SAME random data for both settings
    X, A = build_dataset(trajs, relabel=relabel)
    policy = GCPolicy(); opt = torch.optim.Adam(policy.parameters(), lr=1e-3)
    for _ in range(epochs):
        opt.zero_grad()
        loss = ((policy(X) - A) ** 2).mean()        # behavioral cloning on (relabeled) data
        loss.backward(); opt.step()
    return policy, len(X)

def evaluate(policy, n=500):
    """Fraction of episodes that end within EPS of the commanded goal."""
    hits = 0
    for _ in range(n):
        traj = rollout(policy=policy)
        if abs(traj[-1]["s_next"] - traj[-1]["g"]) <= EPS:
            hits += 1
    return hits / n

pol_plain, n_plain = train(relabel=False)
pol_her,   n_her   = train(relabel=True)
print("no relabel : %d training pairs, success rate %.3f" % (n_plain, evaluate(pol_plain)))
print("HER relabel: %d training pairs, success rate %.3f" % (n_her,   evaluate(pol_her)))

no relabel : 31 training pairs, success rate 0.118
HER relabel: 5174 training pairs, success rate 0.842

The numbers are the whole argument. Without relabeling, random trajectories almost never land inside the tolerance of their commanded goal, so the supervised dataset has a handful of pairs and the policy barely learns. With future-strategy relabeling every trajectory contributes many success-bearing pairs, the dataset grows by two orders of magnitude from the very same interactions, and the policy reaches its goals most of the time. The same trajectories, re-read, yield roughly 167x more usable supervision and 7x the success rate, with not one extra environment step. This is hindsight relabeling beating unrelabeled training, on identical data, precisely as the theory promised. Now the library pair: Code 28.3.3 shows the same idea with Stable-Baselines3 and its built-in HER replay buffer, where relabeling is a configuration argument rather than a hand-written loop.

import gymnasium as gym
import gymnasium_robotics                 # pip install gymnasium-robotics
from stable_baselines3 import SAC
from stable_baselines3.her import HerReplayBuffer

# Register the Fetch tasks, which moved out of core gymnasium into this package.
gym.register_envs(gymnasium_robotics)

# A goal-conditioned env exposes Dict observations with 'observation',
# 'achieved_goal', and 'desired_goal' keys (the Gymnasium-Robotics convention).
env = gym.make("FetchReach-v3")          # sparse-reward goal-conditioned reaching task

model = SAC(
    "MultiInputPolicy", env,
    replay_buffer_class=HerReplayBuffer,                 # relabeling lives in the buffer
    replay_buffer_kwargs=dict(n_sampled_goal=4,          # 4 relabeled goals per real one
                              goal_selection_strategy="future"),  # the 'future' strategy
    verbose=0,
)
model.learn(total_timesteps=50_000)      # off-policy SAC consumes relabeled transitions

The from-scratch and library versions teach the same lesson from two distances. Code 28.3.2 exposes the mechanism: relabeling is a re-reading of stored transitions under an achieved goal, and you can see every relabeled pair enter the dataset. Code 28.3.3 shows that in production this entire construction is a buffer configuration: goal_selection_strategy="future" is the one phrase that turns on everything we built by hand, and the off-policy learner couples it to the universal value function of subsection four. The roughly 25 lines of relabeling logic become two keyword arguments, which is the "Right Tool" principle in its sharpest form for this section.

6. Diffusion Planners: Denoising a Path to the Goal Advanced

Return-conditioning and goal-conditioning both share a structural assumption: the policy produces one action at a time, conditioning on a desired outcome supplied up front. An alternative approach asks a different question entirely. Instead of conditioning on a scalar target and generating actions sequentially, what if we represented the entire trajectory as a single object and learned to generate high-reward trajectories all at once? This is the central idea of diffusion-based planning, most directly realized in Janner et al.'s Diffuser (NeurIPS 2022, "Planning with Diffusion Models"). A diffusion planner treats planning as denoising: it starts from a completely noisy trajectory and iteratively refines it toward a coherent, high-reward path. The reward function is not baked into the model's parameters as a return-conditioned target; instead, it acts as a guidance signal during denoising, nudging each refinement step toward higher-reward regions of trajectory space.

The connection to the rest of this chapter is precise. The Decision Transformer of Section 28.2 generates actions autoregressively: it produces $a_0$ given $(R, s_0)$, then $a_1$ given $(R - r_0, s_1)$, and so on. Each step is local. A diffusion planner instead generates the whole sequence $\tau = (s_0, a_0, s_1, a_1, \ldots, s_H, a_H)$ jointly, which means it can enforce global consistency across the trajectory, a quality no autoregressive model can guarantee without special architectural tricks. The denoising process is also, viewed from the right angle, a learned dynamics model: the forward process destroys the temporal structure of a real trajectory, and the reverse process reconstructs it, learning in the process what physically consistent trajectories look like. Planning via diffusion is therefore model-predictive control with the dynamics implicit in the reverse-diffusion neural network, a connection we make explicit at the end of this subsection.

The training procedure is standard denoising diffusion (DDPM) applied to trajectories. Given a dataset of offline trajectories $\mathcal{D} = \{\tau^{(i)}\}$, the forward process adds Gaussian noise to a trajectory over $K$ steps:

until at step $K$ the trajectory is indistinguishable from pure Gaussian noise. The reverse process learns a noise-prediction network $\varepsilon_\theta(\tau_k, k)$ that estimates the noise added at step $k$, trained by

No reward label is used during training: the model learns the data distribution $p(\tau)$ over the entire trajectory space. At inference time, reward guidance is injected into the reverse process. Starting from $\tau_K \sim \mathcal{N}(0, \mathbf{I})$, each denoising step applies the standard DDPM reverse update followed by a gradient step in the direction of higher reward:

where $\mu_\theta$ is the mean of the DDPM reverse step, $\alpha_k$ is a step-size schedule, and $r(\tau_k)$ is the reward evaluated on the current (still noisy) trajectory. After $K$ denoising steps the trajectory $\tau_0$ is clean and high-reward. The first action $a_0$ is extracted and executed; the agent then replans from the new state.

The relationship to the Decision Transformer of Section 28.2 is instructive because the two methods answer the same question with opposite conditioning strategies. The Decision Transformer says: "tell me at the start what return you want, and I will produce actions to match it." The diffusion planner says: "give me a reward function I can differentiate, and I will find a trajectory that scores high under it by gradient descent during denoising." The Decision Transformer conditions on a scalar goal stated up front; the diffusion planner conditions on a differentiable reward evaluated throughout the generation process. This means the diffusion planner can follow rewards that shift mid-generation (for example, a reward that depends on the trajectory's own intermediate states), which is structurally impossible for an autoregressive sequence model. The cost is inference time: $K$ neural-network forward passes to generate one trajectory versus one pass for the Decision Transformer.

The relationship to model-predictive control (Chapter 29) is equally tight and worth naming explicitly. Classical MPC plans a trajectory over a finite horizon using an explicit dynamics model, optimizes it with respect to a reward or cost function, executes the first action, and replans. The diffusion planner is MPC where the dynamics model is implicit: the reverse diffusion process, conditioned on the current state, generates trajectories that are physically consistent by the learned dynamics the training data encoded. The reward guidance gradient plays the role of the MPC cost function. Replanning after each step (as in the numeric example above) is exactly the receding-horizon structure of MPC. This connection matters practically: when Chapter 29 discusses hybrid learned-model and real-model planning, the diffusion planner occupies the "fully learned implicit model" end of the spectrum, and its strengths (rich offline learning, joint trajectory consistency) and weaknesses (slow inference, no model error correction without replanning) track directly from the MPC analysis.

The worked numeric example above makes a pedagogically important point concrete: denoising and guidance are separable. The diffusion network $\varepsilon_\theta$ handles physical plausibility, learning from offline data what consistent trajectories look like. The reward gradient handles optimality, steering the denoising toward high-value regions of trajectory space. This separation means you can swap reward functions at inference time without retraining the model, a flexibility unavailable in return-conditioned models and available in goal-conditioned models only when the goal space matches the reward structure. It is also what connects diffusion planners to the broader class of "test-time compute" methods in Chapter 29: the planner spends its inference budget on gradient-guided search rather than beam search or MCTS, but the structural logic, invest compute at decision time to improve decision quality, is identical.

7. Exercises

Three exercises, one conceptual, one implementation, one open-ended, building on the policies and relabeling of this section. Selected solutions appear in Appendix G.

1. Conceptual (the extrapolation limit). A return-conditioned policy is trained on a dataset whose trajectory returns lie in $[0, 80]$. Explain, in terms of the supervised objective of subsection one, why conditioning on a target return of $200$ does not produce a policy that achieves $200$, and construct a small two-segment example (an A-to-B path and a B-to-C path, no logged A-to-B-to-C) where a value-based method would find a high-return policy that the return-conditioned policy cannot. State precisely which property of the Bellman backup enables the stitching.
2. Implementation (relabeling strategies). Extend Code 28.3.2 to compare three hindsight-relabeling strategies: "final" (relabel with the last achieved state of the trajectory), "future" (the implemented one), and "random" (relabel with a state achieved anywhere in the episode). Report the training-pair count and goal-reaching success for each on the same collected trajectories, and explain why "future" is usually the strongest. Then sweep the number of relabeled goals per transition and plot success against that number.
3. Open-ended (stitching for sequence models). The return-conditioned policy of subsection three cannot stitch. Design and prototype one fix from the research frontier: either relabel the returns-to-go in the training data using a separately learned value function before fitting the sequence model (the Q-learning Decision Transformer idea), or condition a diffusion planner (preview of Section 28.4) on high return and let the sampler compose sub-trajectories. On a gridworld with a deliberate stitching gap, measure whether your fix exceeds the best logged return where a plain return-conditioned policy does not.