"For years they handed me a value function and a credit-assignment problem and told me to bootstrap my way to a policy, one trembling temporal-difference step at a time, chasing a target that moved every time I touched it. Then someone fed me a pile of trajectories the way you would feed a language model a pile of sentences, marked each one with the return it eventually earned, and simply asked me to predict the next action. No Bellman equation, no target network, no divergence. I had spent a career learning to act; it turned out I only ever needed to learn to autocomplete. I am told this is either the deepest unification in the field or an elaborate way of avoiding the hard parts, and on most days I genuinely cannot tell which."
A Decision Transformer Conditioning on a Return It Cannot Reach
Chapter Overview
Everything in Part VI so far has treated decision making as a control problem: estimate a value function, follow a Bellman equation, push a policy up a gradient, and wrestle the moving targets and bootstrapping instabilities that come with it. This chapter makes a different and surprisingly elegant move. It looks at a trajectory of states, actions, and rewards and notices that this is just a sequence, the same kind of object that Part III spent seven chapters learning to model. If a Transformer can autocomplete the next token of a sentence or the next value of a time series, perhaps it can autocomplete the next action of a behaving agent. Reinforcement learning, recast this way, becomes sequence modeling: collect trajectories, train a sequence model to predict what comes next, and generate good behavior the way a language model generates plausible text. This is one of the major thesis payoffs of the book, the point where Part III and Part VI fuse into a single idea.
The chapter builds that idea through its two founding architectures. The Decision Transformer feeds the model a sequence of returns-to-go, states, and actions, then conditions generation on a desired return: tell it you want a high return and it autocompletes the actions that, in the training data, led to that return, turning policy learning into conditional sequence generation with no dynamic programming at all. The Trajectory Transformer takes the complementary view, modeling the full joint distribution over states, actions, and rewards as a sequence and using beam search over it as a planner. Both abandon the value-iteration machinery of Chapters 24 through 26 in favor of a single autoregressive model, and both inherit, for free, the scaling, the long-context attention, and the engineering maturity that Part III's sequence models already enjoy.
From these foundations the chapter generalizes the conditioning signal. Return-conditioned policies are one instance of a broader pattern, hindsight conditioning, whose oldest form is Schmidhuber's upside-down reinforcement learning: instead of maximizing reward, you learn a command-conditioned mapping from a desired outcome to the actions that achieve it, and you relabel past trajectories with the outcomes they actually reached so that every trajectory becomes a successful demonstration of some goal. Goal-conditioned policies apply the same logic spatially, conditioning on a target state rather than a return, and connect directly to the hindsight experience replay idea that rescues learning from sparse rewards. The chapter then turns to a generative alternative: diffusion planners such as Diffuser and the Decision Diffuser treat planning as sampling, denoising an entire trajectory at once into a feasible, high-return plan rather than rolling out one action at a time, importing the diffusion machinery of Chapter 17 directly into decision making.
The final movement is the bridge that gives the chapter its place in the book's architecture. If a decision problem is a sequence-modeling problem, then any sequence model is a candidate policy, and the chapter makes that explicit by reusing the Transformers and structured state-space models of Part III as the backbone for control. The same attention mechanism that forecasts demand or models language can map a context of past states and a target return to the next action; the same linear-time state-space model that handles long sequences can serve as a memory-efficient policy over long horizons. Generalist agents like Gato take this to its conclusion, training one sequence model across control, vision, and language at once. The lesson the chapter leaves you with is the book's central thesis in its sharpest form: the architectures you learned to forecast with are, with almost no modification, the architectures you decide with.
Prerequisites
This chapter sits at the confluence of two earlier currents, and it assumes you have travelled both. From Part VI it builds directly on Chapter 26: Advanced Reinforcement Learning, in particular offline reinforcement learning: the entire premise of treating decision making as sequence modeling is that you have a fixed dataset of logged trajectories to imitate and condition on, so the distribution-shift and behavior-cloning concerns of the offline setting are exactly the concerns here. From Part III you need the Transformer in working detail, so Chapter 12: Attention and Transformers is a hard prerequisite: self-attention, causal masking, positional encoding, and autoregressive generation are the machinery every method in this chapter runs on, because a Decision Transformer is a Transformer with returns, states, and actions standing in for tokens. The diffusion planners of Section 28.4 assume the score-based and denoising-diffusion foundations from Section 17.4 of Chapter 17: Generative Temporal Models, since a diffusion planner is a trajectory-space denoiser. Readers wanting to refresh the reinforcement-learning vocabulary of returns and policies, the attention mechanism, or the diffusion forward and reverse processes will find the refreshers through the Table of Contents.
If you keep one idea from this chapter, keep this: a trajectory of states, actions, and rewards is a sequence, so reinforcement learning can be recast as sequence modeling, in which a Transformer, structured state-space model, or diffusion model is trained to generate good behavior conditioned on a desired return or goal, with no value iteration or Bellman bootstrapping at all. The Decision Transformer and Trajectory Transformer turn offline trajectories into autoregressive generation and planning; return- and goal-conditioning generalize through hindsight relabeling and upside-down reinforcement learning; diffusion planners denoise whole trajectories into feasible plans; and the deepest payoff is the bridge that the same Part III forecasting architectures serve, almost unchanged, as policies.
Chapter Roadmap
- 28.1 RL as Sequence Modeling The reframing that unifies Part III and Part VI: viewing a trajectory of states, actions, and rewards as a sequence, why predicting the next action is a sequence-modeling problem, and what is gained (scaling, stability, the whole sequence-model toolbox) and given up (explicit value iteration and dynamic programming) by making the switch.
- 28.2 Decision Transformer and Trajectory Transformer The two founding architectures: the Decision Transformer feeding returns-to-go, states, and actions to a causal Transformer and conditioning generation on a target return, and the Trajectory Transformer modeling the full joint distribution over a trajectory and using beam search over it as a planner.
- 28.3 Return-Conditioned and Goal-Conditioned Policies Generalizing the conditioning signal: hindsight relabeling and Schmidhuber's upside-down reinforcement learning that turn every trajectory into a successful demonstration of some outcome, goal-conditioned policies that condition on a target state, and the connection to hindsight experience replay under sparse rewards.
- 28.4 Diffusion Planners Planning as generative sampling: Diffuser denoising an entire trajectory at once into a feasible, high-return plan, the Decision Diffuser adding classifier-free return and constraint conditioning, and why importing the diffusion machinery of Chapter 17 replaces step-by-step rollout with whole-plan generation.
- 28.5 Bridge: Transformers and SSMs from Part III as Policies The thesis payoff made explicit: reusing the attention models and structured state-space models of Part III, almost unchanged, as the backbone for control, the generalist agents (Gato, Multi-Game Decision Transformer) that train one sequence model across control, vision, and language, and the closing of the forecasting-to-decision loop.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single progression on offline trajectory data. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have trained the Decision Transformer of Section 28.2, rolled it out under a target return as Section 28.1 frames it, added the hindsight goal relabeling of Section 28.3, and sketched the diffusion planner of Section 28.4, all on a backbone that Section 28.5 reveals to be a Part III sequence model wearing a policy's clothes.
Hands-On Lab: Decisions as a Sequence
Objective
Train an agent without ever writing a Bellman update. You will take a dataset of offline trajectories and treat decision making as the sequence-modeling problem that Section 28.1 reframes it to be, building a Decision Transformer from Section 28.2 that ingests returns-to-go, states, and actions and learns to autocomplete the next action. You will then roll the trained model out by conditioning it on a target return, watching the same model produce cautious or aggressive behavior depending only on the return token you prepend, the central trick of return-conditioned generation. Next you will add the hindsight goal relabeling of Section 28.3, rewriting trajectories with the outcomes they actually reached so that failures become successful demonstrations of the goals they happened to hit, and observe how this rescues learning from sparse-reward data. Finally you will sketch the diffusion planner of Section 28.4, treating planning as denoising a whole trajectory rather than emitting one action at a time. The single thread is that every step here is sequence generation, and the backbone you train is, as Section 28.5 insists, a Part III architecture reused as a policy.
What You'll Practice
- Encoding offline trajectories as returns-to-go, state, and action sequences for a causal Transformer, the data move of Section 28.1.
- Implementing and training a Decision Transformer on the encoded trajectories, following Section 28.2.
- Rolling the model out under a chosen target return and reading how the conditioning return shapes behavior, the core of Section 28.3.
- Adding hindsight goal relabeling to turn sparse-reward failures into successful goal demonstrations, following Section 28.3.
- Sketching a diffusion planner that denoises an entire trajectory into a plan, the generative alternative of Section 28.4.
Setup
You need a Python environment with torch for the sequence model, numpy for the trajectory bookkeeping, and an offline trajectory dataset, for example a small D4RL-style buffer of transitions saved as states, actions, rewards, and terminals. The one idea that governs the whole lab is that the return-to-go, the discounted sum of rewards from each timestep onward, is the conditioning signal that lets a plain sequence model behave like a policy: compute it once per trajectory and the rest is autocompletion. The code below computes returns-to-go for a single trajectory, the small preprocessing step from Section 28.1 that turns a logged rollout into the conditioned sequence a Decision Transformer trains on.
import numpy as np
def returns_to_go(rewards, gamma=1.0):
# rewards: 1D array for one trajectory, oldest step first
rtg = np.zeros_like(rewards, dtype=np.float32)
running = 0.0
for t in reversed(range(len(rewards))):
running = rewards[t] + gamma * running # discounted sum from t onward
rtg[t] = running
return rtg # the return token prepended to each (state, action)
Steps
Step 1: Encode trajectories as sequences
Load the offline dataset and convert each trajectory into the interleaved sequence of returns-to-go, states, and actions that a Decision Transformer consumes, using the return-to-go helper above, the encoding of Section 28.1. Confirm that the same raw transitions you would have fed a value-based agent now look like a token stream ready for a sequence model.
Step 2: Train a Decision Transformer
Build a causal Transformer that takes the encoded sequence and predicts the next action, and train it by supervised next-action prediction with no Bellman update anywhere, the architecture of Section 28.2. Watch the loss fall as the model learns to autocomplete behavior from the logged data.
Step 3: Roll out under a target return
Deploy the trained model by prepending a desired return-to-go and autoregressively generating actions in the environment, decrementing the return by each reward received, the conditioned generation of Section 28.1. Vary the target return up and down and observe how the same model produces more or less aggressive behavior driven only by the conditioning token.
Step 4: Add hindsight goal relabeling
Relabel trajectories with the outcomes they actually reached, conditioning on a target goal state rather than only a return, so that even unsuccessful rollouts become successful demonstrations of the goals they happened to achieve, the hindsight idea of Section 28.3. Retrain and check whether goal-conditioned generation improves on a sparse-reward slice of the data.
Step 5: Sketch a diffusion planner
Outline, and if time allows prototype, a diffusion planner that treats a fixed-length trajectory as the object to be denoised and samples a high-return plan in one generative pass rather than one action at a time, the alternative of Section 28.4. Contrast its whole-plan generation with the autoregressive rollout of Step 3.
Expected Output
The lab produces a trained agent and a clear demonstration of the chapter's thesis. Steps 1 and 2 yield a Decision Transformer whose falling loss shows that policy learning has become next-action prediction, with no value function in sight. Step 3 produces the signature result of return conditioning: a single trained model whose behavior slides from cautious to aggressive as you raise the target return, making concrete that the return token, not a separate optimization, is steering the policy. Step 4 shows hindsight relabeling extracting useful goal-conditioned behavior from data too sparse for a value-based method to learn from. Step 5 leaves you with a working mental model, and ideally a prototype, of planning as trajectory denoising. The reader finishes able to implement a sequence-model policy end to end and, more importantly, able to see why the backbone they trained is the same Transformer they met in Part III, which is exactly the bridge Section 28.5 draws.
The from-scratch Decision Transformer of Steps 1 and 2 is worth building once so that returns-to-go, the interleaved token stream, and conditioned generation stop being abstractions, but you should rarely reimplement the plumbing in practice. The d3rlpy library ships Decision Transformer alongside the offline value-based methods of Chapter 26 behind a uniform fit and predict API against the same trajectory buffers, collapsing Steps 1 through 3 into a few lines each, and it handles the return-to-go bookkeeping, the sequence batching, and the rollout loop internally. The HuggingFace transformers library provides a pretrained Decision Transformer model class with a documented Gym rollout example, so a tuned agent is a model download and a generation loop away. The discipline the chapter teaches still governs the libraries: a sequence-model policy is only as good as the offline data it imitates and the return it is conditioned on, and a few-line fit call inherits every distribution-shift and coverage limit of the offline setting from Chapter 26. The library makes the architecture free; it does not make the data complete.
Stretch Goals
- Swap the causal Transformer backbone of Step 2 for a structured state-space model from Chapter 13 and compare memory and long-horizon behavior, making the bridge of Section 28.5 concrete rather than rhetorical.
- Sweep the target return across its full range and plot achieved return against requested return, exposing where the Decision Transformer of Section 28.2 can and cannot extrapolate beyond the best trajectory in its data, the central limit of return conditioning.
- Implement classifier-free return guidance for the diffusion planner sketch of Section 28.4 following the Decision Diffuser, and compare the plans it samples against the autoregressive rollouts of Step 3 on the same start states.
What's Next?
This chapter dissolved the boundary between forecasting and decision making by treating a trajectory as a sequence and generating good behavior the way a language model generates text, with the Part III architectures serving directly as policies. But a sequence model trained on logged trajectories imitates and conditions; it does not, on its own, hold an explicit model of how the world responds to its actions or search forward through imagined futures. Chapter 29: World Models and Planning takes that next step. It learns a model of the environment's dynamics, a world model, and uses it to imagine, plan, and search rather than only to autocomplete, connecting the latent dynamics of Part IV, the planning of classical control, and the sequence models of this chapter into agents that dream forward before they act. The Decision Transformer, the diffusion planner, and the conditioning ideas you built here are the generative substrate those world models plan in. The full path through Part VI and the rest of the book is laid out in the Table of Contents.
Bibliography & Further Reading
Founding Architectures
Chen, L., et al. "Decision Transformer: Reinforcement Learning via Sequence Modeling." NeurIPS, 2021. arXiv:2106.01345. arxiv.org/abs/2106.01345
The paper that recast reinforcement learning as conditional sequence modeling, feeding returns-to-go, states, and actions to a causal Transformer and conditioning generation on a target return, the founding architecture of Sections 28.1 and 28.2.
Janner, M., Li, Q., Levine, S. "Offline Reinforcement Learning as One Big Sequence Modeling Problem (Trajectory Transformer)." NeurIPS, 2021. arXiv:2106.02039. arxiv.org/abs/2106.02039
The paper that modeled the full joint distribution over a trajectory as a sequence and used beam search over it as a planner, the Trajectory Transformer that complements the Decision Transformer in Section 28.2.
Return and Goal Conditioning
Schmidhuber, J. "Reinforcement Learning Upside Down: Don't Predict Rewards, Just Map Them to Actions." 2019. arXiv:1912.02875. arxiv.org/abs/1912.02875
The paper introducing upside-down reinforcement learning, which replaces reward maximization with a command-conditioned mapping from desired outcomes to actions, the conceptual root of the return conditioning in Section 28.3.
Andrychowicz, M., et al. "Hindsight Experience Replay." NeurIPS, 2017. arXiv:1707.01495. arxiv.org/abs/1707.01495
The paper introducing hindsight experience replay, relabeling failed trajectories with the goals they actually reached so sparse-reward learning succeeds, the relabeling idea generalized by the goal-conditioned policies of Section 28.3.
Furuta, H., Matsuo, Y., Gu, S. S. "Generalized Decision Transformer for Offline Hindsight Information Matching." ICLR, 2022. arXiv:2111.10364. arxiv.org/abs/2111.10364
The paper that unified return- and goal-conditioning under hindsight information matching, showing the Decision Transformer as one instance of a broad family of conditioning signals, the generalization at the heart of Section 28.3.
Diffusion Planners
Janner, M., Du, Y., Tenenbaum, J. B., Levine, S. "Planning with Diffusion for Flexible Behavior Synthesis (Diffuser)." ICML, 2022. arXiv:2205.09991. arxiv.org/abs/2205.09991
The paper introducing Diffuser, which treats planning as denoising an entire trajectory at once into a feasible, high-return plan, importing the diffusion machinery of Chapter 17 into decision making, the foundation of Section 28.4.
Ajay, A., et al. "Is Conditional Generative Modeling All You Need for Decision-Making? (Decision Diffuser)." ICLR, 2023. arXiv:2211.15657. arxiv.org/abs/2211.15657
The paper introducing the Decision Diffuser, adding classifier-free return, constraint, and skill conditioning to trajectory diffusion, the conditioned generative planner of Section 28.4.
Generalist and Scaled Sequence Agents
Reed, S., et al. "A Generalist Agent (Gato)." Transactions on Machine Learning Research, 2022. arXiv:2205.06175. arxiv.org/abs/2205.06175
The paper training a single sequence model across control, vision, and language as one token stream, the clearest demonstration that a Part III architecture can serve as a universal policy, central to the bridge of Section 28.5.
Lee, K.-H., et al. "Multi-Game Decision Transformers." NeurIPS, 2022. arXiv:2205.15241. arxiv.org/abs/2205.15241
The paper scaling a single Decision Transformer to play many Atari games at once, evidence that the sequence-modeling reformulation scales like a language model, the generalist agent of Section 28.5.
Zheng, Q., Zhang, A., Grover, A. "Online Decision Transformer." ICML, 2022. arXiv:2202.05607. arxiv.org/abs/2202.05607
The paper extending the Decision Transformer from purely offline pretraining to online finetuning with an exploration objective, bridging the sequence-model policy of Section 28.2 back to online interaction.
Tools and Libraries
Seno, T., Imai, M. "d3rlpy: An Offline Deep Reinforcement Learning Library." Journal of Machine Learning Research, 23(315), 2022. github.com/takuseno/d3rlpy
The offline reinforcement-learning library that ships the Decision Transformer alongside value-based offline methods behind a uniform fit and predict API, the off-the-shelf agent that collapses the training steps of the lab into a few lines.
HuggingFace. "Decision Transformer Model Documentation." Transformers Library, 2024. huggingface.co/docs/transformers/model_doc/decision_transformer
The reference implementation and documented Gym rollout example for the Decision Transformer in the Transformers library, the pretrained sequence-model policy the lab's library shortcut builds on.