"They built me a model for every question in this book and then asked me to use all of them at once, in order, without being told which one. A forecaster lived in my left hand and a planner in my right, a memory stretched behind me longer than I could read in one breath, and a stream of fresh observations arrived faster than I could decide what to forget. The trouble was never any single prediction; the trouble was the loop, perceiving while remembering while planning while acting, each step quietly rewriting the world the next step would meet. I learned that an agent is not a bigger model but a smaller commitment renewed forever: choose, watch what the choice does, blame the right earlier choice when it goes wrong, and somehow keep a thread of self running across ten thousand timesteps. The hardest question turned out not to be what should I do next but did the thing I did a hundred steps ago actually help, and on most days no one, including me, could say."
An Agent Still Assigning Credit for Last Tuesday
Chapter Overview
This is the chapter where the book becomes a system. Every part until now contributed a capability in isolation: forecasters that say what comes next, probabilistic models that quantify how unsure they are, representations that compress a stream into something usable, policies that choose actions, and the reasoning of Chapter 30 that asks why and what-if. A temporal agent is the running program that unifies them. It sits inside a perceive-reason-decide-act loop, taking in a stream of observations, updating an internal state, choosing an action, and then meeting the consequences of that action as its next observation. The agent is not a larger model; it is an architecture that orchestrates models over time, and the whole of this chapter is about what that orchestration requires.
Four hard problems organize the agent. The first is memory: a single forward pass cannot hold a long-running task, so the agent needs a hierarchy, a fast working memory for the current step, a longer episodic record of what it has done, and a retrievable long-term store it can search when the present moment is not enough. The second is planning and tool use: rather than emit an answer in one shot, a capable agent decomposes a goal into steps, decides which external tool each step needs, and calls it, and one of those tools can be a model from earlier in this very book, a forecaster invoked to estimate what a series will do, an uncertainty estimator queried before committing to a risky action, a causal estimator from Chapter 30 consulted to predict an intervention's effect. The third is long-horizon credit assignment: when a reward arrives only after many steps, the agent must trace it back to the decisions that earned it, a problem that grows harder the longer the horizon and that defeats naive learning entirely. The fourth, and the one with the least settled answer, is evaluation: an agent that acts over many steps in an open environment is far harder to score than a forecaster on a held-out window, and knowing whether an agent is actually good remains a frontier.
The agent's debt to Part VI is direct. A policy learned in Part VI is the agent's decision rule, the mapping from situation to action that the perceive-reason-decide-act loop executes at its core, and the value functions and returns of those chapters are exactly the signal the credit-assignment machinery here tries to attribute. What the agent adds on top is everything that surrounds the policy: the memory that gives it a usable state when the raw observation is partial, the tools that extend its reach beyond what its weights know, and the reasoning that lets it plan rather than merely react. The agent is a policy that has been given a memory, a toolbox, and a reason to deliberate.
One earlier idea returns here transformed. Chapter 23 framed memory as a belief state, the agent's probability distribution over the hidden world that it carries forward and updates with each observation, the optimal compression of history under partial observability. The memory systems of this chapter are the practical, learned, and retrieval-augmented descendants of that idea: a vector store of past experience is a coarse, searchable belief about what the agent has seen, and the discipline of deciding what to write, what to retrieve, and what to forget is the engineering form of maintaining a belief state when the history is too long to hold in full. The agent remembers because, like the belief-state filter of Chapter 23, it cannot act well on the present observation alone.
Prerequisites
This chapter assumes the sequential decision making of Part VI, beginning with Chapter 22: Markov Decision Processes, because an agent's decision loop is a policy acting in an environment and its long-horizon learning is the credit-assignment problem those chapters formalize, so the perceive-decide-act core here is Part VI made into a running program. It builds on Chapter 12: Attention and Transformers, since the reasoning, planning, and tool-calling behavior of modern agents is realized on transformer backbones, and attention over a context window is the mechanism that lets an agent condition its next action on a retrieved memory or a tool's result. And it draws on Chapter 19: Probabilistic Forecasting and Uncertainty Quantification, because an agent that calls a forecaster as a tool must know how much to trust the number it gets back, and acting well under partial information means reasoning about uncertainty rather than ignoring it. Readers wanting to refresh the policy-and-value vocabulary of reinforcement learning, the attention mechanism, or the calibrated-uncertainty ideas these agents lean on will find the refreshers through the Table of Contents.
If you keep one idea from this chapter, keep this: an agent is not a bigger model but a perceive-reason-decide-act loop wrapped around a policy, given a memory hierarchy to hold the long-running task, tools (including this book's own forecasters and estimators) to extend its reach, and a way to assign credit across a long horizon, and the unsolved problem at the center is knowing whether any of it actually works. Memory turns a stream too long to hold into a searchable belief about the past; planning and tool use turn a single forward pass into a decomposed, deliberate sequence of grounded steps; long-horizon credit assignment traces a delayed reward back to the decisions that earned it; and agent evaluation, measuring success rate and cost across seeds and tasks rather than error on a held-out window, is the frontier that decides whether an agent is good or merely impressive.
Chapter Roadmap
- 31.1 Agent Architectures The anatomy of a temporal agent: the perceive-reason-decide-act loop, the controller that orchestrates models rather than replacing them, reactive versus deliberative designs, and the ReAct pattern that interleaves reasoning traces with actions on a stream of observations.
- 31.2 Memory Systems Giving the agent a usable past: working, episodic, and long-term memory hierarchies, vector stores and retrieval-augmented context, what to write and what to forget, and memory as the practical descendant of the belief state of Chapter 23.
- 31.3 Planning and Tool Use From one-shot answers to deliberate action: goal decomposition, chain-of-thought and tree-of-thoughts reasoning, calling external tools and APIs, and treating this book's own forecasters, uncertainty estimators, and causal models as tools the agent can invoke.
- 31.4 Long-Horizon Tasks and Credit Assignment Learning from delayed and sparse reward: why long horizons defeat naive credit assignment, return decomposition and reward redistribution, reflection and self-revision across attempts, and tracing a final outcome back to the decisions that produced it.
- 31.5 Agent Evaluation The hard problem of knowing whether an agent is good: success rate and cost as primary metrics, evaluation across seeds and task distributions, agent benchmarks and their pitfalls, and why measuring an acting system is far harder than scoring a forecaster.
Once you have worked through the five sections, the Hands-On Lab below assembles them into one working agent. Each lab step maps to a section, so the lab doubles as a review: by the time you finish you will have built the ReAct-style loop of Section 31.1, given it the vector memory of Section 31.2, equipped it with the forecasting tool of Section 31.3, run it on a multi-step task whose reward arrives only at the end as in Section 31.4, and scored it with the success-rate-and-cost protocol of Section 31.5, turning a pile of separate capabilities into a single system you can run, measure, and trust.
Hands-On Lab: Build a Temporal Agent
Objective
Build a working temporal agent from the parts this chapter assembles, then evaluate it honestly. You will construct a ReAct-style loop that interleaves a reasoning trace with actions on a stream, the architecture of Section 31.1, and give it a vector memory it writes to and retrieves from, the memory system of Section 31.2. You will equip the agent with a forecasting tool, a model from earlier in this book that the agent calls when it needs to know what a series will do, the tool-use discipline of Section 31.3, so the agent reasons about the future rather than only the present. You will run it on a multi-step task whose reward arrives only at the end, exposing the long-horizon credit-assignment problem of Section 31.4, and finally you will evaluate it the way agents must be evaluated, across many seeds, reporting a success rate and a cost rather than a single error number, the protocol of Section 31.5. The single thread is that an agent is a loop, not a number: building one is straightforward, but knowing whether it works is the real engineering, and that knowledge comes only from running it many times and measuring what it actually achieved.
What You'll Practice
- Implementing a ReAct-style perceive-reason-decide-act loop that interleaves reasoning and action, the agent architecture of Section 31.1.
- Adding a vector memory the agent writes to and retrieves from, the memory hierarchy of Section 31.2.
- Wrapping a forecaster from earlier in the book as a callable tool and invoking it mid-task, the tool use of Section 31.3.
- Running a multi-step task with delayed reward and observing the credit-assignment difficulty of Section 31.4.
- Evaluating the agent across seeds with success rate and cost rather than a single score, the protocol of Section 31.5.
Setup
You need a Python environment with numpy, a vector index such as faiss or a simple cosine-similarity store for the memory of Section 31.2, and any forecaster you built earlier in the book (a small ARIMA, a neural sequence model, or even a naive baseline) to register as the agent's tool of Section 31.3. A lightweight language-model backend drives the reasoning trace, but the loop below works with any policy that maps an observation and retrieved memory to a chosen action and tool call. The one discipline that governs the whole lab is to keep the loop honest: every step the agent perceives, consults memory, reasons, picks one action or tool call, and only then sees the result, never peeking past the action it has committed to. The code below is the bare skeleton of that loop, the perceive-reason-decide-act cycle of Section 31.1 with a memory read and a tool call wired in.
def run_agent(env, policy, memory, tools, max_steps, seed):
"""One episode of a ReAct-style temporal agent. Returns (success, cost)."""
obs = env.reset(seed=seed)
cost = 0
for _ in range(max_steps):
recalled = memory.retrieve(obs) # consult long-term store
thought, action, tool = policy(obs, recalled) # reason, then decide
if tool is not None:
obs_aug, tcost = tools[tool](obs) # call a tool (e.g. forecaster)
cost += tcost # tool calls are not free
obs = obs_aug
memory.write(obs, thought, action) # remember this step
obs, reward, done = env.step(action) # act, then meet the consequence
cost += 1
if done:
return reward > 0, cost # success and accumulated cost
return False, cost # ran out of steps
Steps
Step 1: Build the ReAct loop
Implement the perceive-reason-decide-act loop of Section 31.1 using the skeleton above, with a policy that emits a short reasoning trace before each action. Run it first with no memory and no tools on a simple task so the bare loop is correct, and confirm the agent alternates a thought and an action each step rather than answering in one shot.
Step 2: Add the vector memory
Give the agent the memory system of Section 31.2: embed each step's observation and action, store them in a vector index, and retrieve the most similar past entries to condition the next decision. Test that the agent recalls a relevant earlier step on a task that requires remembering something seen many steps before, the working-versus-long-term distinction made concrete.
Step 3: Register the forecasting tool
Wrap a forecaster from earlier in the book as a callable tool the agent can invoke, the tool use of Section 31.3. On a step where the agent must anticipate a series, have the policy choose to call the forecaster, fold its prediction into the observation, and decide using the forecast. Note that the agent is now reasoning about the future with a model, not guessing, and that the tool call adds to the episode cost.
Step 4: Run a long-horizon task
Give the agent a multi-step task whose reward arrives only on the final step, the delayed-reward setting of Section 31.4. Run several episodes and observe how hard it is to tell which earlier decision earned or lost the final reward, the credit-assignment problem in the flesh. Optionally add a reflection step in which the agent reviews a failed episode from memory before retrying, and see whether the second attempt improves.
Step 5: Evaluate across seeds
Run the full agent over many seeds and task instances, recording success and cost per episode, then report the success rate and the mean cost with their spread, the evaluation protocol of Section 31.5. Resist summarizing the agent with a single lucky run: a temporal agent is good only if it succeeds reliably and at acceptable cost across the distribution, and the variance across seeds is part of the result, not noise to be hidden.
Expected Output
The lab produces one agent measured honestly. Step 1 yields a loop whose transcript alternates reasoning and action each step, the visible signature of a ReAct architecture rather than a one-shot model. Step 2 shows the agent retrieving a relevant earlier entry and acting on it, memory turning a stream too long to hold into a usable past. Step 3 shows a tool call to the forecaster changing the agent's decision, the moment one of this book's own models becomes an instrument the agent wields rather than the agent itself. Step 4 makes the long-horizon difficulty tangible: a final reward that is hard to attribute to any single earlier step, and, if reflection was added, a measurable lift on retry. Step 5 delivers the only number that matters, a success rate and a cost with their spread across seeds, not a single run. The reader finishes able to build a temporal agent and, more importantly, able to say whether it works, which is the harder and more valuable skill, because an agent that succeeds once is a demo and an agent that succeeds reliably across seeds at bounded cost is a system.
The bare loop of the setup is worth writing once so the perceive-reason-decide-act cycle and the cost of every tool call stop being abstract, but you should rarely hand-roll an agent in production. Frameworks such as LangGraph model the agent as a stateful graph of nodes, giving you the loop, the memory checkpoints, and the tool-calling plumbing of Sections 31.1 through 31.3 as a few lines instead of a hand-written controller, while letting you keep the reasoning policy you choose. Memory-management systems in the MemGPT and Letta lineage handle the working-versus-long-term hierarchy of Section 31.2, paging context in and out so the agent appears to have a memory far larger than its context window, the engineering of the belief state automated. Agent benchmark suites such as AgentBench supply the multi-task, multi-seed evaluation harness of Section 31.5 so you measure success rate and cost on standardized tasks rather than improvising a metric. The discipline the chapter teaches still governs the frameworks: a graph makes the loop convenient but not correct, a memory system stores what you tell it to but cannot decide what is worth remembering, and a benchmark reports a number that is only as meaningful as its task distribution. The library runs the agent; it does not certify that the agent is good, and whether it is good is the question this chapter exists to make you ask.
Stretch Goals
- Add a second tool, an uncertainty estimator from Chapter 19, that the agent calls before a risky action, and measure whether reasoning about its own uncertainty before acting raises the success rate of Section 31.5 at the price of extra tool cost.
- Replace the flat vector store of Section 31.2 with a hierarchical memory that summarizes old episodes into compact notes, and study how aggressively the agent can forget detail before its long-horizon performance degrades.
- Implement a reflection loop in the spirit of Section 31.4 in which the agent, after a failed episode, writes a natural-language lesson into memory and retrieves it on the next attempt, then measure the improvement across attempts as a primitive form of self-driven credit assignment.
What's Next?
This chapter built an agent that perceives, remembers, plans with tools, and learns across a long horizon, but it lived almost entirely in time. Chapter 32: Spatio-Temporal Intelligence adds the missing dimension, taking the temporal machinery of the whole book into systems that must reason over space as well, traffic that flows across a road network, disease that spreads across a map, sensors distributed over a factory floor, where the value at one place and time depends on neighbors in both. The memory and tool-use patterns built here carry forward, but the agent now perceives a field rather than a series, and the structure of space becomes as important as the arrow of time. The full path through Part VII and the rest of the book is laid out in the Table of Contents.
Bibliography & Further Reading
Agent Foundations and Architectures
Russell, S., Norvig, P. "Artificial Intelligence: A Modern Approach." 4th ed., Pearson, 2020. aima.cs.berkeley.edu
The standard text whose agent abstraction, a system that perceives an environment and acts upon it through an agent function, is the conceptual frame for the perceive-reason-decide-act loop of Section 31.1.
Wang, L., et al. "A Survey on Large Language Model based Autonomous Agents." 2023. arXiv:2308.11432. arxiv.org/abs/2308.11432
A broad survey of LLM-based autonomous agents organized around profile, memory, planning, and action, the map of the design space that this chapter's sections traverse.
Reasoning, Planning, and Tool Use
Yao, S., et al. "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR, 2023. arXiv:2210.03629. arxiv.org/abs/2210.03629
The paper introducing the ReAct pattern, interleaving reasoning traces with actions on an environment, the agent architecture of Section 31.1 and the backbone of the lab's loop.
Wei, J., et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS, 2022. arXiv:2201.11903. arxiv.org/abs/2201.11903
The paper showing that eliciting intermediate reasoning steps improves multi-step problem solving, the chain-of-thought decomposition that underlies the planning of Section 31.3.
Yao, S., et al. "Tree of Thoughts: Deliberate Problem Solving with Large Language Models." NeurIPS, 2023. arXiv:2305.10601. arxiv.org/abs/2305.10601
The paper generalizing chain-of-thought to a searchable tree of reasoning paths with lookahead and backtracking, the deliberative planning extreme of Section 31.3.
Schick, T., et al. "Toolformer: Language Models Can Teach Themselves to Use Tools." NeurIPS, 2023. arXiv:2302.04761. arxiv.org/abs/2302.04761
The paper showing a model learning to call external tools and APIs through self-supervision, the tool-use capability that lets the agent of Section 31.3 invoke this book's own forecasters and estimators.
Memory and Generative Agents
Park, J. S., et al. "Generative Agents: Interactive Simulacra of Human Behavior." UIST, 2023. arXiv:2304.03442. arxiv.org/abs/2304.03442
The paper building agents with a memory stream, retrieval, reflection, and planning over long interaction horizons, a concrete realization of the memory hierarchy of Section 31.2.
Packer, C., et al. "MemGPT: Towards LLMs as Operating Systems (Letta)." 2023. arXiv:2310.08560. arxiv.org/abs/2310.08560
The paper introducing a paged memory hierarchy that swaps context in and out of a bounded window, the working-versus-long-term memory engineering of Section 31.2 and the basis of the Letta system.
Self-Reflection and Credit Assignment
Shinn, N., et al. "Reflexion: Language Agents with Verbal Reinforcement Learning." NeurIPS, 2023. arXiv:2303.11366. arxiv.org/abs/2303.11366
The paper in which an agent reflects in language on a failed attempt and stores the lesson for the next try, the self-revision form of credit assignment in Section 31.4.
Arjona-Medina, J. A., et al. "RUDDER: Return Decomposition for Delayed Rewards." NeurIPS, 2019. arXiv:1806.07857. arxiv.org/abs/1806.07857
The paper introducing return decomposition and reward redistribution to attack delayed-reward credit assignment, the long-horizon attribution problem at the heart of Section 31.4.
Evaluation and Tooling
Liu, X., et al. "AgentBench: Evaluating LLMs as Agents." ICLR, 2024. arXiv:2308.03688. arxiv.org/abs/2308.03688
The benchmark suite evaluating agents across diverse interactive environments with success-based metrics, the multi-task evaluation discipline of Section 31.5 and the lab's scoring step.
LangChain. "LangGraph: Building Stateful, Multi-Actor Agent Applications." Documentation. langchain-ai.github.io/langgraph
The framework modeling an agent as a stateful graph of nodes with built-in memory checkpoints and tool-calling, the off-the-shelf engine for the loop of Sections 31.1 through 31.3 used in the lab's Right Tool note.