Agent Architectures

"I perceive, then I reason, then I decide, then I act, and then, because no one wrote a stopping condition, I perceive again. I have been very productive this afternoon. I have also been doing exactly the same thing for six hours."

An Agent That Perceives, Plans, and Acts, and Occasionally Loops Forever

Every system in this book has been, secretly, a fragment of an agent. Back in Section 1.3 we drew the perceive-reason-decide-act loop as the defining shape of temporal intelligence and promised it would return as a running program; this is where that promise is kept. In Chapter 22 we formalized the agent-environment loop of a Markov decision process: an agent observes a state, selects an action, and the environment returns a reward and a next state, around and around. An agent architecture is what you get when you stop treating that loop as a mathematical abstraction and start treating it as code that runs, with a perception module, a reasoning core, a decision rule, an action interface, and a clock. We use the unified notation of Appendix A: $o_t$ for the observation at time $t$, $s_t$ for the agent's internal state, $a_t$ for the action, and $\pi$ for the policy that maps state to action.

Why does this deserve its own chapter rather than a footnote saying "now put the pieces together"? Because the act of closing the loop introduces problems that none of the individual pieces had. A forecaster that runs once and stops cannot loop forever; an agent that runs continuously can. A policy evaluated on a fixed dataset cannot accumulate its own errors; an agent that feeds its own actions back into its own observations does exactly that, compounding small mistakes over a long horizon. The statefulness that lets an agent remember and improve is the same statefulness that lets a bug in one step silently corrupt every later step. Composition is not free: the loop is more than the sum of its modules, and the extra is precisely what an agent architecture must manage.

The competencies this section installs are four. To state precisely what a temporal AI agent is, as a perceive-reason-decide-act cycle carrying state across time, and to connect it to the loops of Chapter 1 and Chapter 22. To place any agent on the reactive-deliberative-hybrid spectrum and name its architecture family. To read the modern LLM-agent stack as a policy core surrounded by memory, planning, and tools, and to locate the temporal dimension in each. And to implement a minimal ReAct loop from scratch, then recognize the same loop inside a production framework. These are the load-bearing skills for the memory systems of Section 31.2 and the planning of Section 31.3.

1. What a Temporal AI Agent Is: The Loop, Running Beginner

A temporal AI agent is a system that runs a perceive-reason-decide-act loop over time. Read slowly, that sentence has four verbs and one preposition, and every word is load-bearing. Perceive: at each step the agent takes in an observation $o_t$ of its environment. Reason: it updates an internal state $s_t$ that summarizes what it has seen and inferred. Decide: it chooses an action $a_t$ from $s_t$ according to a policy. Act: it executes $a_t$, changing the environment. And over time: the whole thing repeats, the environment returns a new observation $o_{t+1}$, and the state $s_t$ carries forward, so the agent is not a function evaluated once but a process unfolding across a horizon. That carried state, the thing that survives from one step to the next, is what makes an agent temporal rather than a stateless input-output map.

This is not a new loop. It is exactly the loop Section 1.3 placed at the foundation of the entire book, where we argued that intelligence becomes temporal precisely when a system must perceive a stream, maintain belief across time, and act with consequences that feed back. There we drew it as a diagram and promised a running implementation in Part VII; an agent is that implementation. And it is the same loop, viewed from the decision side, that Chapter 22 formalized as the agent-environment interaction of a Markov decision process. The MDP gave us the mathematics of the loop, transition kernel, reward, policy, value, and discount. An agent architecture gives us the software of the loop, the modules and the control flow that turn $\pi$ from a table into a program. The agent is the synthesis: the perception-and-reasoning machinery of Parts I through V supplies the state $s_t$; the policy-and-value machinery of Part VI supplies the action $a_t$; the architecture wires them into one cycle.

The single structural fact that separates an agent from a forecaster is the feedback edge, the orange path in Figure 31.1.1. A forecaster reads a series and emits a prediction; nothing it outputs comes back as input. An agent's action enters the world and the world's response becomes the agent's next observation, so the agent is coupled to its own consequences. This closure is the source of everything that is hard and everything that is powerful about agents. It is powerful because the agent can steer the future toward its goal rather than merely predict it. It is hard because the agent can also steer the future into a state its perception misreads, then act on that misreading, and there is no external supervisor in the loop to catch the drift. We unpack both the power and the hazard in the sections that follow.

2. Architecture Families: Reactive, Deliberative, Hybrid Intermediate

The loop of subsection one admits many implementations, and the history of AI is in large part a history of arguments about how much reasoning should sit between perception and action. Three families recur, and the modern LLM agent inherits from all of them, so it pays to name them precisely. At one extreme is the reactive agent, which maps observation to action with little or no internal deliberation: $a_t = \pi(o_t)$, a fast reflex. At the other is the deliberative agent, which builds an internal model of the world and plans over it before acting: it senses, constructs a representation, searches for a plan, then executes, the classic sense-plan-act pipeline of early symbolic robotics. In between sits the hybrid agent, which keeps fast reactive behaviors for the moment-to-moment while a slower deliberative layer sets goals and revises plans, the architecture that actually drives most real systems.

The reactive tradition's sharpest expression is Brooks's subsumption architecture from 1980s robotics, which discarded the central world model entirely and layered simple behaviors (avoid-obstacle, wander, explore) so that higher layers subsume lower ones (a higher behavior can suppress or override the output of a lower one, so obstacle-avoidance can veto wander), producing competent behavior with almost no representation. The deliberative tradition's sharpest expression is the sense-plan-act robot that builds a map, plans a path, and executes it, powerful when the world is well modeled and slow when it is not. Reinforcement learning, the whole of Part VI, produces agents that are mostly reactive at deployment: a trained policy $\pi(a \mid s)$ is a fast mapping from state to action distribution, the deliberation having been amortized into the weights during training. And the LLM-based agent, the subject of the rest of this chapter, is hybrid by construction: the language model is a fast reactive next-token policy, but wrapped in a loop that lets it reason in natural language and call tools, it deliberates. The line of work that opened this door is ReAct (Yao et al., 2022), which interleaves reasoning traces and actions so the model thinks in words, acts with a tool, observes the result, and thinks again.

The loop body of any of these families can be written as a single piece of pseudocode, and writing it once makes the family differences sharp. The agent maintains a state, repeatedly perceives, updates, decides, and acts, until a goal or budget halts it:

The families differ only in two slots. A reactive agent makes UpdateState trivial (the state is just the latest observation) and $\pi$ a fast learned or hand-coded reflex. A deliberative agent makes UpdateState build a rich world model and $\pi$ an explicit planner that searches that model, the planning of Section 31.3. A hybrid agent runs a fast $\pi$ inside the loop while a slower process intermittently revises the goal or the plan. Everything elaborate in agent design, the memory of Section 31.2, the planners of Section 31.3, the tool interfaces, is an elaboration of those two slots inside this one skeleton.

3. The Modern LLM-Agent Stack: Policy, Memory, Planning, Tools Intermediate

The agent architecture that dominates practice in 2026 has a recognizable four-part anatomy, and each part maps onto a slot of the loop from subsection two. At the center is a policy core, almost always a large language model, which plays the role of $\pi$: given the current state rendered as text, it produces the next thought or action. Around that core sit three faculties that turn a stateless next-token predictor into a temporal agent. Memory supplies state that persists beyond the model's context window, the subject of Section 31.2. Planning supplies the deliberative layer that decomposes a goal into steps, the subject of Section 31.3. And tools supply the action interface, the means by which the agent's decisions actually touch the world: a search API, a calculator, a code interpreter, a database query.

A bare language model is a poor agent on its own. It is a stateless function: it maps a context to a next token and forgets everything the instant the context scrolls past. It cannot persist a fact across a long task (no memory), it cannot reliably hold a multi-step plan in its head while executing it (no planning scaffold), and it cannot do anything but emit text (no tools). The agent stack is the set of structures that repair each deficit: memory makes it stateful, planning makes it deliberative, tools make it effective. Figure 31.1.2 draws the stack and labels which loop slot each faculty serves.

The temporal dimension is what distinguishes a temporal agent from a one-shot question answerer, and it shows up in two distinctions worth naming. First, the agent's state evolves over time: the context the LLM sees at step $t$ is a function of everything that happened in steps $1$ through $t-1$, mediated by memory. A temporal agent is, formally, a policy over histories, $a_t = \pi(o_{1:t}, a_{1:t-1})$, and memory is the mechanism that compresses that growing history into a bounded state, exactly the role the hidden state played for the RNN in Chapter 10 and the belief state played for the POMDP agent in Chapter 23. Second, agents run in one of two temporal regimes: episodic, where each task is a self-contained episode that resets the state at the end (answer this question, then forget), and continuing, where the agent persists indefinitely and its state accumulates across tasks (a long-running assistant or monitoring agent). The episodic-versus-continuing distinction, first met for MDPs in Chapter 22, decides how memory must behave: an episodic agent can clear memory at episode end, a continuing agent must manage memory growth forever, the central design pressure of Section 31.2.

4. Failure Modes: Loops, Drift, and Statefulness Bugs Advanced

Closing the loop creates failure modes that no open-loop model has, and a working knowledge of them is what separates an agent that ships from one that quietly burns its budget. Three dominate. The first is the infinite loop: an agent with no reliable stopping condition, or one whose goal-check never fires, perceives and acts forever, repeating the same action or oscillating between two, accomplishing nothing while consuming tokens or actuator cycles. The epigraph's six-hour afternoon is this failure. It arises because the loop of subsection one terminates on GoalMet(s) or a budget, and if the goal is mis-specified, never satisfiable, or never detected, only the budget saves you, and an agent without a hard budget cap has no floor.

The second failure is error accumulation over long horizons, the agent-specific form of the compounding error we first met for autoregressive forecasters in Chapter 5 and for imitation learning in Chapter 27. Because the agent feeds its own actions back into its own observations, a small mistake at step $t$ shifts the state it observes at step $t+1$ into territory it handles slightly worse, producing a larger mistake at $t+2$, and so on. The errors are not independent; they correlate and compound. If each step has even a modest probability $p$ of a consequential error and errors are roughly absorbing (one bad step tends to beget another), the probability that an $H$-step rollout stays on track falls off geometrically, $(1-p)^H$, so long horizons are intrinsically fragile. This is why agent reliability is so much harder than single-call reliability: a model that is right 95 percent of the time per step is right end-to-end on a twenty-step task only about a third of the time.

The third failure is the statefulness bug, the quiet one. Because an agent carries state across steps, a defect in how that state is updated, a memory that is appended to but never cleared, a stale observation that lingers, a variable shared across episodes that should have reset, corrupts not one step but every subsequent step, and the symptom appears far from the cause. An episodic agent that forgets to reset its memory at episode boundaries leaks context from one task into the next; a continuing agent whose memory grows without bound eventually blows its context window and silently drops the early state. These bugs are insidious precisely because the loop hides them: the same code runs every step, so a fault that depends on accumulated state only manifests after enough steps, and reproducing it requires replaying the whole history. The discipline that prevents them is to make state transitions explicit and to test the agent over long rollouts, not single steps.

5. Worked Example: A Minimal ReAct Agent, From Scratch and From a Library Advanced

We now make the loop executable. The plan mirrors the rest of the book: build a minimal ReAct-style agent from scratch so every part of the loop is visible, run it on a toy task, then collapse the same agent into a library skeleton and state the line-count reduction. The toy task is deliberately tiny so the loop, not the task, is the lesson: the agent must answer a multi-step arithmetic-and-lookup question that it cannot answer in one shot, forcing it to reason, act with a tool, observe, and reason again. Code 31.1.1 is the from-scratch agent: a perceive-think-act loop with two tools and a hard step budget, with a tiny rule-based stand-in for the LLM policy so the example runs with no API key and is fully deterministic.

import re

# Two tools the agent can call. Each maps a string argument to a string observation.
def calc(expr):
    """A calculator tool: evaluate a simple arithmetic expression."""
    try:
        return str(eval(expr, {"__builtins__": {}}, {}))   # sandboxed eval, digits/ops only
    except Exception as e:
        return f"error: {e}"

POPULATION = {"france": 68, "germany": 84, "spain": 48}    # millions, a lookup "database"
def lookup(country):
    """A retrieval tool: look up a country's population in millions."""
    return str(POPULATION.get(country.strip().lower(), "unknown"))

TOOLS = {"calc": calc, "lookup": lookup}

def policy(observation, scratchpad):
    """Stand-in for an LLM: given the running scratchpad, emit the next Thought+Action.
    A real agent calls an LLM here; we hand-code the reasoning so the loop is deterministic."""
    text = scratchpad.lower()
    if "lookup[france]" not in text:                       # step 1: get France's population
        return "Thought: I need France's population.\nAction: lookup[France]"
    if "lookup[germany]" not in text:                      # step 2: get Germany's population
        return "Thought: Now I need Germany's population.\nAction: lookup[Germany]"
    if "calc[" not in text:                                # step 3: add the two values
        nums = re.findall(r"Observation: (\d+)", scratchpad)
        return f"Thought: I will add them.\nAction: calc[{nums[0]}+{nums[1]}]"
    total = re.findall(r"Observation: (\d+)", scratchpad)[-1]   # step 4: report the answer
    return f"Thought: I have the total.\nFinish: {total} million"

def react_agent(question, max_steps=8):
    """A minimal ReAct loop: observe -> think -> act with a tool -> observe, until Finish."""
    scratchpad = f"Question: {question}\n"
    for step in range(max_steps):                          # the HARD step budget bounds the loop
        out = policy(None, scratchpad)                     # reason: LLM proposes Thought+Action
        scratchpad += out + "\n"
        if out.startswith("Thought") and "Finish:" in out:  # decision to stop
            return out.split("Finish:")[1].strip(), step + 1, scratchpad
        m = re.search(r"Action: (\w+)\[(.*?)\]", out)      # parse the chosen tool call
        if not m:
            return "no action parsed", step + 1, scratchpad
        tool, arg = m.group(1), m.group(2)
        obs = TOOLS[tool](arg)                             # act: run the tool in the world
        scratchpad += f"Observation: {obs}\n"              # perceive: fold the result back in
    return "step budget exhausted", max_steps, scratchpad  # the loop NEVER runs unbounded

answer, steps, trace = react_agent("What is the combined population of France and Germany?")
print(f"answer = {answer}   (solved in {steps} reasoning steps)")

answer = 152 million   (solved in 4 reasoning steps)

To make the reactive baseline concrete and motivate the numeric comparison, Code 31.1.2 is a one-shot reactive agent that maps the question straight to an answer with no loop and no tools. On a task that needs composition, it has no mechanism to retrieve and combine, so it fails; the contrast is the point.

def reactive_agent(question):
    """A purely reactive baseline: one observation in, one action out, no loop, no tools."""
    # With no tool-use loop, a reflex policy can only answer from what it already knows.
    # The composite question requires two lookups and an addition it cannot perform in one shot.
    return "I cannot compute that in a single step", 1     # 1 "step", and it is wrong

ans_r, steps_r = reactive_agent("What is the combined population of France and Germany?")
print(f"reactive baseline: {ans_r}   (steps = {steps_r}, correct = False)")
print(f"react agent:       152 million   (steps = {steps}, correct = True)")
print(f"steps-to-solve: reactive never solves; react solves in {steps} steps")

reactive baseline: I cannot compute that in a single step   (steps = 1, correct = False)
react agent:       152 million   (steps = 4, correct = True)
steps-to-solve: reactive never solves; react solves in 4 steps

Now the library pair. A production framework supplies the loop, the tool-dispatch, the scratchpad management, the step budget, and the stopping logic, so the same agent reduces to declaring the tools and the goal. Code 31.1.3 is a library-style skeleton in the shape used by LangGraph, LlamaIndex, and AutoGen: register tools, hand the framework an LLM, and call run.

# Library skeleton (LangGraph / LlamaIndex / AutoGen-style). Pseudocode-level API.
from agent_framework import Agent, tool          # any modern agent framework

@tool
def lookup(country: str) -> str:
    "Look up a country's population in millions."
    return str({"france": 68, "germany": 84, "spain": 48}.get(country.lower(), "unknown"))

@tool
def calc(expr: str) -> str:
    "Evaluate a simple arithmetic expression."
    return str(eval(expr, {"__builtins__": {}}, {}))

agent = Agent(llm="your-model", tools=[lookup, calc], max_steps=8)   # the loop is built in
answer = agent.run("What is the combined population of France and Germany?")
print(answer)        # -> "152 million"; the framework ran the ReAct loop internally

The line-count reduction tells the story of the abstraction. The from-scratch agent of Code 31.1.1 was roughly 45 lines: the loop, the scratchpad, the action parser, the tool table, the budget, and a hand-coded policy. The library skeleton of Code 31.1.3 is about 12 lines, a better than three-to-one reduction, and the lines that vanished are exactly the loop machinery, the tool-dispatch, and the stopping logic that every agent needs and no one should rewrite. What the framework handles internally is precisely the content of this section: the loop of subsection one, the hybrid reason-act control flow of subsection two, the tool interface of subsection three, and crucially the step budget and termination of subsection four. Reading the from-scratch version first is what lets you debug the library version when its hidden loop misbehaves, because the failure modes of subsection four live inside that hidden loop.

Step back and read what the code establishes together. Code 31.1.1 made the four-verb loop of subsection one literal and bounded it with the step budget that prevents subsection four's infinite loop. Code 31.1.2 showed the reactive baseline failing the same task for lack of a loop, quantifying what deliberation buys. Code 31.1.3 collapsed the whole thing into a library call, exposing the loop as exactly the machinery a framework provides. The payoff is that an agent is not a black box: it is the loop of Figure 31.1.1, you can write it by hand, a framework writes it for you, and its reliability is governed by the step budget, the memory of Section 31.2, and the planning of Section 31.3 that the rest of this chapter builds.

Exercises

Three exercises, one of each type. Solutions to selected exercises appear in Appendix G.