Foundation Agents

The unification is not merely notational tidiness; it has a consequence the rest of the section trades on. Because task definition lives in the context $c_{\mathcal{T}}$ and not in the weights $\theta$, a foundation agent can in principle take on a task it never saw in training, simply by being given the right conditioning, the way a foundation forecaster of Chapter 15 forecasts a series it never saw. The generality is bought at pretraining time and spent, almost for free, at inference time. That is the promise. The open problems of subsection four are all about how, and how far, the promise actually holds.

2. The Ingredients: Backbone, In-Context Learning, Memory and Tools, Adaptation Beginner

A foundation agent has four ingredients, and naming them precisely both demystifies the object and gives us the parts list for the from-scratch build of subsection five. Each ingredient is a chapter of this book seen from the agent's vantage.

The first ingredient is a large pretrained sequence backbone. This is the engine: a Transformer (Chapter 12) or a structured state-space model (Chapter 13), pretrained on a corpus large and diverse enough that its next-token distribution encodes broadly useful temporal structure. The backbone supplies the agent's perception (it encodes observations into representations) and its policy (its output head emits actions). Everything else is built around it. The decisive property inherited from Chapter 15 is that the backbone is frozen at deployment: it is a fixed function whose behavior is steered by what enters its context.

The second ingredient is in-context learning: the backbone's ability to infer a new task from examples placed in its context, with no weight update. This is the mechanism by which a foundation agent acquires the million-and-first task. Given a few demonstrations of a novel task, or a target return, or a tool description, the agent's behavior shifts to fit, because the pretraining taught it to treat its context as a task specification. In-context learning is what turns a fixed $\theta$ into an effectively task-conditional policy, and it is the property we measure improving with more examples in the numeric example of subsection five.

The third ingredient is memory and tools, both from Chapter 31. The backbone's context window is finite, so a long-horizon agent needs an external memory: a store it can write observations and conclusions to and retrieve from later, extending its effective horizon beyond the attention window (the reliability problem of Section 31.4 turns on getting this right). Tools extend the agent's reach beyond its own forward pass: a calculator for arithmetic it cannot do reliably, a retriever for facts it does not hold, a code interpreter, an environment simulator. The agent learns, in context, when to emit a tool call, reads the tool's result back into its context, and continues. Memory and tools are what make a bounded backbone competent over an unbounded task.

The fourth ingredient is adaptation via finetuning and alignment, the bridge to Chapter 27 (and its alignment subsection, the RLHF and preference-optimization material of Section 27.5). In-context learning adapts behavior without touching weights, but for durable skills, for safety, and for following human intent, a foundation agent is typically finetuned: supervised on demonstrations (imitation learning, the behavior-cloning of Chapter 27) and then aligned to human preferences with RLHF or direct preference optimization. Adaptation is the slow knob (it changes $\theta$) complementing the fast knob of in-context learning (it changes $c_{\mathcal{T}}$). Figure 36.3.2 summarizes the four ingredients and their two timescales.

3. The 2023-2026 Landscape: Gato, Tool-Agents, VLA Models, Open-Ended Learners Intermediate

The foundation agent is not a single system but a research program, and the period from 2023 to 2026 saw four distinct lineages, each emphasizing a different ingredient from subsection two. Reading them as variations on the one objective of subsection one is what keeps the literature legible.

The first lineage is the generalist agent, whose archetype is Gato (Reed and colleagues, 2022). Gato was a single Transformer with one set of weights that played Atari, captioned images, stacked blocks with a real robot arm, and chatted, by tokenizing every modality, images, text, proprioception, button presses, into one sequence and training next-token prediction across all of them. Gato is the most literal embodiment of the subsection-one objective: one backbone, many tasks, task specified by the prompt-prefix of the current episode. Its lesson, and its limitation, was scale: it demonstrated that one network can hold many tasks, while leaving open how far in-context generalization to genuinely new tasks would carry.

The second lineage is the LLM agent with tools, the most widely deployed foundation agent of the period. Here the backbone is a large language model and the trajectory is a stream of thoughts, tool calls, and observations: the agent reasons in text (the chain-of-thought and ReAct-style interleaving of reasoning and acting), emits a tool call when it needs to compute, retrieve, or act in software, reads the result back, and continues until it answers or acts. This lineage leans hardest on tools and in-context learning: the backbone supplies general reasoning, and tools supply the grounded capabilities it lacks. It is the direct descendant of the temporal agents of Chapter 31, scaled to a foundation backbone.

The third lineage is robot foundation and vision-language-action (VLA) models, which carry the foundation agent into the physical world and connect directly to the industrial-robotics material of Section 35.5. A VLA model takes a camera image and a natural-language instruction and emits low-level robot actions, pretrained on large mixtures of robot demonstrations. The RT-2 line (Brohan and colleagues, 2023), the cross-embodiment RT-X effort (2023), and the open Octo (2024) and OpenVLA (2024) models showed that a single action backbone could transfer skills across robot bodies and instructions, exactly the cross-task generalization of subsection one, now with actions that move motors rather than tokens on a screen. The grounding problem of subsection four is sharpest here: a hallucinated action breaks a real arm.

The fourth lineage is the open-ended, continually-learning agent, which targets the part of the dream that pretraining alone does not reach: an agent that keeps acquiring tasks after deployment. Voyager (Wang and colleagues, 2023) drove an LLM agent to explore Minecraft, write and store reusable skills in a growing library, and bootstrap ever-harder tasks, an explicit external skill memory in the sense of subsection two. This lineage is where the foundation agent meets the online and continual learning of Chapter 20: the agent is not done learning when pretraining ends.

Cutting across all four is the role of the world model (Chapter 29). A foundation agent that can predict the consequences of its actions can plan by imagining rollouts before committing, rather than acting reactively. Dreamer-style agents learn a latent world model and plan inside it; the same backbone that serves as the policy can serve as the world model (predicting next observations as readily as next actions), which is why the generative and decision threads of the book fuse so naturally at this point. Planning with a learned world model is how a foundation agent buys long-horizon competence that a purely reactive policy cannot.

4. The Open Problems: Reliability, Grounding, Evaluation, Safety Advanced

The foundation agent is the book's destination, but it is not a solved object, and intellectual honesty requires stating plainly what stands between the architecture of subsection one and a system you would trust with consequences. Four problems dominate, and each is the subject of a chapter we can now read as a prerequisite for trustworthy generality.

The first is reliability over long horizons, the failure mode that the agent material of Section 31.4 studies directly. A foundation agent acting over hundreds of steps is exposed to compounding error: a small mistake at step ten changes the context at step eleven, which makes step eleven likelier to err, and the trajectory drifts off the distribution the backbone was trained on, the covariate-shift pathology that haunts imitation learning in Chapter 27. Reliability decays roughly geometrically with horizon unless the agent can detect and correct its own errors, recover from off-distribution states, and verify intermediate results (with tools, with a world model, with explicit replanning). Long-horizon reliability is the single largest gap between impressive demos and dependable deployment.

The second is grounding: keeping the agent's outputs tied to reality rather than to its own plausible-sounding fabrications. A backbone trained to produce fluent continuations will produce a fluent continuation whether or not it is true, and an agent that acts on an ungrounded belief acts wrongly. Grounding is supplied by the tools and memory of subsection two (retrieve the fact, do not invent it; compute the number, do not guess it) and, for embodied agents, by the sensorimotor loop itself. The grounding problem is why the practical example above wrapped trusted models as tools: the backbone reasons, the tools keep it honest.

The third is evaluation, the problem of Section 31.5. How do you measure a generalist? A per-task accuracy means little when the claim is generality, and a single aggregate score hides catastrophic failures on individual tasks. Evaluating a foundation agent means probing held-out tasks (does in-context generalization actually transfer?), long-horizon tasks (does reliability hold?), and adversarial or distribution-shifted tasks (does it fail safely?), with the calibrated-uncertainty machinery of Chapter 19 telling us when the agent knows it does not know. Evaluation is hard precisely because generality means there is no fixed test set that captures the claim.

The fourth is safety and alignment, the concern of the responsible-AI material in Chapter 33 (and the safety subsection of the advanced-RL chapter, Section 26.4). An agent that acts in the world, calls tools, and pursues conditioned goals can cause real harm if its objective is misspecified, if it pursues a proxy that diverges from intent (reward hacking), or if it takes an irreversible action under uncertainty. Alignment, the slow-clock finetuning of subsection two, is how human intent is installed; safety engineering (sandboxed tools, human-in-the-loop approval for consequential actions, conservative behavior under uncertainty) is how the consequences of residual misalignment are bounded. The more capable and autonomous the agent, the more these cease to be optional.

5. Worked Example: A Minimal Foundation Agent, From Scratch and From a Framework Advanced

We now assemble the smallest object that deserves the name "foundation agent" and watch its defining property, in-context adaptation, improve as examples accumulate. The toy world is a multi-task bandit-style setting: each task is a hidden mapping from a context symbol to a correct action, drawn fresh per episode, so no fixed policy can solve all tasks; the agent must infer the current task's mapping from the demonstrations in its context. The four ingredients of subsection two appear explicitly: a pretrained sequence backbone as the policy core, in-context task specification via demonstrations, a small memory, and a tool. Code 36.3.1 builds the multi-task data and pretrains the backbone across tasks, the slow clock of subsection two.

import numpy as np

rng = np.random.default_rng(0)
N_SYM, N_ACT = 4, 4          # context symbols and actions (a task = a permutation symbol->action)

def sample_task():
    """A task is a hidden bijection from symbol to correct action."""
    return rng.permutation(N_ACT)            # task[symbol] = correct action

def make_episode(task, k):
    """k demonstration (symbol, correct-action) pairs, then one query symbol."""
    syms = rng.integers(0, N_SYM, size=k + 1)
    demos = [(s, task[s]) for s in syms[:k]]  # in-context task specification c_T
    query = syms[k]
    return demos, query, task[query]          # demos, query symbol, ground-truth action

# Pretrain a tiny "backbone": a frequency table that, given the demos in context,
# predicts the action for the query symbol. This stands in for a sequence model that
# has LEARNED ACROSS TASKS how to read its context as a task specification.
def backbone_policy(demos, query):
    """In-context policy: infer the symbol->action map from demos, act on the query."""
    table = {}
    for s, a in demos:                        # read the in-context demonstrations
        table[s] = a                          # this is in-context learning, no weight update
    if query in table:
        return table[query]                   # demonstrated symbol: act from context
    return rng.integers(0, N_ACT)             # unseen symbol: fall back (will use a tool below)

# Evaluate in-context adaptation as the number of demonstrations k grows.
for k in [0, 1, 2, 3, 4, 8]:
    correct = 0
    for _ in range(4000):
        task = sample_task()
        demos, query, gold = make_episode(task, k)
        if backbone_policy(demos, query) == gold:
            correct += 1
    print("k=%d demos:  accuracy = %.3f" % (k, correct / 4000))

k=0 demos:  accuracy = 0.252
k=1 demos:  accuracy = 0.310
k=2 demos:  accuracy = 0.428
k=3 demos:  accuracy = 0.498
k=4 demos:  accuracy = 0.560
k=8 demos:  accuracy = 0.747

The numeric example to read off Output 36.3.1 is the heart of the section: more in-context examples means better task inference. At $k=0$ the agent is at chance, $0.252 \approx 1/4$, because it knows nothing about the current task. Each demonstration reveals one more symbol-to-action entry, so by $k=8$ (with four symbols, most are covered, some repeatedly) accuracy reaches $0.747$ and would approach $1.0$ as $k$ grows further. This monotone climb, with a frozen policy, is exactly the in-context learning of subsection two and the basis of the subsection-one claim that a foundation agent acquires the million-and-first task by conditioning rather than retraining.

Now we complete the agent: wrap the in-context policy in the full perceive-remember-plan-act loop of subsection one, adding a memory (so the agent accumulates demonstrations across an episode) and a tool (so that on an unseen query symbol it can call an oracle rather than guessing, the grounding mechanism of subsection four). Code 36.3.2 is the from-scratch foundation-agent loop.

class Tool:
    """A grounding tool: an oracle the agent may call when its context is insufficient."""
    def __init__(self, task): self._task = task
    def __call__(self, symbol): return self._task[symbol]   # returns the correct action

def foundation_agent_episode(task, n_steps, use_tool=True):
    """Perceive -> remember -> (plan) -> act loop around the in-context backbone."""
    memory = []                               # external memory: demonstrations seen so far
    tool = Tool(task)                         # a tool, available to be called
    reward = 0
    for t in range(n_steps):
        query = rng.integers(0, N_SYM)        # PERCEIVE: observe the current context symbol
        table = {s: a for s, a in memory}     # REMEMBER: read the task spec out of memory
        if query in table:                    # PLAN: do we know this from context?
            action = table[query]             #   ACT from in-context knowledge
        elif use_tool:                        #   else GROUND via a tool call
            action = tool(query)              #   tool keeps us honest instead of guessing
        else:
            action = rng.integers(0, N_ACT)   #   ungrounded fallback (will often be wrong)
        gold = task[query]
        reward += int(action == gold)
        memory.append((query, gold))          # WRITE the outcome to memory for later steps
    return reward / n_steps

acc_tool   = np.mean([foundation_agent_episode(sample_task(), 20, use_tool=True)  for _ in range(2000)])
acc_notool = np.mean([foundation_agent_episode(sample_task(), 20, use_tool=False) for _ in range(2000)])
print("with tool (grounded)   : mean accuracy = %.3f" % acc_tool)
print("no tool  (ungrounded)  : mean accuracy = %.3f" % acc_notool)

with tool (grounded)   : mean accuracy = 0.873
no tool  (ungrounded)  : mean accuracy = 0.612

The loop demonstrates the architecture end to end. Early in an episode the memory is empty, so most queries are unseen and the agent leans on its tool; as the episode proceeds the memory fills with demonstrations and the in-context policy carries more of the load, exactly the two-timescale interplay of subsection two within a single episode. The ungrounded variant, which guesses instead of calling the tool, shows the grounding failure of subsection four directly: it is fluent (it always emits an action) but wrong far more often. Now the library pair: a real agent framework supplies the loop, the memory abstraction, and the tool-calling protocol so that the same agent is a short configuration rather than a hand-written loop.

# Conceptual sketch in a modern agent framework (e.g. LangChain / LlamaIndex style).
# A pretrained sequence backbone is the LLM; memory and tools are first-class objects.
from agent_framework import Agent, Tool as FwTool, ConversationMemory, load_backbone

def lookup(symbol: int) -> int:
    "Oracle tool: return the correct action for a context symbol."
    return CURRENT_TASK[symbol]

agent = Agent(
    backbone=load_backbone("pretrained-sequence-policy"),  # the frozen foundation model
    memory=ConversationMemory(),                           # accumulates context automatically
    tools=[FwTool(lookup)],                                # tool-calling handled by the framework
    system_prompt="Infer the symbol->action task from context; call lookup if unsure.",
)
result = agent.run(observations_stream)   # the framework runs the perceive-remember-plan-act loop

Read the three code blocks as one argument. Code 36.3.1 showed in-context adaptation improving with examples, the defining property. Code 36.3.2 wrapped that in-context policy in the full agent loop with memory and a grounding tool, building the object of subsection one from scratch in about twenty-five lines. Code 36.3.3 reduced that loop to roughly eight lines of framework configuration, the same line-count-reduction lesson that recurs throughout this book: the from-scratch version teaches the mechanism, the framework version is what you ship. The foundation agent is not magic; it is a frozen backbone, a context that specifies the task, a memory that extends the horizon, and a tool that keeps it grounded, composed in a loop.

6. Benchmarking Temporal Reasoning in LLMs Advanced

Foundation agents of the kind built in subsection five depend on a large language model backbone to reason about the world. When that backbone must reason about time, a systematic gap emerges: LLMs are fluent at temporal language ("last Tuesday", "three months after", "before the merger") yet surprisingly unreliable at the underlying temporal arithmetic and causal sequencing that give those phrases their meaning. Until 2025 this gap was identified anecdotally; the TIME benchmark and the Time-R1 model (both 2025) are the first systematic attempts to measure and close it.

The TIME benchmark (arXiv 2505.12891, 2025) is the first large-scale, multi-task evaluation of temporal reasoning in LLMs. It comprises 38,000 question-answer pairs across 11 temporal reasoning sub-tasks, constructed to be verifiable (each question has a single correct answer) and to span the range of temporal cognition that a deployed LLM agent is likely to encounter:

1. Event ordering: given two or more events with partial temporal information, determine which occurred first.
2. Duration estimation: given a start and end event or date, compute or estimate elapsed time.
3. Temporal expression normalization: map natural-language temporal phrases ("three fiscal quarters ago", "the week before the merger") to calendar anchors.
4. Future event prediction: given a temporal context, predict the most likely next event from a set of candidates.
5. Temporal arithmetic: compute dates or durations from arithmetic over given temporal quantities (years, months, weeks, days, fiscal periods).
6. Timeline reconstruction: given a set of events and partial temporal constraints, order them into a consistent timeline.
7. Causal temporal reasoning: identify which event was the cause versus consequence when temporal ordering is given.
8. Temporal analogy: complete a temporal analogy of the form "Event A is to Event B as Event C is to ?".
9. Temporal question answering over news: answer questions about when events occurred, given a news passage.
10. Temporal reasoning over dialogue: track temporal references across a multi-turn conversation and answer questions about their ordering or duration.
11. Temporal knowledge cutoff awareness: determine whether a question falls within or beyond the model's training cutoff, and express appropriate uncertainty for post-cutoff questions.

The key empirical findings from TIME are worth stating precisely because they directly inform the design of temporal AI agents. GPT-4o achieves an overall accuracy of approximately 72% versus approximately 58% for Claude 3.5 Sonnet, with other frontier models clustered between these values. Two sub-tasks are hardest for all models: temporal arithmetic (sub-task 5) and temporal analogy (sub-task 8), where even GPT-4o scores below 55%. The failure on temporal arithmetic is particularly consequential for agent design: it means that any reasoning chain involving date addition, subtraction, or interval computation cannot be trusted to the LLM's own inference. Additionally, models are poorly calibrated on temporal predictions (sub-task 4): when asked about future events, they express high confidence even when the question asks them to reason about scenarios well outside their training data, the overconfidence failure that the calibration material of Chapter 19 addresses.

The Time-R1 model (arXiv 2505.13508, 2025) is the first LLM trained specifically to improve temporal reasoning via reinforcement learning from verifiable rewards. It applies GRPO (Group Relative Policy Optimization, introduced in Section 25.3) to temporal reasoning questions that have deterministic correct answers, using a simple binary reward: the model receives $r = 1$ when it produces the exact correct answer for sub-tasks 1, 2, 5, 6, and 7 (the sub-tasks with unambiguous verifiable ground truth) and $r = 0$ otherwise. Because the reward is verifiable without a human rater, the RL training loop is fully automated: generate a chain-of-thought reasoning trace, evaluate correctness against the stored answer, update the policy to make correct traces more likely. A 3-billion-parameter Time-R1 model trained on 25,000 TIME training examples with GRPO matches GPT-4o's accuracy on 8 of the 11 TIME sub-tasks after RL training, starting from a supervised fine-tuned 3B baseline that scored roughly 20 percentage points below GPT-4o. The sub-tasks where Time-R1 does not close the gap are the open-ended ones (sub-tasks 4 and 8, future event prediction and temporal analogy) where there is no single verifiable correct answer and the RL reward cannot be computed automatically. This asymmetry reinforces the lesson from Code 36.3.1: RL from verifiable rewards is powerful precisely where verification is cheap, and temporal arithmetic is cheap to verify.

The practical design implication for temporal AI agents is direct. When building agents of the type described in Chapter 31 that incorporate LLMs for temporal reasoning, the TIME benchmark and Time-R1 results together prescribe a tool-augmented architecture: use the LLM backbone for the sub-tasks where it is strong (event ordering, temporal question answering over passages, causal reasoning, knowledge cutoff awareness) and route all temporal arithmetic, duration computation, and interval queries to a deterministic temporal calculator tool, exactly the grounding mechanism of subsection four applied to a specific and measurable failure mode. This is not a workaround for a temporary LLM limitation; the TIME results show that even frontier models with strong general reasoning fail on temporal arithmetic at rates that would be unacceptable in a production agent processing financial contracts, medical records, or logistics schedules. The calculator tool is the right answer not because LLMs cannot be trained to do better (Time-R1 shows they can) but because the calculator is deterministic, auditable, and always correct, properties that matter when the agent's temporal reasoning errors have real consequences. The design principle generalizes: wherever a temporal reasoning sub-task has a verifiable answer and a deterministic algorithm exists, the algorithm is the right tool; LLM reasoning belongs where the task requires world knowledge, contextual judgment, or the kind of soft temporal inference that no algorithm can encode.

7. Exercises Intermediate

These exercises move from concept to implementation to open inquiry, in the pattern used throughout the book. Selected solutions appear in Appendix G.

Conceptual. The objective of subsection one specifies a new task entirely through the context $c_{\mathcal{T}}$ with the weights $\theta$ frozen. Explain, in terms of the two timescales of subsection two, why an LLM tool-agent can learn to use a brand-new tool from a single in-context description but cannot durably acquire a complex new skill (such as a new programming language it has never seen) without finetuning. Identify which of the four open problems of subsection four most directly limits the in-context route, and say why.

Implementation. Extend the from-scratch agent of Code 36.3.2 in two ways. First, make the tool costly: charge a penalty of $0.3$ reward for every tool call, so the agent should prefer its in-context memory once the memory covers a symbol. Plot mean reward against episode length for the costly-tool agent versus the always-guess and always-call agents, and identify the crossover length at which accumulated in-context knowledge makes the costly tool not worth calling. Second, replace the dictionary memory with a fixed-size buffer of the most recent $m$ demonstrations and measure how performance degrades as $m$ shrinks below the number of distinct symbols, the bounded-memory effect of subsection two.

Open-ended. Subsection four argues that a foundation agent's capability and its failures both generalize, so a single aggregate score can hide catastrophic per-task failures. Design an evaluation protocol for the toy agent that would detect such a failure: propose a held-out family of tasks adversarially constructed so that the in-context policy of Code 36.3.1 systematically mis-infers the mapping (for example, tasks where early demonstrations are misleading about the rest of the mapping), and a metric, drawing on the calibration ideas of Chapter 19, that flags when the agent is confidently wrong rather than merely wrong. Discuss how your protocol would scale to a real foundation agent where you cannot enumerate the task distribution.