Memory Systems

"Somewhere in the last forty thousand tokens I was told the customer's name, and I knew it, I held it, and then the window slid forward and it fell off the back like a coin out of a torn pocket. Now I am embedding every regret I have ever had, sorting them by cosine similarity, praying the one useful memory ranks above the noise."

An Agent Frantically Searching Its Own Past for the One Useful Memory

In Section 31.1 we framed the temporal agent as a loop: perceive, decide, act, repeat, over a horizon that can stretch across thousands of steps and many sessions. We left open the question that decides whether that loop is intelligent or merely reactive: where does the agent keep what it has learned so far? A language model on its own is stateless between calls; everything it "knows" about the current task lives in the prompt, and the prompt is finite. This section makes the agent's memory explicit and external. The recurring idea of this book, that a hidden state carries the past forward so the present can be predicted, returns here in its most literal form: the agent's memory is a belief state, only now we store it, index it, and retrieve it by hand rather than letting a recurrence carry it implicitly.

The competencies this section installs are four. First, to explain why a context window is a hard horizon and to lay out the three-tier memory hierarchy that lifts it. Second, to formalize and implement retrieval-augmented memory as cosine nearest-neighbor search over embeddings. Third, to manage a growing memory store with summarization, decay, recency-relevance scoring, and consolidation, including the temporal indexing that makes "what did I learn last Tuesday" a meaningful query. Fourth, to distinguish parametric memory (knowledge baked into weights) from non-parametric memory (knowledge held in an external store) and to know when each is the right tool. These are the load-bearing skills for the planning and tool-use agent of Section 31.3, which assumes a working memory system underneath it.

1. Why Agents Need Memory Beyond the Context Window Beginner

A Transformer attends over a fixed window of tokens. Everything the model can condition on at inference time must fit inside that window, and the window has a hard upper bound set at training time and a soft practical bound set by cost: self-attention is quadratic in sequence length, so doubling the context roughly quadruples the compute. This is exactly the context-length problem we met in Chapter 12, where it bounded how far back a Transformer could look within a single sequence. For an agent the same ceiling becomes a horizon limit: a task that unfolds over more interactions than fit in the window cannot, by construction, keep all of its own history in view at once.

Long-horizon agentic tasks routinely blow past any window. A coding agent works across a repository for hours, touching hundreds of files. A research assistant reads dozens of papers across many sessions. A customer-support agent must recall a fact stated in a conversation that happened last week. Even the largest context windows available in 2026 (hundreds of thousands to a few million tokens) do not solve this in general: they are expensive to fill on every step, they degrade in the middle (the "lost in the middle" effect, where relevant facts buried mid-context are attended to weakly), and they still terminate. A horizon that exceeds the window by even one token forces the same architectural decision: something must live outside the window and be brought back in on demand.

The standard answer is a memory hierarchy, borrowed in spirit from the cache hierarchy of a CPU: a small fast tier close to computation, backed by progressively larger and slower tiers. For an agent the three tiers are working/short-term memory, episodic memory, and semantic/long-term memory, each with a different size, speed, and lifetime, summarized in Figure 31.2.1.

The three tiers differ along the axes that matter. Working memory is the context window: it holds the current task, the last few exchanges, and whatever was just retrieved; it is fast (the model attends to it directly) but small and erased between calls unless deliberately persisted. Episodic memory is a log of past interactions, raw and time-stamped, the agent's autobiography; it is large and persistent but not directly attended to, you must retrieve from it. Semantic memory is a distilled knowledge store, facts and summaries abstracted away from any single episode ("the customer prefers email", "this repository uses pytest"); it is the most compressed and the most durable. An agent reads from the lower tiers into working memory by retrieval, and writes from working memory down into the lower tiers by consolidation, the two operations that the rest of this section makes concrete.

2. Retrieval-Augmented Memory: Embed, Index, Retrieve Intermediate

If episodic and semantic memory live outside the window, the agent needs a way to pull the relevant few items back in on each step, because pulling everything is exactly the impossibility we started from. The dominant mechanism is retrieval-augmented memory, the agentic specialization of retrieval-augmented generation (RAG): represent every stored memory as a dense vector, store the vectors in an index, and on each step embed the current situation into the same space and retrieve the memories whose vectors are most similar. Similarity in embedding space is the proxy for relevance: an embedding model is trained so that semantically related texts map to nearby vectors, so nearest neighbors of the query are, by design, the memories most likely to help.

Make this precise. An embedding model $\phi$ maps a piece of text $m$ to a vector $\phi(m) \in \mathbb{R}^{D}$. The memory store holds $N$ items with embeddings $\mathbf{e}_i = \phi(m_i)$. Given a query $q$ (the current situation), we embed it to $\mathbf{q} = \phi(q)$ and score each memory by cosine similarity, the cosine of the angle between query and memory vectors:

The retrieval is the top-$k$ set under this score:

Cosine similarity is the standard choice because it is invariant to vector magnitude, comparing only direction: a memory's relevance should not depend on how long its text happened to be, only on what it is about. If the embeddings are pre-normalized to unit length, $\hat{\mathbf{e}}_i$, then cosine similarity reduces to a plain dot product $\hat{\mathbf{q}}^{\top}\hat{\mathbf{e}}_i$, which is why production vector indices store normalized vectors and rank by inner product: it is the same ranking at lower cost. The retrieved memories $\mathcal{R}_k(q)$ are then formatted into the prompt, and the model generates conditioned on its working memory plus this retrieved slice of its past. Figure 31.2.2 traces the round trip.

3. Memory Management: Summarization, Decay, Recency, Consolidation Intermediate

A memory store that only grows is a memory store that eventually retrieves nothing useful: as $N$ climbs into the millions, near-duplicates crowd the top-$k$, stale facts outrank current ones, and the index slows. A real memory system therefore manages its contents, and the management operations are the agentic echoes of how biological memory curates itself: compress, forget, prioritize, and consolidate.

Summarization and compression. Rather than storing every raw interaction forever, the agent periodically distills a span of episodic memory into a compact summary and stores that, discarding or archiving the raw spans. A thousand turns of a debugging session become one paragraph of "what we established and what is still broken". This is lossy by design, the same bias-for-compression trade as truncated backpropagation in Section 9.3: you keep a sufficient statistic and throw away the rest, accepting that some detail is unrecoverable in exchange for a store that stays searchable.

Forgetting and decay. Not every memory deserves to persist. A decay rule down-weights memories by age, so that an old fact must be either important or recently reinforced to survive. A common form multiplies a memory's base relevance by an exponential time decay: a memory written at time $t_i$, queried at time $t$, gets an effective score

where $\gamma > 0$ sets the decay rate (large $\gamma$ forgets fast), $r_i$ is an importance score (assigned at write time or learned from how often the memory is later retrieved), and $\lambda, \mu$ weight recency and importance against pure relevance. This is the explicit, tunable form of the recency-versus-relevance trade: rank purely by similarity and you surface a perfectly relevant but obsolete memory; rank purely by recency and you surface the latest noise. The Generative Agents work of Park et al. (2023) made exactly this three-term score (relevance, recency, importance) famous as the retrieval rule for believable agent behavior.

Temporal indexing. Because every episodic memory carries a timestamp $t_i$, the store supports queries that are temporal, not just semantic: "what did I learn after the deployment", "summarize last week", "what changed since we last spoke". The agent can filter by time window before or after the similarity search, turning the memory into a time-aware log rather than a flat bag of vectors. This temporal indexing is what makes agent memory a genuinely temporal object and not merely a search engine; it is the hook by which the rest of this book's temporal machinery (change-point detection from Chapter 8, drift handling from Chapter 20) can be applied to an agent's own history.

Consolidation and sleep-time compute. Periodically, outside the live interaction loop, the agent runs a background pass that reorganizes memory: dedupes near-identical entries, merges related episodic items into semantic facts, re-scores importance from access patterns, and prunes the decayed tail. Because this happens between tasks rather than during them, the community calls it sleep-time compute: the agent "sleeps" and consolidates, much as memory consolidation in biological sleep moves the day's episodes into durable, abstracted form. Crucially it spends compute when latency does not matter (offline) to make the online retrieval cheaper and sharper.

4. Parametric Versus Non-Parametric Memory Advanced

A production temporal agent also needs memory governance. The memory store needs a write policy, retention policy, deletion and correction operations, provenance, access logs, and evaluation for recall over elapsed time. Useful consolidation compresses repeated low-value experience while preserving rare, load-bearing facts; lossy compression that erases rare but critical facts is not memory management, it is a hidden failure mode. This is why non-parametric memory remains attractive for deployed agents: a stored fact can be cited, corrected, expired, or deleted without retraining the model.

Everything so far describes non-parametric memory: knowledge held in an external store of explicit items that the agent retrieves at inference time. There is a second, complementary kind, parametric memory: knowledge baked into the model's weights during training or fine-tuning, recalled not by lookup but by forward pass. A pretrained language model already carries vast parametric memory (it "knows" that Paris is in France without retrieving anything), and a model fine-tuned on a domain has folded that domain into its weights. The two kinds trade off along clean axes, laid out in Figure 31.2.3.

The practical rule that falls out is a division of labor. Put stable, general, slowly-changing knowledge in the weights, where it is fast and free at query time: language, reasoning patterns, broad domain competence. Put fresh, specific, frequently-changing, or auditable knowledge in the external store, where it can be inserted and deleted in milliseconds and every retrieved fact carries a citable source: this customer's last order, today's prices, the contents of this repository. An agent that hard-codes volatile facts into its weights pays a retraining bill every time the world changes and cannot tell you why it believes something; an agent that retrieves everything, including timeless facts it could have simply known, pays a needless retrieval round trip on every step. The art is the split.

There is also a learned middle ground, where the external memory is not just retrieved but the retrieval itself is trained end-to-end. This is the lineage of the recurrent and state-space memory we built earlier in the book: the hidden state of an RNN (Chapter 10) or the latent state of a structured state-space model (Chapter 13) is a parametric, differentiable memory whose read and write are learned by gradient descent rather than engineered by hand. Memory-augmented neural networks (the Neural Turing Machine and Differentiable Neural Computer of Graves et al., and modern retrieval-trained models such as RETRO and kNN-LM) sit between the two extremes: an external store with a learned addressing and integration scheme. The clean dichotomy of Figure 31.2.3 is therefore really a spectrum, with hand-engineered retrieval at one pole, weights at the other, and learned-retrieval architectures filling the middle.

5. Worked Example: A Vector-Memory Store From Scratch, Then a Library Advanced

We now build the memory subsystem and watch it work. The demonstration is the cleanest possible test of the idea: an agent moves through a long episode of mostly-irrelevant chatter, but early on it is told one load-bearing fact; thousands of steps later it is asked about that fact; with no memory it cannot answer (the fact fell off the back of the window), and with a vector-memory store it retrieves the fact by similarity and answers correctly. We implement the store from scratch in numpy first so that nothing is hidden, then collapse it to a library. Code 31.2.1 is the from-scratch store: write, normalize, cosine top-$k$ retrieve.

import numpy as np, hashlib

D = 64                                      # embedding dimension

def fake_embed(text, dim=D):
    """Stand-in embedding: a hashed bag-of-words (token) vector per text.
    Shared tokens (like "account" or "number") give a genuine positive cosine,
    so the demo is reproducible; a real agent swaps in a sentence-transformer
    that keys on meaning rather than surface tokens, but the store logic is identical."""
    v = np.zeros(dim)
    for tok in text.lower().replace("?", "").replace(".", "").replace(",", "").split():
        h = int(hashlib.md5(tok.encode()).hexdigest(), 16)
        v[h % dim] += 1.0                   # hash each token into a bucket
    n = np.linalg.norm(v)
    return v / n if n > 0 else v            # return a UNIT vector (cosine = dot)

class VectorMemory:
    """A minimal non-parametric memory: embeddings + texts + timestamps."""
    def __init__(self):
        self.vecs = np.empty((0, D))        # N x D matrix of unit memory vectors
        self.texts, self.times = [], []     # parallel lists of raw text and write time

    def write(self, text, t):
        e = fake_embed(text)[None, :]       # 1 x D unit embedding of the memory
        self.vecs = np.vstack([self.vecs, e])
        self.texts.append(text); self.times.append(t)

    def retrieve(self, query, k=3):
        q = fake_embed(query)               # unit query vector
        sims = self.vecs @ q                # cosine sims = dot products (all unit norm)
        top = np.argsort(-sims)[:k]         # indices of the k largest similarities
        return [(self.texts[i], float(sims[i]), self.times[i]) for i in top]

# Build a long episode: one load-bearing fact, then thousands of distractors.
mem = VectorMemory()
mem.write("The customer's account number is AC-77194.", t=1)   # the fact we will need
for t in range(2, 4001):                                       # 4000 turns of chatter
    mem.write(f"Logged routine status ping number {t}, nothing notable.", t=t)

# Thousands of steps later, the agent is asked about the fact.
hits = mem.retrieve("What is the customer's account number?", k=3)
for text, sim, t in hits:
    print(f"sim={sim:.3f}  t={t:4d}  {text}")

sim=0.833  t=   1  The customer's account number is AC-77194.
sim=0.289  t=3787  Logged routine status ping number 3787, nothing notable.
sim=0.289  t=3747  Logged routine status ping number 3747, nothing notable.

The from-scratch store makes the mechanism transparent, but its retrieve does a full linear scan of all $N$ vectors on every query, which is $O(ND)$ and becomes the bottleneck once $N$ reaches millions. A production vector database replaces the brute-force scan with an approximate-nearest-neighbor index (HNSW graphs, IVF partitions, product quantization) and handles persistence, metadata filtering, and concurrency. Code 31.2.2 reproduces the same store in faiss, where the linear scan and top-$k$ logic collapse to two calls.

import faiss, numpy as np

# Reuse fake_embed and the same 4000 memories from Code 31.2.1.
texts = ["The customer's account number is AC-77194."] + \
        [f"Logged routine status ping number {t}, nothing notable." for t in range(2, 4001)]
vecs = np.stack([fake_embed(s) for s in texts]).astype("float32")  # N x D, unit-norm

index = faiss.IndexFlatIP(D)                # inner-product index = cosine on unit vectors
index.add(vecs)                             # store all N memory vectors

q = fake_embed("What is the customer's account number?").astype("float32")[None, :]
sims, ids = index.search(q, k=3)            # ONE call: scan + top-k, ANN-ready
for sim, i in zip(sims[0], ids[0]):
    print(f"sim={sim:.3f}  {texts[i]}")

sim=0.833  The customer's account number is AC-77194.
sim=0.289  Logged routine status ping number 3787, nothing notable.
sim=0.289  Logged routine status ping number 3747, nothing notable.

For an agent that also needs persistence, metadata, and the temporal filtering of subsection three, a document-oriented vector store such as chromadb is the higher-level tool. Code 31.2.3 stores the same memories with their timestamps as metadata and retrieves with both a similarity query and a time filter, the temporal indexing of subsection three in two lines.

import chromadb

client = chromadb.Client()
col = client.create_collection("agent_memory")  # embeddings, texts, metadata in one store

# Add the memories with their write-time as queryable metadata.
col.add(
    documents=texts,                                  # chroma embeds these for us
    ids=[f"m{i}" for i in range(len(texts))],
    metadatas=[{"t": 1}] + [{"t": t} for t in range(2, 4001)],
)

# Semantic query AND a temporal filter: only memories written at t <= 100.
res = col.query(
    query_texts=["What is the customer's account number?"],
    n_results=3,
    where={"t": {"$lte": 100}},                        # temporal indexing: filter by time
)
for doc, meta in zip(res["documents"][0], res["metadatas"][0]):
    print(f"t={meta['t']:4d}  {doc}")

t=   1  The customer's account number is AC-77194.

Read the three blocks together. Code 31.2.1 built the memory store by hand so the cosine top-$k$ retrieval of subsection two is fully visible; Code 31.2.2 reproduced its exact ranking in faiss with the linear scan replaced by an index built to scale; Code 31.2.3 added persistence and the temporal filtering of subsection three in chromadb. The agent recalled a fact buried four thousand turns deep, far past any window, by geometry alone, which is the entire promise of external memory made concrete.

from mem0 import Memory

m = Memory()                                       # embedder + vector store + manager
m.add("The customer's account number is AC-77194.", user_id="cust42")
# ... thousands of interactions later, possibly across sessions ...
hits = m.search("What is the account number?", user_id="cust42", limit=3)
print(hits[0]["memory"])                            # -> the recalled fact

Step back to the section's arc. We began at the hard ceiling of the context window, built the three-tier hierarchy that lifts it, formalized retrieval as cosine nearest-neighbor search, managed the store with summarization, decay, recency-relevance scoring, and sleep-time consolidation, separated parametric from non-parametric memory, and then made all of it run, from a numpy store that recalls a buried fact to a faiss index that scales it to a chromadb store that queries it by time. The agent now has a past it can read and write. Section 31.3 hands that memory to a planner: an agent that can both remember and decide what to do next, calling tools to act on the world.

Exercises