Chapter 6 — Memory Systems
“Without memory, there is no learning. Without learning, there is no intelligence.”
Why Agents Need Memory
An LLM by itself is stateless — each API call is independent, with no knowledge of previous calls. Agents need memory systems to:
- Maintain conversation context across turns
- Learn from past mistakes and successes
- Access large knowledge bases that don’t fit in the context window
- Remember user preferences across sessions
- Build up working state during complex multi-step tasks
The Memory Hierarchy
Working Memory: The Context Window
The context window is the most immediate form of memory. Everything the LLM can “see” right now lives here:
- System prompt
- Conversation history
- Tool definitions
- Tool call results
- Retrieved documents
The Context Window Problem
As an agent runs, the context window fills up. A typical agent interaction:
| Content | Token Estimate |
|---|---|
| System prompt | ~500 tokens |
| Tool definitions (10 tools) | ~2,000 tokens |
| User message | ~100 tokens |
| Per loop iteration (LLM response + tool result) | ~1,000–5,000 tokens |
| After 10 iterations | ~15,000–55,000 tokens |
With a 128K context window, you have room. But at $2.50/M input tokens, those tokens get expensive fast.
Context Management Strategies
class ContextManager:
def __init__(self, max_tokens: int = 100_000):
self.max_tokens = max_tokens
self.messages = []
def add(self, message: dict):
self.messages.append(message)
self._trim_if_needed()
def _trim_if_needed(self):
"""Sliding window: keep system prompt + recent messages."""
while self._estimate_tokens() > self.max_tokens:
# Never remove system prompt (index 0) or last 5 messages
if len(self.messages) > 6:
self.messages.pop(1) # Remove oldest non-system message
def _summarize_old_context(self):
"""Alternative: summarize old messages instead of dropping them."""
old = self.messages[1:-5] # Everything except system + recent
summary = llm.summarize(old)
self.messages = [
self.messages[0], # system prompt
{"role": "system", "content": f"Previous context summary: {summary}"},
*self.messages[-5:], # recent messages
]
Short-Term Memory: Conversation History
Short-term memory is the conversation buffer — the growing list of messages exchanged between the user, the agent, and tools within a single session.
The agent understands “that” and “last year” because the previous turns are in short-term memory.
Long-Term Memory: Vector Stores
For knowledge that persists across sessions, agents use vector databases. The pattern:
- Store: Convert text into embeddings and save them
- Retrieve: When needed, find relevant memories by semantic similarity
- Inject: Add retrieved memories into the context window
from openai import OpenAI
import chromadb
client = OpenAI()
db = chromadb.Client()
collection = db.create_collection("agent_memory")
def store_memory(text: str, metadata: dict = None):
"""Store a fact in long-term memory."""
embedding = client.embeddings.create(
model="text-embedding-3-small",
input=text
).data[0].embedding
collection.add(
ids=[str(hash(text))],
embeddings=[embedding],
documents=[text],
metadatas=[metadata or {}],
)
def recall_memories(query: str, k: int = 5) -> list[str]:
"""Retrieve relevant memories."""
embedding = client.embeddings.create(
model="text-embedding-3-small",
input=query
).data[0].embedding
results = collection.query(
query_embeddings=[embedding],
n_results=k,
)
return results["documents"][0]
What to Store in Long-Term Memory
| Memory Type | Example | Why |
|---|---|---|
| User preferences | “User prefers Python over JavaScript” | Personalization |
| Past task outcomes | “Last time I searched for X, the best source was Y” | Learning |
| Extracted facts | “Company X was founded in 2019” | Knowledge accumulation |
| Conversation summaries | “In our last session, we planned the API architecture” | Cross-session continuity |
| Error lessons | “Using endpoint X requires auth header Y” | Self-improvement |
Episodic vs. Semantic Memory
Inspired by human cognition, agent memory can be split into:
- "What happened" — specific events and experiences
- "Last time you asked about stocks, you wanted tech sector"
- Stores conversation transcripts, task logs
- Good for: personalization, learning from mistakes
- "What I know" — general facts and knowledge
- "Apple is a tech company headquartered in Cupertino"
- Stores extracted facts, documentation chunks
- Good for: answering questions, reasoning about domains
RAG: Memory From External Sources
Retrieval-Augmented Generation lets agents access vast knowledge bases without storing everything in the context window:
# RAG as an agent tool
def search_knowledge_base(query: str) -> str:
"""Search the company knowledge base for relevant information."""
# 1. Embed the query
query_embedding = embed(query)
# 2. Search the vector database
results = vector_db.search(query_embedding, top_k=5)
# 3. Format results for the LLM
context = "\n\n".join([
f"[Source: {r.metadata['source']}]\n{r.text}"
for r in results
])
return context
Memory in Practice: The Mem0 Pattern
Mem0 popularized a pattern for agent memory that auto-extracts and stores important facts:
from mem0 import Memory
memory = Memory()
# After each conversation, auto-extract and store memories
memory.add("I prefer dark mode and vim keybindings", user_id="user_123")
memory.add("My project uses FastAPI with PostgreSQL", user_id="user_123")
# Before generating a response, retrieve relevant memories
relevant = memory.search("What framework should I use?", user_id="user_123")
# Returns: ["My project uses FastAPI with PostgreSQL"]
Memory Architecture Summary
What’s Next
Memory gives agents knowledge. But an agent also needs to reason — to plan, decompose problems, and correct its own mistakes. That’s planning.
Next: Chapter 7 — Planning & Reasoning →
← Previous: Chapter 5 — Tools & Function Calling · Next: Chapter 7 — Planning & Reasoning →
Last updated: April 2026