Imagine a colleague who forgets, at the end of every task, everything you did together. Every time you started a new project, you’d have to reintroduce yourself, explain what you were working on, what worked and what didn’t, and what you’d already tried. It would be exhausting, and after a while you’d stop working with them.
That’s a plain language model, more or less. It has no memory of any conversation before this one. Every task starts from a blank slate. Which is fine for one-shot answers, but for an agent trying to get real work done across many steps (and often many sessions), it’s a serious limitation. So this article is about the different kinds of memory an agent can have, how each one works, and how to design memory that actually helps rather than adds complexity for its own sake.
Two kinds of memory, and why both matter
The first useful distinction is between short-term memory and long-term memory. These are borrowed loosely from how psychologists talk about human memory, and while agents don’t work the same way people do, the analogy is genuinely helpful.
Short-term memory is what an agent needs to keep track of within the current task. What was the goal? What tools have I called? What did they return? What have I concluded so far? This is the memory that lives inside the agent loop, and it’s what lets an agent do multi-step work coherently. Without it, every iteration would forget everything from the previous one.
Long-term memory is what an agent needs to keep track of across tasks and sessions. What did we agree last week? What are this user’s preferences? What have I learned about how to solve this class of problem? This memory persists beyond the current run and is what makes an agent feel like it’s building on past interactions rather than starting over. Figure 1 shows the two clearly.
Figure 1: Short-term memory lives inside the running agent loop, holding what’s happened in this task so far. Long-term memory sits outside, in a persistent store, holding what’s been learned across tasks and sessions. Well-designed agents have both, and they’re designed differently.
Both matter, and they’re designed differently. Let me walk through each honestly.
Short-term memory: the context window, mostly
The short-term memory of an agent is, in most implementations, just its context window. That’s the model’s working memory from the AI Primer, the finite chunk of text it can hold in view at once. And in an agent, the context window is what carries the running trace of the current task: the goal, the sequence of thoughts, the tool calls, the observations, the user’s input, and any relevant background.
Every iteration of the agent loop, the model reads the current context, decides what to do next, and adds its thought, action, and the observation back to that context. Then the loop continues with the growing trace. That trace is the short-term memory. It’s not stored anywhere else; it’s literally what the model sees each turn.
This has a very practical implication: the context window is a finite budget, and everything competing for space in it, the system prompt, the tool descriptions, the running history, any retrieved information, has to fit together. For simple agents, this is not a problem. For agents doing long, complex tasks, it can become one, because the history grows with each iteration.
Two common techniques manage this well:
Rolling window. Keep the goal and system prompt fixed, but drop the oldest observations once they’re no longer useful. It’s a bit crude, but for many tasks the recent context is what matters most.
Summarisation. When the trace gets long, use the model itself to summarise the earlier part into a compact “here’s what happened and what we’ve learned so far” note, and replace the raw history with the summary. This preserves the substance of earlier steps while freeing up context space. It’s more sophisticated and works well for genuinely long tasks.
In practice, most agents just make sure the trace fits in the window and don’t worry about summarisation until they see it’s needed. Modern context windows are generous. Don’t optimise for a problem you don’t have.
Long-term memory: three shapes worth knowing
Long-term memory is more design-heavy, because unlike short-term memory (which is largely “just fits in the context window”), long-term memory needs to be stored somewhere, retrieved when relevant, and managed as it grows. There are three broad patterns for how this works, and understanding them is what separates a “chatbot with a fancy label” from an agent that actually gets smarter over time. Figure 2 shows them.
Figure 2: Long-term memory tends to come in three shapes. Episodic memory records what happened in past tasks. Semantic memory holds learned facts and preferences. Procedural memory captures how to do specific things well. Each is designed differently, and each solves a different problem.
Episodic memory: what happened. This stores the record of past interactions. Previous conversations, previous tasks the agent has completed, previous outcomes. The typical implementation is a database of conversation logs or task traces, made searchable (often with embeddings, using the semantic-search idea from the RAG track). When a new task comes in, the agent can retrieve relevant past episodes (“last time this user asked about X, we did Y”) and use them to inform its current work.
Semantic memory: what’s true. This is the distilled facts the agent has picked up. User preferences (“Priya prefers concise responses”). Facts about the domain (“our support team responds to enterprise tickets first”). Standing instructions the user has given (“don’t send anything without asking first”). These are usually stored as a small, structured knowledge base that gets injected into the agent’s context at the start of a task, either directly or via retrieval.
Procedural memory: how to do things. This is the trickiest and most experimental of the three. It’s how the agent learns to do specific tasks better over time. If the agent has completed similar tasks a hundred times, ideally it has picked up the general shape of “how to do this task,” and future runs benefit from that shape. In practice, this looks like storing successful task templates or “playbooks” that the agent can retrieve and adapt when it recognises a similar goal.
Not every agent needs all three. Many production agents get by with just semantic memory (user preferences and standing instructions), and add episodic memory only when there’s a clear reason (“we want the agent to be able to reference past conversations”). Procedural memory is genuinely useful but demands more sophisticated infrastructure. As with tool design, start with the smallest set that solves your actual problem.
How memory usually flows in a running agent
Let me walk you through how memory shows up in practice, because it’s more concrete than the categorisation might suggest. Here’s the flow, in pseudocode:
def run_agent(user_input, user_id):
# Load relevant long-term memory before the loop starts.
facts = semantic_memory.load(user_id) # user preferences, standing instructions
past = episodic_memory.search(user_input) # relevant past episodes
context = build_initial_context(
system_prompt = SYSTEM_PROMPT,
facts = facts,
recent_episodes = past,
user_input = user_input,
)
for step in range(MAX_STEPS):
thought, action = model.think(context)
observation = execute(action)
context = context + [thought, action, observation] # short-term memory grows
if is_done(action):
break
# Optionally update long-term memory after the task.
episodic_memory.save(user_id, context)
semantic_memory.update_from(context) # learn any new preferences
return final_message(context)Notice the shape. Long-term memory feeds the start of the task, filling the initial context with relevant facts and history. The agent then runs its loop, growing the short-term memory as it goes. And when the task’s done, the long-term memory gets updated with what was learned, so the next task benefits. That “load at start, update at end” pattern is the essence of memory-augmented agents, and it’s remarkably powerful once you see it.
The honest trade-offs
Memory sounds like a pure win, but it comes with real costs and risks worth naming.
More context means more cost and more latency. Every token of memory you inject into the initial context is a token you pay for and process. Adding memory to an agent that’s already borderline on context or latency can make it worse, not better.
Memory can be wrong or stale. If your semantic memory records that “Priya prefers concise responses” three years ago, and Priya’s tastes have changed, the agent will confidently produce concise responses because that’s what its memory says. Freshness matters.
Memory has privacy implications. Anything the agent remembers, it can potentially surface later. If you’re storing episodic memory that includes sensitive interactions, you need to think carefully about access, retention, and redaction. The guardrails article of the LLMOps track applies here in full force.
More memory doesn’t always mean smarter behaviour. An agent overloaded with historical context can actually get worse at the current task, because the signal for what to do now gets diluted by everything the memory dredged up. Retrieval quality matters, and less is often more.
For all these reasons, memory in agents is a design choice, not a checkbox. Add each piece of memory because it solves a concrete problem, not because it seemed like a good idea. If the agent works well without long-term memory, don’t add it. If it’s forgetting things that clearly matter, add the specific memory type that fixes that specific problem.
Where memory fits with everything else
Memory is one of the four pillars of an agent alongside the loop, the tools, and planning. Together, they’re what turn a language model into something that can work across time. But even with tools and memory, an agent needs to decide sensibly what to do next, and for complex tasks that means thinking a step further than “what tool do I call now?” It means planning. That’s the next article, and it’s where the model’s reasoning starts to really shine.
Next in the series: Planning & Reasoning in Agents, decomposing goals and adapting when things go sideways.