What Is AI Agent Memory? Types, Architectures, and How to Choose

AI agent memory is the mechanism that lets an agent retain and reuse information across turns and across sessions. Without it, every conversation starts from nothing: the model sees only what is in its context window right now, and anything learned five minutes or five weeks ago is gone.

This matters more than it sounds. A model with a two-hundred-thousand-token context window still has no memory. It has a large short-term buffer that empties completely when the session ends. Memory is the layer that decides what survives that, what gets retrieved later, and how. The distinction is easy to lose because both feel like the model "knowing" something in the moment, but only one of them is still there tomorrow, and only one of them can be audited, corrected, or shared with a colleague's agent.

Why do agents need memory at all?

The obvious answer is continuity: nobody wants to re-explain their situation every session. But the operational reasons are sharper.

Cost. Re-establishing context is not free. Every session that rebuilds the same background pays for those tokens again. At scale, the retrieval and re-reading overhead becomes a material share of the bill, which is why the topic sits next to LLM gateway cost control in most infrastructure discussions.

Latency. Rebuilding context is sequential. The agent calls a tool, reads the result, decides it needs another, calls again. Each hop adds time, and the user waits through all of it.

Accuracy. An agent that re-discovers context each time re-discovers it inconsistently. It might find the right document on Monday and a stale one on Thursday. Memory that persists gives repeatable answers.

Capability. Some questions are simply unanswerable without accumulated state. "Has this customer raised this issue before?" requires knowing what happened before. No context window size fixes that.

What are the types of AI agent memory?

The vocabulary borrows from cognitive science, which is useful once you map it to what actually gets stored.

Short-term memory

Short-term memory is the working context: the current conversation, recent tool results, the system prompt. It lives in the context window and disappears when the session ends.

The engineering problem here is not storage but selection. Context windows are large, and filling them is easy, but attention degrades as they fill and every token costs money. Practical short-term memory is mostly about compaction: summarizing older turns, dropping tool outputs that are no longer relevant, keeping the parts that still matter.

Long-term memory

Long-term memory persists across sessions. It is written during or after a session and retrieved in later ones. This is the part people usually mean by "agent memory," and it divides into three kinds.

Episodic memory records what happened. A specific meeting, a particular decision, an exchange with a customer. Episodic memories are time-stamped and specific, and their value is that they can be replayed or cited.

Semantic memory records what is true. Facts, entities, and relationships extracted from episodes: who owns this account, what this project depends on, which vendor was chosen. Semantic memory is distilled rather than raw, and it is what supports reasoning rather than recall.

Procedural memory records how to do things. Learned workflows, tool-use patterns, formatting conventions the user prefers. In practice this is often implemented as instructions rather than retrieved memories, but it is a distinct category.

The distinction that matters commercially is episodic versus semantic. An agent that only stores episodes can find a conversation. An agent that has distilled those episodes into facts can answer a question that no single conversation contains.

How is agent memory actually built?

Three architectural approaches dominate, and they make genuinely different trade-offs.

Vector-based memory

Text is embedded into vectors and stored in a vector database. Retrieval finds the chunks most semantically similar to the query.

This is the simplest approach and the most widely deployed. It works well for "find me something like this" and poorly for anything requiring precision about entities or structure. Ask a vector store "how many open issues does this team have" and it will return passages that talk about open issues rather than a count, because similarity search has no concept of counting.

Graph-based memory

Information is stored as entities and the relationships between them. Retrieval traverses the graph rather than ranking by similarity.

Mem0's own explainer describes the difference plainly: graph memory stores "information as entities and the relationships between them, rather than as isolated chunks of text." That structure is what makes multi-hop questions answerable. "Which customers are affected by the outage in the service that Dana owns" requires following three relationships, which is a traversal, not a similarity match.

The cost is extraction. Something has to decide what the entities and relationships are, and that step is where graph systems succeed or fail.

Temporal knowledge graphs

A refinement of the graph approach that adds time. Facts are not just true, they are true during a period, and new facts can supersede old ones without deleting the history.

This solves the staleness problem that plagues both vector and plain graph memory. When a project's owner changes, a naive store now holds two contradictory facts. A temporal graph records that the first was valid until a date and the second is valid now, so a query about today gets one answer and a query about last quarter gets the other.

The Zep paper, published on arXiv, introduces exactly this architecture and reports outperforming MemGPT on the Deep Memory Retrieval benchmark. Its open-source implementation, Graphiti, has become the most visible temporal graph project in the space.

Hybrid

Most serious systems end up combining approaches: a vector store for semantic recall over raw content, and a graph for entities and relations. Mem0 uses a dual store of exactly this shape.

The reason is that neither alone is sufficient. Vectors handle "what was said about this" and graphs handle "what is true about this," and real questions need both.

What does the framework landscape look like?

Three open-source projects define the developer-facing category, and it is worth knowing what each is actually for.

[Mem0](https://docs.mem0.ai/) is a memory layer for LLM applications, focused on extracting durable facts from conversation and making them retrievable. It is open source with a managed cloud option, and its centre of gravity is per-user personalization: preferences, traits, history.

[Zep](https://www.getzep.com/ai-agents/temporal-knowledge-graph) builds on Graphiti's temporal knowledge graph and positions around enterprise scale and governance. Its distinguishing technical claim is fact succession over time rather than flat storage.

[Letta](https://www.letta.com/), formerly MemGPT, comes out of the MemGPT paper, which framed memory as an operating-system problem: the agent manages its own paging between a limited context window and external storage. Letta is the productized version of that idea, and it is the most agent-centric of the three.

These are developer frameworks. You wire them into your application, decide what gets written, and own the result. That is the right shape for a product team building a specific agent, and the wrong shape for an enterprise that wants memory across every tool and every team, because per-agent memory does not compose into organizational knowledge.

A fuller side-by-side is in the Mem0 vs Zep vs Letta comparison.

Does agent memory save money or cost money?

Both, and which one dominates depends on a ratio worth calculating before you build.

Memory costs money in three places: the extraction pass that writes it, the storage and indexing, and the retrieval work on every query. A graph system with a model-driven extraction step is doing real inference on every incoming document, and that is not free.

It saves money in one place, but it is a large one: not rebuilding context. An agent without memory re-queries systems, re-reads documents, and re-establishes the same background every session. Those tokens are paid for every time, and the sequential tool calls that produce them also cost latency the user experiences directly.

The ratio that decides it is how often the same context gets reused. A support agent handling forty tickets a day against the same product and the same customer base reuses context constantly, and memory pays back quickly. A one-shot research agent that touches a different domain each run reuses almost nothing, and the extraction overhead is close to pure cost.

Two practical notes. First, measure the retrieval overhead per query, not just the accuracy gain, because a system that adds two thousand tokens of retrieved memory to every call has raised your floor. Second, the saving compounds with team size in a way single-user testing will not reveal: when one person's session produces a fact that fifty colleagues' agents can use, the arithmetic changes completely. That is also precisely the point where per-agent frameworks stop capturing the benefit.

What goes wrong with agent memory in production?

The failure modes are consistent enough to plan around.

Confident staleness. The agent asserts something that was true and is not any more. This is the most damaging failure because it looks like competence. It comes from a store that overwrites or accumulates without succession, and it is the single strongest argument for temporal structure.

Memory poisoning. Something incorrect gets extracted and stored, then retrieved repeatedly and treated as established. Because memories are usually surfaced without provenance, nobody can trace where the wrong fact came from. Storing the source of every memory costs little and is the only practical remedy.

Context crowding. Retrieval returns twelve loosely relevant memories, they fill the window, and answer quality drops. Teams usually diagnose this as a model problem and it is a retrieval problem. Fetching fewer, better-scoped memories almost always beats fetching more.

Permission leakage. A memory extracted from a document one person could see gets served to someone who could not. The memory layer has become a route around access control that the source systems enforce correctly. In a single-user product this cannot happen; in an enterprise it is the first thing security will ask about.

Identity confusion. Two people named Dana, two projects called Atlas, an account that was renamed. Entity resolution is unglamorous and it determines whether a graph is useful or actively misleading.

Silent divergence. Two agents build separate memories of the same events and slowly disagree. Nobody notices until their answers conflict in front of a customer.

None of these are exotic. They are the predictable consequences of storing facts about a changing world, and the systems that handle them well are the ones that treated forgetting, provenance, and permissions as first-class from the start rather than as later additions.

How do you evaluate agent memory?

Feature lists converge quickly. These are the questions that separate systems in production.

What gets written, and who decides? Automatic extraction is convenient and lossy. Explicit writes are precise and burdensome. Most systems do some of both, and the balance determines how much junk accumulates.

How does it handle contradiction? When a new fact conflicts with an old one, does the system overwrite, keep both, or record a succession? This is the single biggest differentiator, and the one most likely to produce wrong answers if handled badly.

Can it answer structured questions? Enumeration, counting, negation, and multi-hop traversal are the tests. Similarity search fails all four. If your users ask "how many" or "which ones haven't," a vector store alone will disappoint them.

How does it forget? Memory that only grows becomes slower, more expensive, and more contradictory. Pruning, decay, and retirement of outdated facts are features, not omissions.

Does it respect permissions? In a single-user product this does not arise. In an enterprise it is the whole game: a memory system that surfaces a fact to someone who should not see it has created a data-leak path that did not exist before.

What is the retrieval cost per query? Graph traversal and re-ranking are not free. Measure the tokens and latency added per call, not just the accuracy improvement.

How do you implement agent memory in practice?

Most of the difficulty is not in the storage layer. It is in deciding what goes in and what comes out.

The write path

Something has to decide what is worth remembering. There are three approaches and they fail differently.

Write everything. Store the full transcript and figure it out at retrieval time. Simple, and it degrades badly. Retrieval quality falls as the store grows, contradictions accumulate, and cost rises with no ceiling.

Extract automatically. A model reads the session and emits durable facts. This is what most frameworks do, and it is the right default. The risk is that extraction is a judgement call made by a model with no view of what will matter later, so it will drop things you wanted and keep things you did not.

Write explicitly. The application decides, through a tool the agent calls or a rule the developer wrote. Precise, and it puts the burden on you to anticipate what matters.

In production, hybrids win: automatic extraction as the baseline, explicit writes for the handful of things you know are load-bearing. Whichever you choose, log what got written. The first time an agent asserts something wrong, you will want to know which session produced it.

The read path

Retrieval is where memory either helps or quietly poisons the output.

The naive implementation fetches the top-k most similar memories and prepends them. This fails in two directions. Fetch too few and the agent misses the relevant fact. Fetch too many and you have filled the context with near-misses that compete with the actual question, which degrades answers and costs tokens.

Better implementations filter before they rank: scope by user, by entity, by time window, and only then rank what remains. This is the practical argument for structure. A graph lets you say "facts about this account, valid now" before similarity ever enters the picture, and graph databases exist precisely because that kind of constrained traversal is what relational and vector stores both handle poorly.

The forgetting path

The path teams skip, and the one that determines whether the system still works in a year.

Memory that only accumulates gets slower, more expensive, and more self-contradictory. Three mechanisms matter: decay, where unused memories lose retrieval priority; succession, where a newer fact supersedes an older one with the history preserved; and retirement, where facts about things that no longer exist are removed outright.

Systems without a forgetting story do not fail immediately. They fail after six months, which is worse, because by then the memory is load-bearing.

How does memory interact with tools and context?

Memory is one of three ways an agent gets information, and conflating them causes bad architecture decisions.

Tool calling fetches live data on demand. It is authoritative and current, and it is slow and sequential. MCP has become the standard for exposing tools, and the protocol documentation is explicit that it governs access rather than retention.

Retrieval pulls from a prepared corpus. Fast, but limited to what was indexed in advance.

Memory supplies what the system already learned. Fastest of the three, because the comprehension work happened when the information arrived rather than at query time.

The right design uses all three and knows which is which. Live account balance is a tool call, never memory. Company policy is retrieval. Who owns this account and what they decided last quarter is memory.

A common failure is using tool calls where memory belongs. An agent that re-queries five systems every session to rebuild the same picture is paying full latency and token cost for something that did not change. Google Cloud's guidance on connecting models to systems is worth reading alongside any memory design for exactly this reason: access and retention are different problems with different costs.

A note on benchmarks

The category has benchmarks, including Deep Memory Retrieval and LongMemEval, and results get quoted heavily in vendor marketing. They are worth reading and worth discounting.

Two reasons. The benchmarks are young and narrow relative to real workloads, and most published comparisons come from a vendor whose system wins. Independent write-ups such as Vectorize's framework comparison are more useful than any single vendor's blog, but the honest position is that a week of your own traffic will tell you more than any published number.

Where does per-agent memory stop being enough?

Every framework above solves memory for an agent. That is a real problem and they solve it well.

The problem it does not solve is memory for an organization. A company running twenty agents with per-agent memory has twenty disconnected partial pictures. The sales agent knows what sales said. The support agent knows what support said. Neither knows that the churn risk flagged in a support ticket is the same account the sales agent is forecasting to renew.

Organizational memory is a different shape:

• It spans systems, not just conversations. Meetings, tickets, documents, and messages, not only what someone typed at an agent.
• It is shared, so every person and agent draws on one picture rather than each accumulating their own.
• It inherits permissions from the source systems, so what a given person or agent can see is governed rather than assumed.
• It is maintained, continuously updated and pruned rather than growing until it contradicts itself.

That is the layer Coworker AI works at. It connects to 50+ tools, maintains organizational memory across them, and runs agents that act on what they find. It exposes that memory over MCP too, so the context is available inside whatever AI tools your team already uses. Plans are Pro at $29.99 per user per month, Max at $149.99, and Enterprise pricing on request.

Book a demo if you want to see what an agent does differently when the memory is organizational rather than per-agent.

Frequently asked questions

What is AI agent memory?

AI agent memory is the mechanism that lets an agent retain information across turns and sessions rather than starting from zero each time. It covers short-term working context inside the current session and long-term storage that persists between them, including episodic records of events and semantic facts distilled from those events.

Is a large context window the same as memory?

No. A context window is short-term working space that empties when the session ends. Memory is what survives that. A model with a very large window still cannot answer "has this customer raised this before" unless something persisted the earlier interaction.

What is the difference between episodic and semantic memory?

Episodic memory stores discrete events, such as a specific meeting or message, time-stamped and specific. Semantic memory stores distilled facts and relationships extracted from those events, such as who owns an account. Episodic memory supports recall, semantic memory supports reasoning across things no single event contains.

Should agent memory use vectors or a knowledge graph?

Vectors are better for open-ended semantic recall over raw text. Graphs are better for questions about entities, relationships, counting, and multi-hop traversal. Most production systems use both, because similarity search cannot answer structured questions and graph traversal is not a good fit for "find something like this."

What is a temporal knowledge graph?

A knowledge graph where facts carry validity periods, so a new fact can supersede an old one without deleting it. This lets the system answer questions about the present correctly while retaining history, and it avoids the contradiction problem that arises when a store simply accumulates facts that conflict.

How is agent memory different from RAG?

Retrieval-augmented generation fetches relevant documents at query time from a corpus that was largely fixed in advance. Agent memory is written by the agent's own experience and accumulates over time. RAG answers from documents; memory answers from what has happened. Many systems use both.

Does agent memory create a security risk?

It can. A memory layer that stores facts extracted from permissioned sources and then serves them without inheriting those permissions becomes a route around access control. Any enterprise deployment should confirm that permissions travel with the stored facts, which connects directly to broader data privacy compliance.

Related reading

• Mem0 vs Zep vs Letta: Agent Memory Frameworks Compared
• What Is MCP (Model Context Protocol)?
• What Is an LLM Gateway?
• AI Agent Orchestration Platforms, and How to Choose
• Agent Workflows Explained
• LLM Agent Architecture
• Enterprise AI Agents: What Business Leaders Need to Know
• Agentic AI vs Generative AI
• 16 Ways to Use AI For Knowledge Management
• 25 Agent Performance Metrics You Need to Track
• 8 Best AI Agent Builders for Enterprise Teams
• Ensuring Data Privacy Compliance When Using Enterprise AI