Every AI agent you build today operates inside a finite context window. That window is the agent's entire working universe: what the user just said, what tools it just called, what the system prompt contains. When the window fills or the session ends, everything disappears. The agent starts fresh.

"long-running agents still have goldfish memory — every new context window = new intern who forgot everything from yesterday" — @TheAhmadOsman

The fix is architectural, not cosmetic. Bigger context windows help at the margins, but they don't change the fundamental structure. What changes the structure is a memory layer that lives outside the context window — one that stores information durably, structures it for retrieval, and injects the right pieces at the right time.

This post covers how that architecture actually works: the types of memory your agent needs, the storage approaches that support each type, the retrieval strategies that bring the right context forward, and the patterns that extend this to multi-agent systems.

The Three Types of Memory Your Agent Needs

Memory research distinguishes three types of long-term memory, and the distinction maps cleanly onto what agents need in practice.

Episodic Memory: What Happened

Episodic memory is the record of events. For an AI agent, that means: what did this user ask in the last session? What debugging path did they go down? What decision did they make, and when?

This is the most intuitive type of memory to implement — save a summary at the end of each session, load recent summaries at the start of the next. But episodic memory alone has a scaling problem. A store of 500 session summaries is too large to inject wholesale into every context. The agent needs to retrieve selectively: surface the episodes that are relevant to what the user is working on right now.

That selective retrieval is where storage and retrieval architecture start to matter.

Semantic Memory: What Is Known

Semantic memory is extracted, structured knowledge — facts and relationships that have been distilled out of raw events. Not "in session 47 we talked about the rate limiter," but "the rate limit is 100 requests per minute, enforced by the Redis-backed middleware in api/middleware/rateLimit.ts."

Semantic memory is more compact than episodic memory and more directly useful. When an agent encounters a question about rate limiting next month, a semantic memory containing that fact is immediately applicable — no reasoning required to extract the key detail from a session transcript.

The challenge is building semantic memory. It requires a pipeline that reads episodic content and extracts structured facts — entities, relationships, values. That extraction can be automated through LLM-based parsing, but the output needs to be stored in a way that supports both semantic search and structured lookup.

Procedural Memory: How to Act

Procedural memory covers behavioral patterns: how this user prefers to receive code (with tests, without explanations), what conventions the codebase uses (TypeScript strict mode, Result types), what the agent should always or never do.

In practice, procedural memory is often implemented as a set of preference records or instruction sets that get injected into the system prompt at session start. They're relatively static once established, but they shape every response the agent gives. Getting them right — and keeping them current as preferences evolve — has an outsized effect on agent quality.

Named memories work well for this. A preference document keyed by user ID is deterministic to retrieve, and updates supersede previous versions automatically.

How Agents Store Memory: Three Architectural Approaches

Given those three memory types, how do you actually store them? Three dominant approaches exist in production systems today, each with distinct trade-offs.

Approach 1: Vector Database (Pure Semantic Search)

The simplest approach: convert each memory to a vector embedding and store it in a vector database. At retrieval time, embed the current query and find the nearest neighbors.

Vector databases excel at episode recall. "What did we discuss about payment processing?" surfaces memories whose text is semantically similar to that query. The implementation is straightforward — embed, store, query — and the tooling ecosystem is mature.

The limitation is structural. Vector databases find memories that say similar things. They don't find memories that are connected through shared concepts when the language doesn't overlap. A memory about your Redis configuration and a memory about your rate limiter may be deeply connected — same system, same team, same infrastructure decision — but if neither mentions the other's keywords, vector search won't surface that connection.

At small memory set sizes, this limitation is invisible. At scale — hundreds or thousands of memories across months of work — it becomes the primary source of retrieval gaps.

Approach 2: Knowledge Graph (Entity and Relationship Traversal)

A knowledge graph stores memories as nodes and extracts the relationships between them: entities (technologies, people, systems), facts (configuration values, thresholds, constraints), and semantic links (this memory is about the same system as that memory; this decision supersedes that one).

At retrieval time, a query doesn't just find similar text — it traverses the graph. A search for "rate limiting" finds memories explicitly about rate limiting, and also follows edges to find memories about the Redis configuration backing the rate limiter, the API gateway configuration that calls into it, and the infrastructure team decision that set the original thresholds.

Multi-hop traversal ("what are all the systems that depend on this database, and what do we know about each of them?") is structurally possible with a graph and structurally impossible with a vector store alone.

The trade-off is extraction quality. The graph is only as useful as the entities and relationships that have been extracted into it. Poor extraction produces a sparse graph with few useful connections. Good extraction — precise entities, clean relationships, accurate facts — produces a graph that finds what vector search misses.

Approach 3: Hybrid (Graph + Vectors, Multi-Signal Retrieval)

The production-ready approach combines both. Every memory gets a vector embedding for semantic similarity and gets processed through an extraction pipeline that populates the knowledge graph with entities, facts, and relationships.

At retrieval time, multiple signals run in parallel:

Vector similarity: what memories are semantically close to this query?
Keyword matching: what memories contain the specific terms in this query?
Entity traversal: what memories share entities with what the query mentions?
Topic co-occurrence: what memories cluster around the same topics?
Fact matching: what memories contain structured facts relevant to this query?

The results from all signals merge into a single ranked list. A query for "authentication" finds memories that say "authentication," memories connected to entities like JWT or OAuth, and memories whose extracted facts reference token expiration or signing keys — even when the text doesn't overlap.

This is what MemNexus does. Search runs five signals simultaneously and returns a unified result set, each result annotated with which signals matched:

mx memories search --query "authentication approach" --explain
# Result: "JWT middleware configuration"
# Matched via: entity connection (JWT), extracted fact (24h expiry), topic overlap (security)

Data Flow: From Raw Input to Contextual Retrieval

Here's how a memory moves through the system from creation to retrieval.

1. Ingestion

A memory enters the system as natural-language content — a session summary, a decision record, a debugging note. It doesn't require structured input.

mx memories create \
  --content "Switched from session-based auth to JWT. Tokens expire in 24h. Refresh token rotation enabled. Decision made in response to scaling issues with session store under load." \
  --conversation-id "conv_auth_migration"

2. Extraction

Immediately after ingestion, an extraction pipeline processes the content. It identifies entities (JWT, session store, refresh token), extracts structured facts (expiry: 24h, rotation: enabled), and assigns topics (authentication, security, infrastructure, migration). These outputs populate the knowledge graph — creating nodes for each entity, edges between the memory and those nodes, and edges between related entities.

You write natural language. The extraction pipeline produces the structure. The graph accumulates as more memories arrive, becoming richer and more connected over time.

3. Retrieval

When the agent needs context, it issues a search. The retrieval layer runs multiple signals against the combined vector + graph store and returns a ranked result set. Results include not just content but metadata: which entities matched, what facts were found, whether the memory is current or superseded by a newer one.

mx memories search --query "current authentication implementation"
# Returns: JWT migration memory (current)
# Excludes: session-based auth memory (superseded)

4. Context Assembly

The retrieved memories get assembled into a context block and injected into the agent's system prompt or tool response. The agent now has relevant facts, prior decisions, and entity relationships available — without those things consuming the context window for the entire session.

The session context window is for active reasoning. The memory layer is for accumulated knowledge. Keeping these separate is what makes both work well.

Retrieval Strategy Matters as Much as Storage

Developers typically spend more time thinking about storage than retrieval. In practice, retrieval strategy is where the quality difference between memory systems shows up.

Three retrieval strategies are worth understanding:

Semantic search finds memories whose meaning is close to the query. Good for open-ended recall ("what do we know about payment processing?"), weak when the language in stored memories doesn't match the query language.

Graph traversal follows entity and relationship connections from the query to related memories. Good for finding connected context ("everything related to the Redis cluster"), requires a populated graph.

Hybrid re-ranking runs multiple retrieval passes and re-ranks the merged result set by relevance. This is what production systems should use. It's slower than a single vector query but produces meaningfully better results because it surfaces what each individual pass misses.

One retrieval detail that matters disproportionately: supersession filtering. Memory stores evolve. Decisions change. Preferences update. If your retrieval returns both the original decision and the revised one, your agent has to reason through which is current — burning tokens on a problem your retrieval layer should solve.

Tracking supersession relationships in the graph and filtering by default is a small implementation cost with large downstream benefits. The agent receives only current information and can reason with it directly.

Architectural Patterns: Single-Agent and Multi-Agent

The memory architecture described above applies to a single agent serving a single user. Two additional patterns extend it.

Pattern: Scoped Memory Partitioning

A single memory store can serve multiple users and multiple projects if memories are tagged with scope at creation time and retrieval is filtered by scope. This is more practical than running separate memory stores per user.

The practical implication: retrieval queries should always include scope context. An agent that retrieves without scope filtering will surface memories from other users or projects, degrading result quality and potentially leaking context across boundaries.

Pattern: Shared Memory Across Agent Teams

When multiple agents operate on the same task — a research agent, a coding agent, and a review agent working in parallel — they need access to shared context without duplicating it.

The memory layer is the shared substrate. Any agent can write a memory; any agent can retrieve it. The knowledge graph connects memories regardless of which agent created them. A memory the research agent saved about a third-party API's rate limits is immediately available to the coding agent implementing the integration.

# Research agent saves a finding
mx memories create \
  --content "Stripe webhook events can arrive out of order. Implement idempotency keys on all event handlers. Events may replay up to 24h after original delivery." \
  --conversation-id "conv_stripe_research"

# Coding agent retrieves it later, independently
mx memories search --query "Stripe webhook handling"
# Returns the research finding — shared knowledge, no duplication

This is where the knowledge graph's multi-hop traversal pays off most clearly. A coding agent looking at a payment integration retrieves memories connected to Stripe, which connects to webhook handling, which connects to the idempotency research — across an entity graph populated by multiple agents operating independently.

For a practical walkthrough of this pattern, see Teams of Agents, Shared Memory.

What MemNexus Builds on This Architecture

MemNexus implements the hybrid graph + vector architecture described above. Every memory you save goes through entity extraction, fact extraction, and topic classification automatically. Search runs five signals in parallel — semantic similarity, keyword matching, entity traversal, topic overlap, and fact matching — and returns a merged, re-ranked result set with supersession filtering on by default.

You interact with it through the CLI, SDK, or MCP. If your tool supports MCP — Claude Code, Cursor, Windsurf, GitHub Copilot, and others do — it connects without any custom integration code:

# Save a memory from the CLI
mx memories create --content "Switched auth to JWT, 24h expiry, refresh rotation enabled."

# Or let your MCP-connected agent call create_memory directly during a session

# Retrieve with graph-aware search
mx memories search --query "authentication implementation" --explain

# Build full context before a session
mx memories build-context --context "working on API auth middleware"

The build-context command replaces three or four separate searches at session start. One call returns active work, relevant facts, and recurring gotchas — pulling from episodic, semantic, and procedural memory simultaneously.

The Properties That Make a Memory Architecture Production-Ready

If you're evaluating or building a memory layer for your agents, these are the properties that distinguish systems that hold up at scale from those that work in demos:

Automatic extraction over manual tagging. A memory system that requires structured input is a system developers won't use consistently. Extraction from natural language is the baseline.

Supersession tracking. Decisions change. A system without versioning surfaces outdated information and makes agents less reliable over time, not more.

Multi-signal retrieval. Pure vector search has a recall ceiling. Hybrid retrieval with entity traversal and fact matching raises that ceiling substantially.

Scope isolation. In multi-user or multi-project settings, retrieval must be filterable by scope. Unscoped retrieval is a correctness and privacy problem.

Low retrieval latency. Memory retrieval happens in the hot path — before or during agent inference. Latency here adds directly to end-user wait time.

Transparent result provenance. When a result surprises you, you need to know why it was returned. Explain modes and match-source annotations are what make retrieval debuggable.

Where Memory Architecture Is Heading

The current state of the art is reactive retrieval: the agent requests context when it needs it. The industry is moving toward anticipatory retrieval — memory layers that surface relevant context before the agent asks, based on what the agent is currently doing.

As agents call tools, edit files, or reference entities, a memory layer with graph structure can identify what's connected to those actions and pre-populate the context with what's likely to matter next. This would remove retrieval latency from the hot path and reduce the cognitive overhead of explicitly querying for context.

The broader trend is a tighter feedback loop: the agent's behavior informs what gets stored, and what gets stored informs the agent's future behavior. That's the loop that makes agents improve over time rather than resetting.

MemNexus is the persistent memory layer for AI agents and coding assistants. It works with Claude Code, Cursor, Windsurf, GitHub Copilot, and any MCP-compatible tool — no custom integration required. We're currently in gated preview.

Join the waitlist to get early access and bring persistent, graph-aware memory to the agents you're building.

AI Agent Memory Architecture: How Persistent Memory Actually Works Under the Hood