TL;DR — Distributed AI agents need more than a context window; they need a memory substrate that spans working, episodic, and semantic tiers across many nodes. This post walks through the architecture of such systems — from write paths and consistency trade-offs to vector stores, TTLs, and the orchestration patterns that keep agents coherent at scale.

Why Agents Need a Memory Substrate, Not Just a Prompt

Anyone who has shipped a non-trivial one has felt the pain: a multi-agent workflow runs beautifully in the first turn, then forgets what tool B learned from tool A by turn three. The naive fix — pasting prior outputs back into the prompt — collapses once you have more than two agents, more than a few minutes of runtime, or any kind of long-horizon task.

The problem isn’t that the model is forgetful. The model is, by design, stateless per call. Memory is an infrastructure concern, and like all infrastructure concerns, it has a shape that determines whether the system scales or suffocates.

What we want is an autonomous memory system: a set of services that agents write to and read from without orchestrating that I/O themselves, that handle eviction, consolidation, and retrieval, and that remain consistent enough across a fleet of agents to keep the swarm coherent.

This is closer to building a tiny distributed database than it is to engineering prompts. That’s the lens we’ll use.

The Three Tiers of Agent Memory

Production-grade agent memory systems almost always decompose into three tiers, mirroring the cognitive science distinctions between working, episodic, and semantic memory.

Working memory: the per-turn scratchpad

Working memory holds the state of the current task — the conversation so far, intermediate tool outputs, the plan the agent is executing. Latency budget here is tight: sub-100ms reads are common, sub-10ms is the goal. Storage is usually in-process:

  • A Redis or DragonflyDB instance for the agent’s session.
  • An in-memory ring buffer for tool call history with a hard cap on tokens.
  • A Postgres row holding structured scratch state for long-running workflows.

The defining property is scope: working memory is visible to one agent (or one agent + its direct collaborators) and dies when the task ends.

Episodic memory: the timeline of what happened

Episodic memory is a chronologically ordered log of significant events — tool calls, decisions, user interactions, errors. It’s the “what did we do and why” layer. Storage choices lean on append-only systems:

  • Kafka or Redpanda topics partitioned by agent ID or tenant.
  • A wide-column store like Cassandra for high-cardinality event retention.
  • Object storage (S3, GCS) with Parquet files partitioned by date for cheap long-tail archival.

Retrieval here is usually time-windowed: “give me everything agent-7 did between 14:00 and 14:05.”

Semantic memory: the distilled knowledge

Semantic memory is the compressed, retrieval-friendly layer. Think vector embeddings of past interactions, knowledge triples, summaries of recurring patterns. This is where RAG lives. Storage is purpose-built:

  • A vector database (Qdrant, Milvus, Weaviate, Pinecone) for similarity search.
  • A knowledge graph (Neo4j, Memgraph) for relational reasoning.
  • A traditional search index (Elasticsearch, Meilisearch) for lexical recall.

The defining property is derivation. Semantic memory is built from episodic memory, typically by an asynchronous consolidation job.

The Write Path: What Gets Committed Where

The most consequential design decision is the write path. Concretely: when an agent takes an action, what gets stored, where, and with what consistency?

A pattern that has worked well in production looks like this:

  1. The agent emits a structured event after every meaningful action — a tool call result, a sub-task completion, an error. This is a small JSON document with a stable schema.
  2. The event goes to a Kafka topic, partitioned by tenant or session.
  3. A memory writer service consumes the stream and fans out:
    • Appends the raw event to episodic storage (Cassandra / object storage).
    • Embeds the event and writes to the vector store with metadata for filtering.
    • Updates working memory if the event is part of an active session.
  4. A separate consolidation job runs periodically — every few minutes — to roll up episodic events into summaries, prune old vector entries, and surface recurring patterns as new semantic entries.

The crucial discipline is that the agent itself never writes directly to a vector DB or knowledge graph. It only emits events. Everything downstream is handled by services that can retry, batch, and reorder.

An example event schema

{
  "event_id": "01J8X3...",
  "session_id": "sess_4f8a",
  "agent_id": "agent_research_03",
  "timestamp": "2026-09-03T11:00:26.333Z",
  "type": "tool_call_result",
  "tool": "web_search",
  "input": {"query": "Q3 revenue ACME"},
  "output_summary": "ACME reported $1.2B, up 14% YoY",
  "success": true,
  "tokens_used": 412,
  "trace_id": "otel_abc123"
}

That single record flows to every tier.

Consistency: How Coherent Does the Swarm Need to Be?

This is where distributed-systems instincts start paying off. Agents reading from a memory tier expect a certain consistency guarantee, and getting it wrong produces subtle, infuriating bugs.

Read-your-writes within a session

Within a single agent’s session, the agent should see its own writes immediately. Working memory must therefore offer read-your-writes consistency. That almost always means same-region, single-leader storage — Redis, a single Postgres primary, or a strongly consistent key-value store like etcd.

Causal consistency across collaborating agents

When agent A hands off to agent B, agent B must see what agent A wrote. This is causal consistency: if A’s event logically preceded B’s start, B’s read must reflect A’s write. In practice this means:

  • Session handoffs include a vector clock or a logical timestamp.
  • The receiving agent’s first action is to read from the canonical session store, not from a stale local cache.
  • Kafka’s per-partition ordering gives you causal consistency “for free” within a session, as long as you route by session key.

Eventual consistency across the semantic tier

The vector store and knowledge graph can be eventually consistent. A 30-second lag between an event being written to episodic storage and appearing in the vector index is fine — agents searching for that event aren’t searching for it yet. This is what makes the architecture tractable: the layer that needs to be fast and fresh (working memory) is small, and the layer that needs to be deep and rich (semantic memory) is allowed to be slow.

The pattern is identical to what you’d build for any CQRS system, and the original CQRS documentation by Greg Young is still the cleanest articulation of why separating the write and read models pays off.

Retrieval: How Agents Find What They Need

A memory system you can’t query is a write-only log. Retrieval deserves as much architectural thought as storage.

Hybrid retrieval is the default

Pure vector search is not enough. Production agents combine:

  • Dense retrieval over embeddings for semantic similarity.
  • Sparse retrieval (BM25 or similar) for exact terms — error codes, names, IDs.
  • Metadata filtering by tenant, time window, agent type.
  • Graph traversal for “what else is related to this entity.”

A common production pipeline is what the Qdrant hybrid search docs describe: pre-filter by metadata, run vector + BM25 in parallel, fuse scores with reciprocal rank fusion.

Retrieval-augmented consolidation

The consolidation job uses retrieval against itself. When summarizing the last hour of agent activity, it doesn’t re-read every event — it samples, embeds, and retrieves the most representative ones. This is the same trick used in Haystack’s summarization pipelines and in production RAG systems at scale.

Caching the obvious cache

Many retrieval queries are repetitive — “what tools does this agent have access to?”, “what’s the user’s role?”. A small in-process cache, or a shared Redis cache keyed by query hash + session, eliminates 80%+ of retrieval traffic in typical workloads.

Orchestration Patterns in Production

Memory is useless without orchestration that uses it well. Three patterns show up repeatedly.

Pattern 1: The handoff envelope

When one agent calls another, the handoff includes not just the request but a memory envelope — a snapshot of relevant context pulled from the receiving agent’s previous memory tiers. The receiving agent can accept or reject pieces of the envelope. This is how you get seamless multi-agent flows without every agent re-deriving context from scratch.

class HandoffEnvelope:
    session_id: str
    caller: str
    request: str
    relevant_events: list[Event]      # from episodic memory
    relevant_facts: list[Fact]        # from semantic memory
    working_state: dict               # from working memory
    constraints: list[Constraint]

Pattern 2: The shared semantic layer

For tightly coupled multi-agent systems — a planner + executor + critic, for instance — the semantic memory tier is shared. All three agents write to and read from the same vector store, scoped by a shared workflow_id. This avoids the “telephone game” failure mode where the planner’s understanding of the goal diverges from the executor’s.

Pattern 3: The memory curator

In long-running deployments, a dedicated curator agent (or a non-agent background service) periodically:

  • Identifies facts that are now stale and evicts them.
  • Merges redundant entries.
  • Promotes frequently-retrieved episodic events into semantic facts.
  • Demotes semantic facts that are no longer being retrieved.

This is the equivalent of garbage collection, and skipping it is the single most common reason agent memory systems become useless after a few weeks in production.

Failure Modes Worth Designing For

A few failure modes show up often enough to be worth naming.

Memory poisoning. A compromised or buggy agent writes misleading events that get embedded and retrieved by other agents. Mitigation: signed events, per-agent trust scores, and quarantine for low-trust writers. OWASP’s LLM Top 10 lists several relevant threats.

Stale context cascade. An agent retrieves a fact that was true an hour ago, builds a plan on it, and downstream agents inherit the stale assumption. Mitigation: TTLs on semantic entries, explicit “as of” metadata, and a freshness score surfaced in retrieval.

Tier drift. Working memory and episodic memory disagree about what’s currently happening, often because a write to one succeeded and the other failed. Mitigation: outbox pattern — events are written to an outbox in the same transaction as the working memory update, then relayed to the episodic tier asynchronously.

Retrieval amplification bias. Agents preferentially retrieve and act on whatever they retrieve most easily. If the vector index is dominated by recent events, older but more relevant facts get ignored. Mitigation: recency penalties, mandatory time-window diversity in results, and human-in-the-loop sampling.

Key Takeaways

  • Agent memory is an infrastructure problem, not a prompt problem. Treat it like a distributed database.
  • Three tiers — working, episodic, semantic — each with different latency, consistency, and storage characteristics, cover almost every production scenario.
  • The write path should be event-sourced: agents emit structured events, downstream services handle persistence to each tier.
  • Consistency is per-tier: read-your-writes for working memory, causal for cross-agent handoffs, eventual for the semantic layer.
  • Retrieval is hybrid by default — dense + sparse + metadata + graph — and benefits from aggressive caching.
  • A dedicated curator process is what keeps a memory system useful past its first few weeks.
  • Design explicitly for memory poisoning, stale context cascades, tier drift, and retrieval bias.

Further Reading