The Complete AI Agent Architecture: From Perception to Action

TL;DR — An AI agent is a loop: perceive the world, reason about the next step, pick a tool, execute, observe the result, and repeat. Production-grade agents add three layers the demos skip — durable memory, structured tool routing, and explicit guardrails for failure modes. Get those right and your agent scales; skip them and you ship a demo.

There’s a recurring pattern in the AI agent discourse: a polished demo video of a model clicking through a browser, a five-tweet explainer with overlapping circles, and then silence when someone asks how it actually fails. Most production teams I talk to are not struggling to get an LLM to do something. They’re struggling to get it to stop doing the wrong thing reliably, repeatedly, and within budget.

This post is the architecture I wish someone had handed me the first time I shipped an agent into a real pipeline. It’s not a survey of every framework — it’s the layer cake that any serious agent system ends up with, regardless of whether you build it on LangGraph, the OpenAI Agents SDK, CrewAI, or hand-rolled code. We’ll walk top-down from perception to action, then look at the cross-cutting concerns — memory, observability, guardrails — that determine whether your agent survives contact with real users.

Why “Just an LLM” Isn’t Enough

A language model is a function from text to text. Give it a prompt, get a completion. That’s powerful for drafting an email but useless for “reconcile the last 24 hours of Stripe payouts against the support inbox.” The moment a task requires action — calling an API, writing a file, opening a ticket — the LLM stops being the system and becomes one component of it.

The leap from chatbot to agent is fundamentally a leap in agency. The model now chooses what to do next based on observed outcomes, not just user input. That changes the architecture in three concrete ways:

1. State becomes external. The conversation history isn’t enough; you need a durable record of goals, intermediate results, and tool outputs.
2. Tools become first-class. Every action the agent can take needs a schema, a permission boundary, and an error contract.
3. Loops replace single turns. Agents iterate. A single bad tool call shouldn’t end the run, and a successful one shouldn’t end the reasoning prematurely.

Everything that follows is downstream of those three shifts.

The Five-Layer Mental Model

When I sketch an agent on a whiteboard, I draw five layers stacked vertically. They map cleanly onto most production systems I’ve seen, from Anthropic’s building effective agents guidance to the runtime architecture described in the LangGraph docs.

┌─────────────────────────────────────┐
│ 5. Action / Output Layer            │  ← side effects in the world
├─────────────────────────────────────┤
│ 4. Tool & Integration Layer         │  ← typed interfaces to the world
├─────────────────────────────────────┤
│ 3. Reasoning / Planning Layer       │  ← the LLM loop
├─────────────────────────────────────┤
│ 2. Memory & Context Layer           │  ← state across turns
├─────────────────────────────────────┤
│ 1. Perception / Input Layer         │  ← raw signals from the world
└─────────────────────────────────────┘

Let’s walk through each, then look at how they fit together.

Layer 1: Perception and Input

Perception is the unsexy half of every agent demo. The model gets the glory; the perception layer decides what the model gets to see.

In production, inputs are not clean chat messages. They’re a soup of:

• User messages across multiple channels (web, Slack, email, voice transcripts).
• Structured events from upstream systems (“customer 4421 just hit their API rate limit”).
• Retrieved documents from a vector store.
• Tool outputs from previous turns.
• System state — what the user is currently looking at, what time it is, what plan they’re on.

A good perception layer normalizes these into a single context window with consistent ordering, trims aggressively (most agent failures are context-window failures, not model failures), and annotates with metadata the reasoning layer will need. That last point is underrated: a retrieved chunk without a source citation isn’t usable in a grounded agent. A timestamp on a system event changes how the planner reasons about it. Perception is where you earn the right to ask the LLM a coherent question.

# A perception layer in 2026 is usually a builder, not a string concat
from langchain_core.messages import SystemMessage, HumanMessage

def build_context(user_msg: str, history: list, retrieved: list, state: dict):
    messages = [
        SystemMessage(content=f"You are an agent for tenant {state['tenant_id']}. "
                              f"Current tools: {state['enabled_tools']}. "
                              f"Budget remaining: {state['tool_calls_left']} calls."),
    ]
    # Retrieved docs go in as structured content, not flat strings
    for doc in retrieved[:5]:
        messages.append(SystemMessage(
            content=f"<doc source='{doc.source}' ts='{doc.timestamp}'>{doc.text}</doc>"
        ))
    messages.extend(history[-10:])  # last 10 turns
    messages.append(HumanMessage(content=user_msg))
    return messages

Layer 2: Memory and Context

Memory is the layer where most homegrown agents die. There are three distinct kinds, and they need different storage:

The mistake teams make is conflating these. You can’t dump an entire Postgres row into a vector store and expect good retrieval, and you can’t shove long-term preferences into the system prompt — they’ll get evicted the moment the context window fills.

A clean architecture separates them:

# memory.yaml — a sketch of the storage split
memory:
  working:
    location: "in-context"
    budget_tokens: 4000
  episodic:
    location: "postgres://agent-sessions"
    schema: "thread_id, role, content, tool_calls, ts"
    retention_days: 30
  semantic:
    location: "pinecone://user-facts"
    embedding_model: "text-embedding-3-small"
    write_policy: "extract_on_close"
    read_policy: "top_k=8, filter by tenant_id"

The write policy is the part most teams skip. Memories don’t write themselves. You need an extraction step — either a cheaper model call or a rule-based pass — that converts “the user mentioned they’re migrating from Postgres 13 to 16 next quarter” into a persistent fact the agent will see next session. Without it, you have a chatbot with amnesia.

For a deeper treatment of how memory fits into agent design, the MemGPT paper is still the cleanest reference, even though the implementation landscape has moved on.

Layer 3: Reasoning and Planning

This is the LLM loop. It’s also the layer where the most architectural decisions get made, mostly because “reasoning” is doing a lot of work.

In practice, agents reason in one of three modes:

• ReAct (reason + act): The model emits a thought, then a tool call, then observes the result, then repeats. This is the default mode in most frameworks and the right starting point for 80% of tasks.
• Plan-and-execute: The model first produces a multi-step plan, then executes it step by step. Better for long-horizon tasks where you want a human to review the plan before any side effects fire.
• Routing / multi-agent: The entry point is a classifier that picks a specialist sub-agent. Common in customer support flows where billing questions and technical questions need different tools and different prompts.

The choice between them is a reliability vs. flexibility trade-off. ReAct is flexible but unpredictable — the model might decide to skip a step. Plan-and-execute is predictable but rigid — if the plan is wrong, the run fails. Multi-agent systems buy you specialization at the cost of coordination overhead and a new failure mode (agents arguing with each other or looping).

# ReAct loop, the bones of most agents
def agent_loop(state, max_steps=12):
    for step in range(max_steps):
        decision = llm.with_structured_output(Action).invoke(state.messages)
        if decision.action_type == "final_answer":
            return decision.answer
        result = execute_tool(decision.tool, decision.args)
        state.messages.append(ToolMessage(content=result, tool_call_id=decision.id))
        state.spend += cost_of(decision)
        if state.spend > state.budget:
            raise BudgetExceeded()
    raise MaxStepsExceeded()

Two things to notice. First, the state.spend check — every production agent needs a budget, and “budget” is more than dollars. It’s also tool calls, wall-clock time, and tokens. Second, the structured output. Free-form tool calls are a debugging nightmare. Force the model to return a typed schema and your logs become queryable.

For more on when to pick which pattern, the Anthropic guide on building effective agents is the clearest public framing of the trade-offs.

Layer 4: Tools and Integration

Tools are how agents touch the world. They’re also where most security incidents start, which is why this layer deserves more attention than it usually gets.

A well-designed tool has five properties:

1. A typed schema. Not just “here’s a JSON blob” — typed parameters with descriptions the LLM can read and validation the runtime can enforce.
2. A permission scope. Which tenant, which user, which resource. Tools should never inherit the model’s permissions; they should be explicit.
3. An idempotency story. If the agent retries, does the tool double-charge the customer? Use idempotency keys.
4. An error contract. What does the tool return on failure? A structured error the LLM can reason about (“404: invoice not found”) is far better than a stack trace.
5. A cost label. Not just dollars — latency, downstream load, rate-limit cost. The orchestrator needs this to budget.

// A tool definition that respects all five properties
{
  name: "refund_invoice",
  description: "Issue a full or partial refund against a Stripe invoice.",
  parameters: {
    type: "object",
    properties: {
      invoice_id: { type: "string", pattern: "^in_[a-zA-Z0-9]+$" },
      amount_cents: { type: "integer", minimum: 1 },
      reason: { type: "string", enum: ["duplicate", "fraudulent", "requested_by_customer"] }
    },
    required: ["invoice_id", "amount_cents", "reason"]
  },
  scopes: ["billing:write"],
  idempotency: { key_from: ["invoice_id", "amount_cents"], ttl_seconds: 86400 },
  cost: { dollars: 0.0, latency_ms_p50: 420, rate_limit_units: 1 },
  errors: {
    "invoice_not_found": "No invoice with that ID exists for this tenant.",
    "already_refunded": "This invoice has already been fully refunded."
  }
}

Tools also benefit from being grouped into registries. A registry is just a versioned collection of tool definitions the orchestrator can load per-tenant or per-environment. Stripe has its own internal registry for its Stripe Agent Toolkit, which is worth reading as a reference for what “tool done right” looks like at scale.

Layer 5: Action and Output

The final layer is where side effects actually happen. The rule is simple: the orchestrator owns this layer, not the model. The model proposes an action; the orchestrator validates it, gates it, and dispatches it.

Concretely, this means:

• A human-in-the-loop checkpoint for irreversible actions above some dollar or risk threshold.
• A queue between the model and the side-effecting system, so you can replay, audit, or cancel.
• A canonical record of every action taken, including the inputs and the model state at the time.

Action log entry
───────────────
run_id:        run_7f3a
thread_id:     thr_882
tool:          refund_invoice
args:          {invoice_id: in_8821, amount_cents: 1500, reason: duplicate}
model_state:   step 4 of 12, spend $0.04
approved_by:   auto (under $50 threshold)
result:        ok, refund_id: re_4491
ts:            2026-09-02T21:08:11Z

This log is not optional. It’s the difference between “we think the agent did the right thing” and “we can prove it.”

Patterns in Production: What Actually Ships

Theory is cheap. Here are the patterns I see in agent systems that are actually running in production at meaningful scale.

The Two-Channel Memory Pattern

Many teams have converged on splitting an agent’s context into two channels: a conversation channel (the human-readable thread) and a scratchpad channel (private notes the model writes to itself). The scratchpad persists across turns and across sessions; the conversation does not.

This is the pattern underlying Devin’s approach and most serious coding agents. The model can leave itself breadcrumbs — “the bug is in the retry logic, not the cache invalidation” — without polluting the user-visible transcript.

The Router-and-Specialist Pattern

Rather than one mega-agent with 50 tools, production systems lean toward a thin router in front of several specialist agents, each with a tight tool set and a focused prompt.

         ┌── billing_agent (5 tools)
router ──┼── support_agent (8 tools)
         └── dev_agent (12 tools, code execution)

This is the same insight that drove microservice adoption: bounded context, bounded failure. A billing agent that hallucinates can’t break the dev agent.

The Evaluator-in-the-Loop Pattern

Every non-trivial agent needs an evaluator. Either a second model call (“did the previous step succeed?”) or a deterministic check (“does the output JSON parse?”). The evaluator’s job is to decide whether the agent should retry, escalate, or proceed.

def evaluate(state, last_result):
    # Deterministic checks first — they're cheap
    if not last_result.ok:
        return "retry" if state.retries < 3 else "escalate"
    # Then a model-based check for subjective quality
    verdict = evaluator_llm.invoke(f"Did this answer the user's question?\n"
                                   f"User: {state.user_msg}\n"
                                   f"Result: {last_result.content}")
    return verdict.action  # "proceed" | "retry" | "ask_user"

This pattern shows up in nearly every mature agent framework, including the evaluator loops in the OpenAI Agents SDK.

Cross-Cutting Concerns

The layers above describe the happy path. The cross-cutting concerns describe whether your agent survives in production.

Guardrails and Failure Modes

Name the failure modes explicitly. The common ones:

• Infinite loops: The agent calls the same failing tool repeatedly. Cap step count and tool call count.
• Context overflow: The agent keeps adding to its own context. Trim aggressively.
• Confused tool selection: The agent calls the wrong tool. Constrain the tool set per turn and use embeddings to retrieve tools, not a flat list.
• Hallucinated parameters: The agent invents an invoice ID. Validate every parameter before execution.
• Prompt injection via retrieved content: A document instructs the agent to ignore prior instructions. Treat retrieved content as untrusted.

Each of these has a known mitigation. None of them are free.

Observability

If you can’t see what your agent is doing, you can’t fix it. The minimum viable observability stack:

• Trace every LLM call with prompt, completion, latency, cost, and tool calls.
• Trace every tool call with inputs, outputs, and errors.
• Sample conversations for human review — automated metrics miss subtle regressions.
• Tag by tenant and user so you can spot per-customer degradation.

Tools like LangSmith, Langfuse, and Arize Phoenix exist because this is a hard problem. Pick one before you ship, not after your first outage.

Cost and Latency Budgets

Every agent step is a network call. An agent that takes 12 reasoning steps with retrieval at each step is a latency disaster. Design with budgets:

• Hard latency budget per turn. If you can’t answer in 8 seconds, escalate or defer.
• Hard dollar budget per session. Track spend in real time; refuse to continue when exhausted.
• Cache aggressively. The same tool call from the same context should hit a cache, not the live API.

A useful rule of thumb from the AWS Builders’ Library applies: budget for the median, design for the tail, and never let a single bad path consume the whole budget.

Key Takeaways

• An AI agent is a loop, not a single call: perceive, reason, act, observe, repeat. Every production agent has this shape regardless of framework.
• Memory is plural. Working, episodic, and semantic memory need different storage and different write policies. Conflating them is the most common architectural mistake.
• Tools deserve the same engineering rigor as a public API: typed schemas, scoped permissions, idempotency, structured errors, and cost labels.
• The orchestrator — not the model — owns side effects. Add human-in-the-loop checkpoints for irreversible actions and log every decision.
• Failure modes are knowable and finite. Cap steps, trim context, constrain tools, validate parameters, and treat retrieved content as untrusted input.
• Observability and budgets are not optional. Trace every call, tag by tenant, and refuse to exceed per-session limits.

Further Reading

• Building Effective Agents — Anthropic
• LangGraph Documentation
• OpenAI Agents SDK
• MemGPT: Towards LLMs as Operating Systems
• Stripe Agent Toolkit Documentation
• Timeouts, Retries, and Backoff with Jitter — AWS Builders’ Library