TL;DR — Production RAG is less about picking an embedding model and more about orchestrating ingestion, retrieval, reranking, and evaluation as a single system. The teams that win treat chunking strategy, hybrid search, and freshness as first-class concerns — and instrument every stage so they can debug “the model hallucinated” all the way down to “the wrong paragraph was retrieved.”

Retrieval-Augmented Generation (RAG) started life as a clever way to ground a language model in private data without fine-tuning. In 2026, it is the default architecture for any product that needs an LLM to answer questions about documents it wasn’t trained on: enterprise search, developer copilots, customer support automation, legal review, and on. The interesting engineering has moved past the naive “embed your docs and stuff the top-k into the prompt” demo. What’s left is the hard part: making the system correct, fast, cheap, and fresh enough that real users trust it.

This post walks through how to architect a RAG system for production. We’ll move from the data pipeline inward, touch the retrieval stack, spend real time on reranking and evaluation, and finish with the failure modes that only show up once you have traffic.

The Anatomy of a Production RAG System

A RAG system is really three pipelines stitched together:

  1. Ingestion — documents arrive, get normalized, chunked, embedded, and written to one or more indexes.
  2. Retrieval — a query comes in, gets embedded, and is matched against those indexes to produce a candidate set.
  3. Generation — the candidates are packed into a prompt, sent to an LLM, and the response is post-processed and returned.

In a toy, all three collapse into a single function call. In production, each one has its own service, its own SLOs, and its own failure modes. Most “RAG is broken” reports are actually one of these three pipelines misbehaving in isolation, not a flaw in the overall architecture.

            ┌────────────────┐
   docs ──▶ │  Ingestion     │ ──▶ vector index + keyword index + metadata store
            └────────────────┘

            ┌────────────────┐
   query ─▶ │  Retrieval     │ ──▶ top-K candidates (hybrid search + rerank)
            └────────────────┘

            ┌────────────────┐
  prompt ─▶ │  Generation    │ ──▶ answer + citations
            └────────────────┘

Keep this diagram in your head as we go, because most decisions are really about which of these three boxes owns a given responsibility. Caching at the query layer belongs to retrieval. Caching at the prompt layer belongs to generation. Document versioning belongs to ingestion. Conflating them is how you end up with stale answers you can’t explain.

Ingestion: Where RAG Goes Wrong Before the User Asks Anything

Most teams underestimate ingestion because it feels like ETL. It is ETL — and ETL is famously where data quality problems are born. Garbage in, hallucination out.

Chunking Is a Modeling Decision

The single highest-leverage choice in ingestion is chunk size and chunk boundary strategy. Embeddings have a fixed context window; the chunk has to fit, and the semantic coherence of what’s in one chunk determines what gets retrieved together.

Three strategies dominate in production:

  • Fixed-size with overlap — easiest. 512-token windows with 64-token overlap. Works fine for dense technical docs, badly for anything with section structure.
  • Structural — split on headings, code fences, table rows. Respects the author’s intent. This is what LangChain’s RecursiveCharacterTextSplitter and Unstructured.io approximate.
  • Semantic — embed sentence-by-sentence and group until similarity drops. Highest quality, highest cost, hardest to debug.

A practical rule of thumb: start with structural splitting on headings, evaluate, and only escalate to semantic chunking if retrieval precision is clearly bounded by chunk boundaries. Don’t start with semantic chunking because it sounds smartest.

Embedding Models Have a Half-Life Too

A common production mistake is to embed everything with the cheapest model you can find, then change models six months later. Embeddings are not interchangeable. If you swap text-embedding-3-small for text-embedding-3-large, or move between providers, you must re-embed everything — the vector space changes and old vectors become noise.

Treat your embedding model as a versioned dependency with a migration plan:

# embeddings.yaml
model: text-embedding-3-large
dimensions: 3072
distance: cosine
indexed_at: "2026-08-14"
schema_version: 3

Store the schema version alongside each vector. When you re-embed, dual-write for a transition period, then cut over. This is the same discipline as a database migration, and it deserves the same tooling.

Metadata Is the Hidden Retrieval Signal

Pure vector search ignores everything except the embedding. In production, the cheapest precision boost is almost always metadata filtering: restrict candidates to documents in the right tenant, in the right time window, of the right document type, before the embedding similarity is even computed.

A few metadata fields that pay for themselves immediately:

  • tenant_id and acl_groups for multi-tenant isolation.
  • source_url and document_id so you can cite and dedupe.
  • updated_at so you can filter by freshness or evict old ones.
  • document_type (runbook, policy, contract, code) so rerankers can weight differently.

Postgres with pgvector and metadata columns is a perfectly reasonable place to start. You do not need a dedicated vector database on day one. You will know you need one when your index no longer fits in RAM, your recall degrades under filter pressure, or your embedding model and your OLTP database stop agreeing on workload characteristics.

Retrieval: Hybrid Search Wins Almost Every Time

Once your documents are indexed, the question is how to find the right ones. The honest answer in 2026 is that no single retrieval method wins everywhere. The systems that perform best in benchmarks — and in user studies — combine at least two.

The Case for Hybrid Retrieval

Dense retrieval (embeddings + ANN search) excels at semantic match: “how do I roll back a deployment” matches “deployment revert procedure” even though the words differ.

Lexical retrieval (BM25, Elasticsearch, the keyword half of PG full-text search) excels at exact match: API names, error codes, part numbers, SKU strings. If a user pastes a stack trace, the variable NullPointerException needs to be matched as a string, not paraphrased into “an error where something was unexpectedly empty.”

In practice, the best retrieval layer is a hybrid that scores documents with both methods and fuses the rankings. Reciprocal Rank Fusion (RRF) is the simplest fusion that works:

def reciprocal_rank_fusion(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Most production systems — Qdrant’s hybrid search, Weaviate’s hybrid query, Elasticsearch’s RRF retriever — expose this directly.

Query Rewriting and Expansion

User queries are short, ambiguous, and full of pronouns. “What about the second one?” is a hard retrieval query. Two patterns help:

  • HyDE (Hypothetical Document Embeddings) — ask the LLM to generate a hypothetical answer, embed that, and search with it. Counterintuitively effective for short queries, as originally shown in Gao et al., 2022.
  • Multi-query expansion — generate 3–5 rephrasings of the query, retrieve for each, then fuse. Cheap, embarrassingly parallel, often the single biggest recall win you can ship in a week.

Both add latency and cost. Both are worth it for systems where retrieval recall is the bottleneck, which is most of them.

Filters and Access Control Belong Here, Not in the LLM

A recurring security anti-pattern is to embed access-control logic in the prompt (“only use information the user is authorized to see”). The LLM will leak. Access control has to be enforced at the index level: every retrieval is restricted to documents whose ACL metadata intersects with the user’s identity. There is no shortcut around this, and the OWASP LLM Top 10 lists prompt-side authorization bypasses for exactly this reason.

Reranking: The Cheapest Precision You Can Buy

If you retrieved 50 candidates with hybrid search, you probably want to send 5 to the LLM. The job of the reranker is to choose those 5 with much higher precision than either retrieval method can give you alone.

Cross-Encoders vs. LLM Rerankers

Two schools:

  • Cross-encoder rerankers — small models like BGE-reranker-v2-m3, Cohere Rerank 3.5, or Jina Rerank. They take (query, document) pairs and score relevance. Sub-100ms per candidate pair. Cheap. Easy to host. This is the default.
  • LLM-as-reranker — feed the candidates to a strong LLM and ask it to rank them. More accurate on complex reasoning queries, ~10–100x more expensive and slow. Worth it for high-stakes queries, wasteful for general traffic.

A pragmatic architecture: rerank the top 50 with a cross-encoder, cut to top 10, then optionally have an LLM pick the top 3–5 for the final prompt. This two-stage cascade is the standard pattern in the BEIR benchmark literature and in real systems.

Position Bias in the Prompt

LLMs read the top of the context window more carefully than the bottom. Empirically, document position in the prompt changes answer quality, even when retrieval is perfect. Counter this with:

  • Randomizing or interleaving the order of retrieved chunks during evaluation.
  • Forcing citations — require the LLM to cite the chunk index it used. This both reduces position bias and gives you a signal for evaluation.

Generation: Prompt, Model, and Guardrails

The generation stage looks deceptively simple — it’s a prompt and an API call — but it’s where most user-visible quality lives.

Prompts Are Code

Treat the prompt template as versioned code. Store it in source control, evaluate it on a golden set, and ship changes behind a flag. Three things every production prompt should have:

  1. Explicit instructions on what to do when retrieval returns nothing (“say you don’t know; do not invent”).
  2. Citation requirement (“respond with [doc_id: chunk_id] after every claim”).
  3. Structured output schema when the answer is meant to be machine-consumed.

Structured outputs (JSON mode, tool use) deserve a special callout: they turn the LLM into a deterministic parser, which means downstream code can rely on shape, not vibes. The OpenAI structured outputs docs and the Anthropic tool use docs cover the patterns.

Model Choice Is a Latency/Cost/Quality Triangle

Pick the model per route, not per product. Routing simple lookups to a small fast model and complex multi-hop questions to a large reasoning model is now standard. Notion’s Q&A and Glean have written publicly about this pattern. The implementation is usually a lightweight classifier — often the same LLM with a structured prompt — that picks a route in <50ms.

Streaming and Cancellation

Users abandon responses that take more than ~2 seconds to start rendering. Stream tokens. And make sure your retrieval service can cancel an in-flight LLM call when the user navigates away — most providers expose cancellation tokens, and ignoring them is a quiet way to burn budget on answers nobody reads.

Evaluation: The Part You Skip and Then Regret

The single most common reason RAG projects fail to mature is the absence of an evaluation harness. Without one, every change is a coin flip, and you can’t tell whether your new reranker helped or hurt.

Build a Golden Set Before You Ship

You need:

  • ~200–500 representative queries with graded or known-correct answers.
  • Per-query expected source documents so you can measure retrieval separately from generation.
  • Continuous updates as users find new failure modes.

The dataset does not need to be hand-curated forever. Bootstrap it from production logs filtered for cases where users rephrased, clicked “report incorrect,” or where citations were missing. Add synthetic edge cases from your own red-teaming.

Metrics at Each Stage

Evaluate each stage independently:

StageMetricWhy
RetrievalRecall@K, MRR, nDCG@KAre the right docs in the top-K?
RerankingPrecision@K after rerankDid reranking surface the right ones?
GenerationFaithfulness, answer relevanceDid the model use the retrieved context correctly?
End-to-endHuman-rated correctness, citation accuracyThe only metric the user cares about

Tools like Ragas, DeepEval, and the TruLens framework automate the LLM-as-judge side of this. They’re not a substitute for human eval, but they let you ship changes weekly instead of quarterly.

Online Evals Catch What Offline Misses

Offline metrics don’t catch regression on distribution shift. Run lightweight online evals on a sample of production traffic: log the query, the retrieved docs, the answer, the citations, and a user signal (thumbs, dwell time, copy-to-clipboard). Then periodically re-score historical logs with a stronger LLM judge. This is how Datadog’s LLM observability and similar platforms frame the loop.

Operational Realities

The things above are the architecture. The things below are what makes it survive contact with production traffic.

Freshness Is a Pipeline Problem

A RAG system that can’t ingest a new document in under five minutes is, for many use cases, wrong. The ingestion pipeline needs:

  • Webhooks or change-data-capture from source systems (Notion, Confluence, S3, GitHub).
  • Async embedding workers that don’t block ingestion on the LLM provider.
  • Index updates that are atomic enough that a query never sees a partial document.

Most teams implement this with a queue (SQS, Kafka, Pub/Sub) feeding a worker pool. The pattern is identical to any CDC pipeline, which is the point — RAG ingestion is, fundamentally, an ETL problem with an embedding step.

Latency Budgets and Caching

A typical interactive budget is ~3 seconds end-to-end. Break it down:

  • Query embedding: 50–150ms
  • Hybrid retrieval: 50–200ms
  • Reranking (50 candidates): 200–500ms
  • LLM generation (first token, streaming): 300–800ms

If you’re over budget, caching is the answer more often than model swaps. Two cache layers:

  • Exact-query cache — Redis with the query hash as key, the answer as value. Saves 30–60% of traffic in support and FAQ workloads.
  • Semantic cache — cache by embedding similarity, not exact match. Catches rephrasings. Tools like GPTCache and most vector DBs now support this.

Cost Is Embedding Volume, Not Token Spend

Surprise #1 for most teams: embedding and re-embedding is the dominant cost, not LLM tokens. A million documents re-embedded is real money. The mitigations:

  • Embed only on actual changes (content-hash dedup).
  • Use smaller embedding models where they suffice.
  • Batch aggressively — embedding APIs are dramatically cheaper per token at batch sizes of 100+.

Failure Modes Worth Naming

A non-exhaustive list of things that have bitten real teams:

  • Silent index drift — the source-of-truth document changed but the embedding didn’t. Detect with periodic content-hash sweeps.
  • Top-K saturation — returning 50 docs with no rerank is not “more retrieval,” it’s noise. Always rerank.
  • Multilingual collapse — embeddings trained mostly on English degrade badly on other languages. Detect and route.
  • Citation hallucination — the LLM cites a plausible-looking chunk ID that doesn’t exist. Validate against the index.
  • Stale context windows — the LLM’s knowledge cutoff and your retrieval corpus disagree. Be explicit in the prompt about which is authoritative for which kind of question.

Key Takeaways

  • Treat RAG as three pipelines — ingestion, retrieval, generation — with their own SLOs and failure modes.
  • Chunking strategy and metadata design dominate retrieval quality more than embedding model choice.
  • Hybrid retrieval (dense + lexical) plus a cross-encoder reranker is the production default; LLM rerankers are an escalation, not a baseline.
  • Enforce access control at the index, never in the prompt. The prompt is not a security boundary.
  • Build an evaluation harness with offline + online metrics before you ship; without it, every change is gambling.
  • Operational concerns — freshness, latency, cost, named failure modes — are where production RAG is won or lost.

Further Reading