TL;DR — Production RAG is less about clever prompts and more about disciplined pipelines: deterministic ingestion, hybrid retrieval over well-chunked documents, vector stores tuned for your access pattern, and a tight feedback loop for evaluation. Treat retrieval as a first-class system, not a sidecar, and the rest of the architecture follows.

Why Most RAG Demos Don’t Survive Production

A RAG demo is a single Jupyter notebook: load a few PDFs, embed them, shove the vectors into a store, ask a question, and watch a clean answer appear. Production RAG is something else entirely. It’s a distributed system that has to handle versioned document corpora, embedding model rollouts, semantic caches, query rewriting, hybrid retrieval, latency budgets, and a cost model that scales with the number of tokens you didn’t mean to send to the LLM.

The failure modes are well known and well documented in production write-ups: Pinterest’s lessons on grounding LLMs at scale and Notion’s writeup on RAG at scale both emphasize that retrieval quality, not generation quality, is the dominant variable. Most “the LLM got it wrong” complaints are actually “the retriever returned the wrong chunks.”

The architecture below treats retrieval, embedding, and generation as independent subsystems with their own scaling, observability, and rollback stories. That’s the difference between a demo and a system.

The Production RAG Pipeline

A production RAG system has two pipelines: an indexing pipeline that runs offline and a query pipeline that runs online. Confusing the two is the source of many production incidents.

Indexing (offline)              Query (online)
─────────────────              ──────────────
Sources → Fetch → Parse         User query
        → Chunk                       ↓
        → Embed                  Query rewrite
        → Store                  → Embed (same model)
        → Version               → Retrieve (hybrid)
                                 → Rerank
                                 → Compress
                                 → LLM generate
                                 → Stream response

The two pipelines share the embedding model and the vector store, and that shared state is where most of the operational complexity lives.

Ingestion: The Part Everyone Underestimates

Ingestion is the unsexy half of RAG, and the half that determines whether your system is useful at all. The naive approach — split each document on every 500 tokens — produces chunks that routinely cut a sentence in half, lose table structure, and treat a 12-page PDF as if it were a flat text file.

Production ingestion does four things differently:

  1. Format-aware parsing. PDFs need layout extraction (tables, headers, footers) — libraries like PyMuPDF and Unstructured handle this far better than pdftotext. HTML needs to be cleaned of nav, ads, and scripts before chunking. Confluence and Notion exports have their own quirks.
  2. Structural chunking. Where possible, chunk along document boundaries — sections, headings, list items, table rows — instead of fixed token windows. LangChain’s MarkdownHeaderTextSplitter and LlamaIndex’s SentenceSplitter with metadata preservation are the workhorses here.
  3. Enrichment before embedding. Append the document title, section path, and a short summary to each chunk. A query about “Kubernetes networking” should match a chunk that starts with Title: Production K8s Guide > Networking > CNI Plugins > , not just the bare paragraph.
  4. Deterministic IDs. Every chunk needs a stable ID derived from (source_doc_id, section_path, content_hash). Without this, re-ingestion creates duplicates and your vector store bloats silently.
import hashlib
from dataclasses import dataclass

@dataclass
class Chunk:
    chunk_id: str
    doc_id: str
    section: str
    content: str
    metadata: dict

    @staticmethod
    def make(doc_id: str, section: str, content: str, metadata: dict) -> "Chunk":
        h = hashlib.sha256(f"{doc_id}::{section}::{content}".encode()).hexdigest()[:16]
        return Chunk(
            chunk_id=f"{doc_id}:{h}",
            doc_id=doc_id,
            section=section,
            content=content,
            metadata={**metadata, "section": section},
        )

This is unglamorous, and it’s exactly the code that separates a system that runs for six months from one that corrupts its index in week three.

Vector Store Selection: It’s an Access-Pattern Decision

There is no “best” vector database. There is only the right database for your access pattern, scale, and operational maturity. The Pinecone, Weaviate, and Milvus comparison posts all converge on the same conclusion: pick by workload.

The Three Workloads

1. Low-latency single-tenant retrieval (< 50ms p99, < 10M vectors). This is most B2B RAG. Pinecone serverless, pgvector on Postgres, and Qdrant all fit. pgvector is the right answer when you already run Postgres and your vector count is modest — you get transactions, backups, and joins for free. Qdrant and Pinecone win when you want to outsource the operational burden.

2. High-throughput, large-scale retrieval (> 10M vectors, billions of vectors). Milvus and Weaviate are designed for this. They support sharding, replicas, and hybrid search natively. The trade-off is operational complexity — you are now running a distributed system, and the on-call burden is real.

3. Embedded or edge use cases. Chroma and LanceDB are designed to run in-process or on a single node. They are not for production multi-tenant RAG, but they are excellent for prototyping and for client-side retrieval.

Hybrid Search: Don’t Trust Embeddings Alone

Pure vector search is good at semantic similarity and terrible at exact match. If a user asks “show me all invoices from vendor 4421,” you want BM25, not cosine similarity. The pattern in production is hybrid search: combine vector retrieval with a lexical retriever, then fuse the rankings.

Reciprocal Rank Fusion (RRF) is the standard fusion algorithm because it doesn’t require score calibration between the two retrievers:

def rrf_fuse(vector_results: list, bm25_results: list, k: int = 60) -> list:
    """Reciprocal Rank Fusion — combine two ranked lists without score calibration."""
    scores: dict[str, float] = {}
    for rank, doc_id in enumerate(vector_results):
        scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    for rank, doc_id in enumerate(bm25_results):
        scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Most production systems also add a reranker on top of the fused list — a cross-encoder like BGE-reranker or Cohere Rerank that re-scores the top 50–100 candidates. This is the single biggest retrieval-quality win you can buy, and it’s cheap relative to the LLM call that follows.

Patterns in Production: What Teams Actually Ship

Looking across the public RAG write-ups from AWS, GCP, and the various vendor engineering blogs, four patterns show up over and over.

Query Rewriting and Decomposition

Users don’t write queries that match documents. They write “why is my bill so high this month” when the document is titled “Invoice Adjustment Procedure.” The fix is a query rewrite step: pass the original query through a small, fast LLM (or even a rules engine) that produces a search-optimized version. The LangChain MultiQueryRetriever and the Microsoft GraphRAG approach are good starting points.

For complex queries, decomposition — breaking “compare the pricing tiers of plans A, B, and C” into three sub-queries — is even more effective, but it costs latency.

A tenant-scoped RAG system should never let a query from tenant A retrieve tenant B’s documents. The fix is metadata filtering at the vector store level, pushed down to the index. Both Pinecone and Qdrant support this efficiently. The pattern is:

results = vector_store.search(
    query_embedding=embed(query),
    filter={
        "tenant_id": {"$eq": tenant_id},
        "doc_type": {"$in": ["policy", "faq"]},
        "created_at": {"$gte": cutoff_date},
    },
    top_k=50,
)

Filter first, then vector search, not the other way around. Most vector stores can prune the search space using the metadata filter before computing distances.

Semantic Caching

LLM calls are expensive. A surprising fraction of RAG traffic is the same question asked many times. A semantic cache — hash the query embedding, return the cached answer if the cosine similarity to a recent query is above a threshold — reduces cost by 30–60% in most production systems I’ve seen. GPTCache and Redis with vector support are the common implementations.

The cache itself is a tiny RAG system: an embedding model, a vector store, and a TTL policy. Keep it simple.

Streaming and Partial Results

Users hate waiting 4 seconds for a complete answer. Stream the LLM tokens as they arrive, but stream retrieval status before that — “Searching knowledge base… Found 5 relevant documents… Generating answer…” — so the UI feels alive. This is one of the cheapest UX improvements available, and most teams skip it.

Scaling the Index: The Hard Problem

Scaling reads is well understood. Scaling the index is not, because the index is stateful and the state is large.

Re-indexing Without Downtime

You will need to re-index. The embedding model will change, the chunking strategy will improve, and the document corpus will grow. The pattern that works:

  1. Build the new index in a separate namespace or collection. Never write to the live index in place.
  2. Dual-write for a validation period. New writes go to both old and new indexes.
  3. Backtest the new index against logged queries. You have a query log (you do, right?) — replay the last 30 days of queries against the new index and compare retrieval quality to the old.
  4. Cut over with a feature flag. Flip 1% of traffic, watch the metrics, ramp.

This is the same pattern you’d use to migrate any production database, and the same tooling applies.

Embedding Cost and Latency

Embedding is the hidden cost of RAG. A 1M-document corpus with 500-token average chunks is 500M tokens to embed on every re-index. At OpenAI’s text-embedding-3-small pricing, that’s a meaningful number; at text-embedding-3-large, it’s a budget line item.

Three mitigations work:

  • Batch embedding calls. Most providers give 5–10x throughput at the same latency if you send 100 chunks per request instead of 1.
  • Use a smaller embedding model for first-pass retrieval, rerank with a larger one. The MTEB benchmark shows that some 100M-parameter models are within 2% of 1B-parameter models on most retrieval tasks.
  • Cache embeddings aggressively. The same chunk embedded twice should hit a cache. Most production systems use Redis or a simple content-hash → embedding map.

Evaluation: The System You Don’t Have Yet

Most teams ship RAG without a proper evaluation harness, and then they can’t tell whether a chunking change helped or hurt. The fix is a small evaluation suite that runs on every index change.

# eval/rag_eval.yaml — run on every index change
datasets:
  - name: "production_query_sample"
    path: "s3://eval-data/q1-sample.jsonl"  # 500 real queries
  - name: "synthetic_hard_cases"
    path: "s3://eval-data/synthetic.jsonl"   # edge cases from QA

metrics:
  - name: "recall_at_10"
    type: "retrieval"
  - name: "answer_faithfulness"
    type: "llm_judge"   # "Does the answer only use info from the retrieved context?"
  - name: "answer_relevance"
    type: "llm_judge"
  - name: "p50_latency_ms"
    type: "system"
  - name: "p99_latency_ms"
    type: "system"

Frameworks like RAGAS, TruLens, and DeepEval give you this out of the box. The LLM-judge metrics are not perfect, but they are far better than no metrics, and they improve dramatically when you give the judge a clear rubric.

The query log is gold. The first thing you should build after shipping RAG is a pipeline that stores every (query, retrieved_chunks, generated_answer, user_feedback) tuple. After a month, you have a labeled dataset for free, and your evaluation harness becomes a real signal.

Key Takeaways

  • Retrieval quality dominates generation quality. Spend the engineering time on chunking, hybrid search, and reranking, not on prompt engineering.
  • Pick your vector store by access pattern, not by leaderboard rank. pgvector for modest scale on existing Postgres, Qdrant/Pinecone for managed multi-tenant, Milvus/Weaviate for billion-scale.
  • Hybrid search with reranking is the default. Pure vector search leaves too much recall on the table.
  • Treat the index as a deployable artifact. Version it, backtest it, cut over with a feature flag.
  • Cache semantically, stream tokens, and surface retrieval status. The cheap UX wins matter.
  • Build the eval harness before you need it. The query log is the most valuable dataset you own.

Further Reading