TL;DR — RAG quality is bounded by ingestion quality. The five hardest problems are bad chunking, stale indexes, mixed-modality documents, identity collisions during updates, and embedding-model drift. Production teams solve them with structured chunking, change-data-capture reindexing, vision-aware parsers, content-addressable IDs, and embedding-version pinning.

Why Ingestion Is Where RAG Wins or Loses

Retrieval-augmented generation has a marketing problem. Every demo looks magical because the corpus is small, clean, and curated. The moment you point a RAG pipeline at a real company’s SharePoint, Confluence, PDFs, scanned invoices, and Notion pages, the magic evaporates. Retrieval returns the wrong paragraph. The LLM hallucinates a clause that doesn’t exist. A “simple” question about last quarter’s revenue pulls a slide from three years ago.

The cause almost never lives in the prompt or the model. It lives in the ingestion layer — the boring, unglamorous, deterministic pipeline that turns human-authored documents into vectors the system can search. Once data enters an embedding space, the system has already lost most of its chances to recover meaning that wasn’t preserved at parse time. Garbage in, structured garbage out.

After running RAG pipelines against millions of documents across legal, fintech, and developer-tools companies, I’ve come to believe that ingestion is at least 80% of the engineering effort and 95% of the failure surface. The five problems below are the ones that show up over and over.

Problem 1: Chunking Breaks Meaning at the Wrong Boundaries

The first instinct is to chunk by token count — split every 512 tokens, slide a window of 64, ship it. This works for blog posts. It falls apart for everything structured: tables that get sliced between header and rows, code blocks split mid-function, numbered lists orphaned from their lead-in sentence, financial disclosures where a footnote and its parent clause end up in different chunks.

The retrieval problem is straightforward: when a user asks “what is the cap on liability in section 7?”, the relevant sentence lives in chunk N, but the qualifier that defines what counts as “section 7” lives in chunk N-3. Neither chunk scores well alone. The model gets one and confabulates the other.

Structured Chunking Patterns That Ship

The fix is to chunk along the document’s own semantic seams before falling back to size-based splitting:

  • Hierarchical chunking. Parse the document into a tree (title → section → subsection → paragraph → sentence), then index multiple granularities. At query time, retrieve the leaf chunk but return the parent context window. Libraries like LlamaIndex’s HierarchicalNodeParser and LangChain’s ParentDocumentRetriever implement this directly.
  • Format-aware splitters. Don’t use a generic text splitter on a PDF, a Markdown file, and a Jupyter notebook. Use Unstructured for mixed docs, Markitdown for Office formats, and language-specific splitters for code. Each preserves structure the generic splitter destroys.
  • Table-aware extraction. Render tables as their own chunks — either as Markdown (which embeddings handle reasonably well) or as natural-language statements (“Product A revenue in Q3 was $4.2M, up 12% YoY”) synthesized by an extraction step. Hybrid text+table retrieval beats either alone.
  • Sentence-window retrieval. Embed individual sentences but return a surrounding window of N sentences at query time. This is the default in LlamaIndex’s sentence-window node parser and produces noticeably better answers on long-form prose.

A Practical Heuristic

Start with semantic chunking at the section level, then sub-chunk only if a section exceeds your embedding model’s effective context (typically 256–512 tokens for most models, though OpenAI’s text-embedding-3-large accepts up to 8191). Index both levels. Let retrieval choose.

from llama_index.core.node_parser import HierarchicalNodeParser, SentenceSplitter

# Coarse-grained first
coarse_parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]
)

# Then a sentence-level fallback
fine_parser = SentenceSplitter(chunk_size=128, chunk_overlap=20)

The exact numbers matter less than the principle: respect document structure, index multiple granularities, retrieve the right one.

Problem 2: Indexes Go Stale Within Hours

Most teams build ingestion as a one-shot batch job: dump the corpus, embed it, deploy the index, ship the feature. Then someone edits a doc, someone else deletes a page, and the index lies. In production, the corpus is a living thing — it changes constantly — and the index is a snapshot.

The failure mode is subtle but severe. A user asks about a policy that was updated last Tuesday; the index returns the version from eight months ago; the LLM confidently answers based on the outdated text. This is the single most common cause of “the bot gave wrong info” bug reports.

Change-Data-Capture Reindexing

The production-grade answer is to treat the corpus like a database and the index like a materialized view. When source content changes, propagate the change.

  • Source connectors with webhooks. Confluence, Notion, Google Drive, and SharePoint all emit change events. Subscribe to them and queue a reindex job per changed document. Airbyte and Fivetran now expose vector-store destinations that handle this orchestration for you.
  • Filesystem watchers for S3. Use S3 event notifications to fire on Put and Delete, then trigger a per-object Lambda or worker. This is the pattern used by most document-search products that ingest from a customer’s bucket.
  • Cron-driven reconciliation as a backstop. Even with webhooks, schedules drift. Run a nightly job that diffs the corpus against the index (by content hash) and repairs drift. Treat reconciliation as a first-class system, not an afterthought.
  • Soft-deletes with tombstones. When a document is removed, mark its chunks as deleted rather than physically removing them, and run a compaction job weekly. This avoids the worst race conditions and keeps audit trails clean.

Embedding Idempotency

Every chunk should carry a deterministic ID derived from (source_doc_id, chunk_index, content_hash, embedding_model_version). If any input changes, the ID changes, and the index can upsert cleanly without duplicates. This is the same pattern as content-addressable storage and pays off the same way: rebuilds become diffs.

-- Conceptual: how an index update looks
INSERT INTO vector_index (id, embedding, metadata)
VALUES (?, ?, ?)
ON CONFLICT (id) DO UPDATE
SET embedding = EXCLUDED.embedding,
    metadata = EXCLUDED.metadata,
    indexed_at = NOW();

Problem 3: Mixed-Modality Documents Defeat Text-Only Parsers

A surprising share of “documents” are not actually text. They’re PDFs with embedded images, scanned receipts, slides with diagrams, screenshots in a Notion page, charts that contain the actual answer. Tesseract-on-image-extraction catches some of this but produces noisy, layout-broken text that embeddings handle poorly.

The result: a query about a chart’s underlying numbers returns nothing, because the chart was rasterized into a JPEG during PDF conversion and the text extractor saw a blank rectangle.

Vision-Aware Parsers and Multimodal Embeddings

Two patterns work in production:

  1. Vision-LLM extraction. Send each page image to a multimodal model (GPT-4o, Claude with vision, Gemini) and ask for a structured representation: titles, paragraphs, table data, figure descriptions. The cost is real but bounded — typically $0.01–$0.05 per page — and the quality jump is dramatic. Unstructured now ships with vision-model extractors that do this out of the box.
  2. Multimodal embeddings directly. Models like Cohere’s embed-multilingual-vision and CLIP embed images and text into the same space. For slide decks and product catalogs, this lets users search “the bar chart showing Q3 growth” and find the actual chart, not a paragraph that mentions it.

A typical pipeline combines both:

Page → Render to image → Vision LLM extracts text + table markdown + figure caption
     → Text chunks → text-embedding-3-large
     → Figure captions + extracted tables → same embedding model
     → Original images → CLIP embedder, indexed separately
     → All stored in same vector DB with metadata flagging modality

The flag matters. At retrieval time, you can boost or filter by modality depending on the query. “Show me the dashboard” should weight images; “summarize the contract” should weight text.

Problem 4: Identity Collisions During Updates

This is the problem nobody talks about until they hit it. You reindex a document and suddenly the system returns two chunks for every retrieval — one from the old version, one from the new. Or worse, a renamed document leaves orphan vectors pointing at content that no longer exists.

The cause is almost always a poorly chosen primary key. Teams use filenames, which change; URLs, which redirect; auto-incrementing IDs from the source system, which don’t survive migration; or worse, no ID at all, just the vector itself (which can’t be updated — only deleted and reinserted).

Content-Addressable IDs Done Right

The fix is to assign each chunk a stable ID derived from immutable properties:

chunk_id = sha256(
    source_system          # "confluence", "gdrive", "s3"
    + ":" +
    source_doc_id          # canonical doc ID from the source system
    + ":" +
    chunk_index            # position within the doc
    + ":" +
    content_hash           # sha256 of the chunk's text
    + ":" +
    embed_model_version    # "text-embedding-3-large@1"
)

This ID survives renames, moves, and embedding upgrades. When content changes, the hash changes, the ID changes, the old chunk is orphaned (and cleaned up by a sweep job), and the new chunk is inserted. No collisions. No duplicates. No “ghost chunks from last quarter’s reindex” haunting production.

Pair this with a metadata table that maps chunk_id → source_uri → indexed_at → embedding_version, and you can answer the most important production question instantly: “is what the user is seeing actually in the index, and from when?”

Problem 5: Embedding-Model Drift Silently Degrades Quality

You upgraded from text-embedding-ada-002 to text-embedding-3-large and everything got better. Then six months later you reindexed a subset of documents to “clean things up” using the new model — but the rest of the corpus is still on the old one. Now your retrieval is incoherent: similar documents live in incompatible vector spaces, and nearest-neighbor queries return garbage across the seam.

This is embedding-model drift, and it is one of the most underappreciated sources of quality regression in production RAG.

Version Pinning and Two-Phase Migrations

The disciplined approach:

  1. Pin the model version in the chunk ID. Already covered above — embed_model_version is non-negotiable.
  2. Never mix models in a single index. Either every chunk uses v1 or every chunk uses v2. Mixing is almost always worse than using either alone.
  3. Migrate in two phases. First, dual-write: new chunks go to both v1 and v2 indexes, served side by side with shadow traffic to compare. Second, switch reads to v2 and backfill the rest of the corpus. Pinecone and Weaviate both support named vectors and collection aliases that make this cleaner.
  4. Evaluate continuously. A retrieval eval set — 200–500 queries with ground-truth relevant chunks — should run nightly against the live index. Track recall@k, MRR, and a “freshness” metric measuring how often the top result is from the most recent version of the document. When these numbers move, you know before users do.
# Example eval config (simplified)
queries:
  - q: "What is the data retention policy?"
    relevant_chunks:
      - doc_id: "policy-2024"
        chunk_index: 7
    must_be_fresher_than: "2024-01-01"

The eval set is your early-warning system. Build it before you build the chatbot.

Architecture: A Production RAG Ingestion Pipeline

Putting it all together, a pipeline that survives the five problems looks roughly like this:

[Source Systems]
   ↓ (webhooks, CDC, cron)
[Ingestion Queue — SQS / Kafka / Pub/Sub]
   ↓
[Parser Worker]
   - format detection (PDF, DOCX, MD, HTML, image)
   - vision-LLM extraction for image-heavy pages
   - structured chunking (hierarchical + sentence-window)
   ↓
[Embedder Worker]
   - pinned model version
   - deterministic chunk IDs (sha256 of canonical inputs)
   - writes to BOTH vector store AND metadata store
   ↓
[Vector Store — Pinecone / Weaviate / pgvector / Qdrant]
   ↓
[Reconciliation Worker — nightly]
   - diffs corpus vs index
   - repairs drift
   - compacts tombstones
   ↓
[Retrieval Service]
   - query rewriting
   - hybrid search (BM25 + dense)
   - metadata filters (modality, freshness, source ACL)
   - reranking with cross-encoder
   ↓
[LLM — Claude / GPT / Gemini]

The components that look like overhead — the metadata store, the reconciliation worker, the eval harness — are precisely what separates a demo from a system that handles a million queries a month without degrading.

Key Takeaways

  • Chunk along semantic seams, not token counts. Hierarchical chunking, format-aware splitters, and sentence-window retrieval beat naive fixed-size splitting on every corpus that isn’t a blog.
  • Treat the index as a materialized view, not a snapshot. Webhook-driven updates, content-hash-based IDs, and nightly reconciliation are non-optional at production scale.
  • Mixed-modality documents need vision-aware extraction. Text-only parsers silently destroy the information your users actually want to query.
  • Stable IDs come from content, not from filenames or URLs. Content-addressable chunk IDs eliminate duplicates, ghost chunks, and update races.
  • Embedding-model version drift is a silent killer. Pin versions in IDs, migrate in two phases, and run a retrieval eval set nightly so you catch regressions before users do.

Further Reading