Building Scalable RAG Pipelines with Hybrid Search and Advanced Re-Ranking Techniques

Table of Contents

1. Introduction
2. What Is Retrieval‑Augmented Generation (RAG)?
3. Why Scaling RAG Is Hard
4. Hybrid Search: The Best of Both Worlds
   • 4.1 Sparse (BM25) Retrieval
   • 4.2 Dense (Vector) Retrieval
   • 4.3 Fusion Strategies
5. Advanced Re‑Ranking Techniques
   • 5.1 Cross‑Encoder Re‑Rankers
   • 5.2 LLM‑Based Re‑Ranking
   • 5.3 Learning‑to‑Rank (LTR) Frameworks
6. Designing a Scalable RAG Architecture
   • 6.1 Data Ingestion & Chunking
   • 6.2 Indexing Layer
   • 6.3 Hybrid Retrieval Service
   • 6.4 Re‑Ranking Service
   • 6.5 LLM Generation Layer
   • 6.6 Orchestration & Asynchronicity
7. Practical Implementation Walk‑through
   • 7.1 Prerequisites & Environment Setup
   • 7.2 Building the Indexes (FAISS + Elasticsearch)
   • 7.3 Hybrid Retrieval API
   • 7.4 Cross‑Encoder Re‑Ranker with Sentence‑Transformers
   • 7.5 LLM Generation with OpenAI’s Chat Completion
   • 7.6 Putting It All Together – A FastAPI Endpoint
8. Performance & Cost Optimizations
   • 8.1 Caching Strategies
   • 8.2 Batch Retrieval & Re‑Ranking
   • 8.3 Quantization & Approximate Nearest Neighbor (ANN)
   • 8.4 Horizontal Scaling with Kubernetes
9. Monitoring, Logging, and Observability
   10 Real‑World Use Cases
   11 Best Practices Checklist
   12 Conclusion
   13 Resources

Introduction

Retrieval‑Augmented Generation (RAG) has emerged as a powerful paradigm for leveraging large language models (LLMs) while grounding their output in factual, up‑to‑date information. By coupling a retriever (which fetches relevant documents) with a generator (which synthesizes a response), RAG systems can answer questions, draft reports, or provide contextual assistance with far higher accuracy than a vanilla LLM.

However, as organizations move from prototypes to production, they encounter a new set of challenges:

• Scale: Millions of documents, thousands of concurrent queries, and latency constraints measured in milliseconds.
• Relevance: Simple keyword matching (BM25) or pure vector similarity often fails to capture nuanced intent.
• Cost: Dense vector search and LLM calls are expensive; optimizing for price without sacrificing quality is essential.

This article dives deep into Hybrid Search—the combination of sparse (BM25) and dense (vector) retrieval—and Advanced Re‑Ranking Techniques that together enable RAG pipelines to scale, stay relevant, and remain cost‑effective. We’ll walk through the theory, architecture, and a full‑stack implementation using open‑source tools (FAISS, Elasticsearch, Sentence‑Transformers) and commercial LLM APIs (OpenAI).

By the end of this guide, you’ll have a production‑ready blueprint you can adapt to your own knowledge‑base, customer‑support, or analytics workloads.

What Is Retrieval‑Augmented Generation (RAG)?

RAG is a two‑step workflow:

1. Retrieval: Given a user query q, a retriever returns a set of k documents D = {d₁,…,dₖ} that are most likely to contain the answer.
2. Generation: The LLM receives q and the retrieved passages (often concatenated or formatted as “retrieved context”) and produces a response r.

Core Benefits

The quality of a RAG system hinges on the retriever and the generator. While the LLM side is often a black box, the retrieval layer can be engineered, tuned, and scaled extensively—hence the focus on hybrid search and re‑ranking.

Why Scaling RAG Is Hard

To overcome these, we need a modular, asynchronous, and observable architecture where each component can be independently scaled and optimized.

Hybrid Search: The Best of Both Worlds

Hybrid search combines sparse (term‑based) and dense (vector‑based) retrieval. Each approach has strengths:

By fusing their scores, we can achieve higher recall and precision.

4.1 Sparse (BM25) Retrieval

BM25 is a probabilistic model that scores documents based on term frequency and inverse document frequency. It’s implemented natively in most search engines (Elasticsearch, OpenSearch, Solr). Example query:

GET /my-index/_search
{
  "query": {
    "match": {
      "content": "how to reset a password"
    }
  },
  "size": 10
}

4.2 Dense (Vector) Retrieval

Dense retrieval uses a neural encoder (e.g., Sentence‑Transformers) to embed queries and documents into a high‑dimensional space. Similarity is measured with cosine or inner product. FAISS, Annoy, or ScaNN provide fast Approximate Nearest Neighbor (ANN) search.

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')
doc_embeddings = np.load('doc_embeddings.npy')   # shape (N, 384)
index = faiss.IndexFlatIP(doc_embeddings.shape[1])
index.add(doc_embeddings)

query = "reset password steps"
q_emb = model.encode([query])
D, I = index.search(q_emb, k=10)   # D: scores, I: indices

4.3 Fusion Strategies

There are several ways to combine sparse and dense scores:

1. Score Interpolation (Linear Fusion)
   final_score = α * sparse_score + (1-α) * dense_score
   α ∈ [0,1] is a hyper‑parameter tuned on a validation set.

2. Reciprocal Rank Fusion (RRF)
   For each document d retrieved by method m, compute 1 / (k + rank_m(d)). Sum across methods. RRF is robust to score scaling differences.

3. Hybrid Index (Elasticsearch + k‑NN plugin)
   Elasticsearch now supports a dense_vector field and a knn query that can be combined with BM25 in a single request.

GET /my-index/_search
{
  "size": 10,
  "query": {
    "bool": {
      "should": [
        { "match": { "content": "reset password" } },
        {
          "knn": {
            "field": "embedding",
            "query_vector": [0.12, -0.08, ...],
            "k": 10,
            "num_candidates": 100
          }
        }
      ]
    }
  }
}

Hybrid search dramatically improves recall (the chance that a relevant document appears in the top‑k) while maintaining precision after re‑ranking.

Advanced Re‑Ranking Techniques

Even after hybrid retrieval, the top‑k set may contain noisy candidates. Re‑ranking refines the order using richer models that consider the full query‑document interaction.

5.1 Cross‑Encoder Re‑Rankers

A cross‑encoder concatenates query and document and passes them through a transformer, outputting a relevance score. Unlike bi‑encoders (used for dense retrieval), cross‑encoders can model fine‑grained interactions but are slower, which is why they are typically applied only to a small candidate set (e.g., top‑100).

from sentence_transformers import CrossEncoder

cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-12-v2')
pairs = [(query, doc) for doc in candidate_texts]
scores = cross_encoder.predict(pairs)   # returns a list of relevance scores
ranked = sorted(zip(candidate_texts, scores), key=lambda x: x[1], reverse=True)

5.2 LLM‑Based Re‑Ranking

Large language models can be prompted to judge relevance:

import openai

def llm_rank(query, docs):
    prompt = f"""You are an expert information retrieval system. Given the user query:
"{query}"
Rank the following passages from most to least relevant. Return a JSON list of indices.

Passages:
"""
    for i, d in enumerate(docs):
        prompt += f"{i+1}. {d[:200]}...\n"

    response = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        max_tokens=256,
    )
    return eval(response.choices[0].message.content)   # e.g., [2,0,1,...]

LLM re‑ranking is expensive but can be limited to a handful of top‑k candidates, delivering state‑of‑the‑art relevance.

5.3 Learning‑to‑Rank (LTR) Frameworks

Frameworks like XGBoost, LightGBM, or RankNet can be trained on labeled click‑through or relevance data. Features may include:

• BM25 score
• Dense similarity
• Document length
• Entity overlap
• Cross‑encoder score (as a feature, not a full re‑ranker)

A typical pipeline:

import xgboost as xgb
import pandas as pd

# Assume we have a DataFrame with feature columns and a relevance label
dtrain = xgb.DMatrix(df[feature_cols], label=df['relevance'])
params = {'objective': 'rank:pairwise', 'eval_metric': 'ndcg'}
model = xgb.train(params, dtrain, num_boost_round=200)

LTR models run fast at inference time and can be updated continuously with new user feedback.

Designing a Scalable RAG Architecture

Below is a reference architecture that separates concerns, enables horizontal scaling, and supports hybrid search + re‑ranking.

+-------------------+      +--------------------+      +-------------------+
|   Ingestion Layer | ---> |   Indexing Service | ---> |   Search Service |
+-------------------+      +--------------------+      +-------------------+
                                          |                |
                                          v                v
                                   +-------------------+  +-------------------+
                                   |   Sparse Index    |  |   Vector Index    |
                                   | (Elasticsearch)  |  | (FAISS/ANN)       |
                                   +-------------------+  +-------------------+
                                          |                |
                                          +-------+--------+
                                                  |
                                          +-------------------+
                                          | Hybrid Retrieval  |
                                          +-------------------+
                                                  |
                                          +-------------------+
                                          | Re‑Ranking Service|
                                          +-------------------+
                                                  |
                                          +-------------------+
                                          | LLM Generation    |
                                          +-------------------+
                                                  |
                                          +-------------------+
                                          | API / Front‑End   |
                                          +-------------------+

6.1 Data Ingestion & Chunking

• Source Types: PDFs, HTML, markdown, relational DB dumps, APIs.
• Chunking Strategy: Split documents into 200‑300 token passages with overlap (e.g., 50 tokens) to preserve context.
• Metadata Enrichment: Store source ID, section titles, timestamps, and language tags.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=300,
    chunk_overlap=50,
    separators=["\n\n", "\n", " ", ""]
)

chunks = splitter.split_text(raw_document)

6.2 Indexing Layer

• Sparse Index: Elasticsearch with BM25, custom analyzers (e.g., n‑gram for partial matches).
• Dense Index: FAISS IndexFlatIP (or IVF‑PQ for large corpora). Embeddings stored both in FAISS and persisted in a KV store for metadata lookup.

6.3 Hybrid Retrieval Service

A service (e.g., FastAPI) receives a query, computes both BM25 and dense results, and fuses them using RRF or linear interpolation.

def hybrid_search(query, k=10, alpha=0.6):
    bm25_hits = es.search(index="docs", body={"query": {"match": {"content": query}}, "size": k})
    dense_hits = dense_retriever.search(query, k=k*2)   # oversample for RRF
    fused = rrf_fusion(bm25_hits, dense_hits, k)
    return fused

6.4 Re‑Ranking Service

• Stage‑1: Cross‑encoder re‑ranker on top‑k (e.g., 100) candidates.
• Stage‑2 (optional): LLM re‑ranker on top‑10 for final polishing.
• Caching: Store recent query‑document scores to avoid recomputation.

6.5 LLM Generation Layer

• Prompt Template:

  Context:
  {retrieved_passages}
  
  Question: {user_query}
  
  Answer (cite sources with IDs):
  

• Streaming: Use OpenAI’s streaming API or a self‑hosted model with text-generation-inference to deliver partial answers quickly.

6.6 Orchestration & Asynchronicity

• Message Queue: Kafka or RabbitMQ to decouple retrieval, re‑ranking, and generation.
• Task Scheduler: Celery or Temporal for retries, timeouts, and retries.
• Async Framework: FastAPI with async endpoints allows concurrent handling of many queries.

Practical Implementation Walk‑through

Below is a concrete example that you can run locally or adapt to a cloud environment. It uses:

• Elasticsearch (sparse index) – Docker image docker.elastic.co/elasticsearch/elasticsearch:8.9.0
• FAISS (dense index) – Python library
• Sentence‑Transformers for embeddings
• Cross‑Encoder for re‑ranking
• OpenAI for final generation
• FastAPI as the HTTP layer

7.1 Prerequisites & Environment Setup

# Create a virtual environment
python -m venv rag-env
source rag-env/bin/activate

# Install packages
pip install fastapi uvicorn elasticsearch[async] sentence-transformers faiss-cpu \
            torch transformers openai tqdm

Start Elasticsearch:

docker run -d --name es \
  -p 9200:9200 -e "discovery.type=single-node" \
  docker.elastic.co/elasticsearch/elasticsearch:8.9.0

7.2 Building the Indexes (FAISS + Elasticsearch)

import json, os, uuid
from elasticsearch import AsyncElasticsearch
from sentence_transformers import SentenceTransformer
import numpy as np, faiss

# 1️⃣ Load documents (example: a folder of .txt files)
def load_docs(path):
    docs = []
    for fname in os.listdir(path):
        if fname.endswith(".txt"):
            with open(os.path.join(path, fname), "r", encoding="utf-8") as f:
                text = f.read()
                docs.append({"id": str(uuid.uuid4()), "content": text})
    return docs

docs = load_docs("./data")

# 2️⃣ Chunk & embed
model = SentenceTransformer('all-MiniLM-L6-v2')
chunks = []
embeddings = []

for doc in docs:
    # Simple split on paragraphs
    paragraphs = doc["content"].split("\n\n")
    for para in paragraphs:
        if not para.strip():
            continue
        chunk_id = f"{doc['id']}_{len(chunks)}"
        chunks.append({"id": chunk_id, "content": para, "parent_id": doc["id"]})
        emb = model.encode(para, normalize_embeddings=True)
        embeddings.append(emb)

emb_matrix = np.vstack(embeddings).astype('float32')

# 3️⃣ Create FAISS index
dim = emb_matrix.shape[1]
faiss_index = faiss.IndexFlatIP(dim)   # inner product (cosine after normalization)
faiss_index.add(emb_matrix)

# Persist index
faiss.write_index(faiss_index, "faiss.index")
np.save("ids.npy", np.array([c["id"] for c in chunks]))

# 4️⃣ Index into Elasticsearch
es = AsyncElasticsearch(hosts=["http://localhost:9200"])

async def index_es():
    # Create index with BM25 (default) and a dense_vector field
    mapping = {
        "mappings": {
            "properties": {
                "content": {"type": "text"},
                "embedding": {"type": "dense_vector", "dims": dim}
            }
        }
    }
    await es.indices.create(index="rag-docs", body=mapping, ignore=400)

    # Bulk index
    actions = []
    for i, chunk in enumerate(chunks):
        action = {
            "_index": "rag-docs",
            "_id": chunk["id"],
            "_source": {
                "content": chunk["content"],
                "embedding": embeddings[i].tolist()
            }
        }
        actions.append(action)

    # Use async bulk helper
    from elasticsearch.helpers import async_bulk
    await async_bulk(es, actions)

import asyncio
asyncio.run(index_es())

7.3 Hybrid Retrieval API

from fastapi import FastAPI, HTTPException
from elasticsearch import AsyncElasticsearch
import numpy as np, faiss, json

app = FastAPI()
es = AsyncElasticsearch(hosts=["http://localhost:9200"])

# Load FAISS index & IDs
faiss_index = faiss.read_index("faiss.index")
ids = np.load("ids.npy", allow_pickle=True)

def rrf_fusion(bm25_hits, dense_hits, k=10, rrf_k=60):
    """Reciprocal Rank Fusion implementation."""
    scores = {}
    # BM25 part
    for rank, hit in enumerate(bm25_hits["hits"]["hits"], start=1):
        doc_id = hit["_id"]
        scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (rrf_k + rank)
    # Dense part
    for rank, idx in enumerate(dense_hits, start=1):
        doc_id = ids[idx]
        scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (rrf_k + rank)
    # Sort and return top‑k IDs
    sorted_ids = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:k]
    return [doc_id for doc_id, _ in sorted_ids]

@app.get("/search")
async def hybrid_search(query: str, k: int = 10):
    # 1️⃣ BM25 via Elasticsearch
    bm25_resp = await es.search(
        index="rag-docs",
        body={"size": k*2, "query": {"match": {"content": query}}},
        _source=False   # we only need IDs now
    )
    # 2️⃣ Dense via FAISS
    q_emb = model.encode([query], normalize_embeddings=True).astype('float32')
    D, I = faiss_index.search(q_emb, k*2)   # retrieve more for fusion
    dense_ids = I[0].tolist()
    # 3️⃣ Fuse
    final_ids = rrf_fusion(bm25_resp, dense_ids, k=k)
    # 4️⃣ Retrieve full passages
    docs = await es.mget(index="rag-docs", body={"ids": final_ids})
    passages = [doc["_source"]["content"] for doc in docs["docs"] if doc["found"]]
    return {"query": query, "passages": passages}

7.4 Cross‑Encoder Re‑Ranker with Sentence‑Transformers

from sentence_transformers import CrossEncoder

cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-12-v2')

def cross_rank(query, passages, top_n=5):
    pairs = [(query, p) for p in passages]
    scores = cross_encoder.predict(pairs)
    ranked = sorted(zip(passages, scores), key=lambda x: x[1], reverse=True)
    return [p for p, _ in ranked[:top_n]]

7.5 LLM Generation with OpenAI’s Chat Completion

import openai, os
openai.api_key = os.getenv("OPENAI_API_KEY")

def generate_answer(query, context_passages):
    context = "\n\n".join([f"[{i+1}] {p}" for i, p in enumerate(context_passages)])
    prompt = f"""You are an expert assistant. Use the following context passages to answer the question. Cite the passage numbers in your answer.

Context:
{context}

Question: {query}
Answer:"""
    response = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        max_tokens=512,
    )
    return response.choices[0].message.content.strip()

7.6 Putting It All Together – A FastAPI Endpoint

@app.get("/qa")
async def answer_question(query: str, k: int = 10):
    # Hybrid retrieval
    bm25_resp = await es.search(
        index="rag-docs",
        body={"size": k*2, "query": {"match": {"content": query}}},
        _source=False
    )
    q_emb = model.encode([query], normalize_embeddings=True).astype('float32')
    D, I = faiss_index.search(q_emb, k*2)
    dense_ids = I[0].tolist()
    final_ids = rrf_fusion(bm25_resp, dense_ids, k=k)
    docs = await es.mget(index="rag-docs", body={"ids": final_ids})
    passages = [doc["_source"]["content"] for doc in docs["docs"] if doc["found"]]

    # Cross‑encoder re‑rank
    top_passages = cross_rank(query, passages, top_n=5)

    # LLM generation
    answer = generate_answer(query, top_passages)

    return {"question": query, "answer": answer, "sources": top_passages}

Run the service:

uvicorn main:app --host 0.0.0.0 --port 8000

Now you have a scalable RAG pipeline that:

1. Retrieves via hybrid search (BM25 + dense vectors).
2. Refines results with a cross‑encoder.
3. Generates a grounded answer with OpenAI’s LLM.

You can further scale each component (FAISS sharding, Elasticsearch clustering, GPU‑accelerated cross‑encoders) without changing the API contract.

Performance & Cost Optimizations

8.1 Caching Strategies

• Query‑Document Cache: Store top‑k results for frequently asked questions (e.g., using Redis).
• Embedding Cache: Cache embeddings for new documents to avoid recomputation on updates.
• LLM Response Cache: For static knowledge (e.g., policy documents), cache final answers keyed by a hash of (query, context_ids).

8.2 Batch Retrieval & Re‑Ranking

When serving many concurrent queries, batch the dense retrieval calls to FAISS (or use faiss.IndexIVFFlat with multi‑probe). Batch cross‑encoder inference using the model’s predict on a list of pairs to amortize GPU overhead.

8.3 Quantization & Approximate Nearest Neighbor (ANN)

• Product Quantization (PQ): Reduces index size by 10‑20× with minimal loss in recall.
• IVF (Inverted File) + HNSW: Enables sub‑millisecond ANN search on billions of vectors.
  FAISS provides utilities such as faiss.IndexIVFPQ and faiss.IndexHNSWFlat.

8.4 Horizontal Scaling with Kubernetes

• Stateless Services: Retrieval, re‑ranking, and generation services can be containerized and autoscaled with Horizontal Pod Autoscaler (HPA).
• StatefulSets: FAISS indexes can be stored on a shared PV or in memory with sidecar replication.
• Service Mesh (Istio/Linkerd): Enables traffic splitting for A/B testing new ranking models.

Monitoring, Logging, and Observability

Instrument each FastAPI endpoint with middleware that records request IDs, timestamps, and downstream service durations. Use OpenTelemetry to propagate tracing across services.

Real‑World Use Cases

1. Customer Support Knowledge Base
   Hybrid search quickly surfaces policy documents, while a cross‑encoder ensures the most relevant troubleshooting steps rise to the top. The LLM then composes a personalized reply, citing document IDs for auditability.

2. Legal Document Review
   Legal teams ingest contracts, case law, and statutes. Dense retrieval captures semantic similarity (e.g., “force majeure”), while BM25 finds exact clause numbers. An LTR model trained on attorney relevance judgments re‑ranks, and the LLM drafts a summary with citations.

3. Enterprise Internal Search
   Employees query across wikis, tickets, and code repositories. Hybrid search unifies unstructured text and code snippets. The re‑ranking pipeline incorporates code‑specific features (e.g., token overlap with function names) to surface the most actionable results.

Best Practices Checklist

• Data Quality: Clean, deduplicate, and enrich documents with metadata before indexing.
• Chunk Size: 200–300 tokens with overlap; adjust based on LLM context window.
• Embedding Model: Choose a model that matches your domain (e.g., all-MiniLM-L6-v2 for general English, multilingual-e5-large for multilingual corpora).
• Fusion Hyper‑parameters: Tune α (linear interpolation) or RRF k on a validation set of queries.
• Re‑Ranking Budget: Limit cross‑encoder to ≤ 100 candidates; use LLM only for the final top‑5.
• Cost Monitoring: Set alerts on LLM token usage; consider open‑source alternatives (e.g., Llama‑3‑8B) for high‑volume scenarios.
• Observability: Export latency histograms per pipeline stage; trace end‑to‑end request IDs.
• Security: Encrypt stored embeddings; restrict API keys; audit logs for data leakage.

Conclusion

Building a scalable RAG pipeline is no longer a research‑only endeavor; it’s an engineering problem that can be solved with a thoughtful combination of hybrid search, advanced re‑ranking, and robust orchestration. By fusing BM25’s lexical precision with dense vector semantics, and then applying sophisticated re‑ranking (cross‑encoders, LLMs, or learning‑to‑rank models), you achieve both high recall and high precision—essential for real‑world deployments where user trust depends on accurate, grounded answers.

The end‑to‑end example provided demonstrates how to:

1. Ingest and chunk documents.
2. Build both sparse and dense indexes.
3. Perform hybrid retrieval with RRF fusion.
4. Refine results using a cross‑encoder.
5. Generate a citation‑rich answer with an LLM.

With proper caching, quantization, and horizontal scaling, the pipeline can serve thousands of queries per second while keeping operational costs manageable. Adding observability ensures you can iterate, debug, and continuously improve the system as your knowledge base grows.

Whether you’re powering a customer‑support chatbot, a legal research assistant, or an internal enterprise search, the principles and code patterns outlined here give you a solid foundation to build on. Happy indexing, and may your answers always be grounded!

Resources

• FAISS – Facebook AI Similarity Search – A library for efficient similarity search and clustering of dense vectors.
  FAISS GitHub Repository

• Elasticsearch – The Distributed Search Engine – Official documentation covering BM25, dense_vector fields, and hybrid search.
  Elasticsearch Documentation

• OpenAI API – Chat Completion – Guides on using OpenAI’s models for generation and re‑ranking.
  OpenAI API Docs

• Sentence‑Transformers – State‑of‑the‑art Sentence Embeddings – Pre‑trained models and tutorials for both bi‑encoders and cross‑encoders.
  Sentence‑Transformers Documentation

• Learning to Rank with XGBoost – Practical guide to building LTR models for information retrieval.
  XGBoost Ranking Tutorial

These resources, combined with the code snippets above, should equip you to design, implement, and operate a production‑grade RAG system that scales gracefully while delivering high‑quality, source‑grounded answers.