TL;DR — Semantic search is a distributed systems problem the moment your index crosses a few hundred million vectors. Production-grade vector databases combine sharded ANN indexes (HNSW, IVF, or DiskANN), consistent-hashing replication, hybrid sparse-plus-dense retrieval, and careful observability to deliver sub-100ms p99 recall at scale.

Why “Vector DB” Is Really a Distributed Systems Problem

A single-node vector index is straightforward. You embed your corpus, build an HNSW graph, and serve approximate nearest neighbor (ANN) queries in milliseconds. The trouble starts when your corpus stops fitting in RAM on one machine, your QPS exceeds what one CPU can serve, or your customers expect 99.9% availability. At that point, you are no longer running a vector index — you are running a distributed database whose primary access pattern is nearest-neighbor search.

The hard problems shift accordingly. You need to partition the vector space across nodes in a way that preserves locality, since random sharding destroys ANN performance. You need replication that stays consistent under concurrent writes, but you do not want to pay the cost of synchronous consensus on every embedding insert. You need a query coordinator that can scatter-gather across shards, merge candidates, and return top-k in bounded time. And you need all of this without losing the recall that made semantic search useful in the first place.

This post walks through the architecture choices that determine whether your semantic search system stays up at 10x growth or quietly degrades into a relevance nightmare.

The Core Components of a Distributed Vector Database

Before diving into trade-offs, it helps to name the moving parts. Most production systems — whether you use Milvus, Qdrant, Weaviate, Pinecone, or a hand-rolled stack on top of FAISS — share the same conceptual layers.

  • Embedding pipeline — turns text, images, or events into dense float vectors, typically 384 to 1536 dimensions.
  • Index layer — stores vectors in an ANN structure (HNSW, IVF-PQ, ScaNN, DiskANN) tuned for recall-vs-latency.
  • Partitioning layer — decides which shard owns which vector. This is the single most consequential design decision.
  • Replication layer — keeps N copies of each shard consistent (or eventually consistent) across failures.
  • Coordinator / proxy — accepts a query, fans it out to the relevant shards, merges candidates, and returns top-k.
  • Metadata and filter layer — handles scalar filters (date ranges, tenant IDs, tags) that almost always accompany vector queries.
  • Observability and ops — recall monitoring, shard rebalancing, index compaction, and the inevitable incident response.

A useful mental model: a vector database is a search engine whose documents happen to live in a high-dimensional space and whose ranking is computed geometrically rather than via BM25.

Sharding Strategies: Why Random Sharding Is a Trap

The most common early mistake is to shard vectors by hashing the vector ID or by hashing a random key. This is fine for key-value workloads, but devastating for ANN search, because semantically similar vectors — which the user actually wants returned together — end up scattered across every shard in the cluster. Each shard has to return its own local top-k, and the global top-k is reconstructed from a thin slice of candidates at each node. Recall collapses.

Locality-Preserving Sharding

Production systems prefer locality-preserving partitioning:

  • By tenant or partition key — if you serve a multi-tenant SaaS and each tenant’s data is searchable only within their namespace, shard by tenant ID. Each query is routed to a single shard, and recall is identical to a single-node system. This is the easiest path to scale and the one Pinecone’s “pod-per-tenant” model and Qdrant’s collection-level sharding both exploit.
  • By cluster (vector quantization) — train a k-means or product quantizer over the corpus, assign each centroid to a shard, and route each vector to the shard that owns its nearest centroid. At query time, probe the top-N centroids, which usually overlap a small subset of shards. Milvus uses this approach with its “partition key” feature and Weaviate exposes it as replication-aware sharding.
  • By metadata range — when your vectors carry structured attributes (time, region, category), range partitioning is unbeatable for filtered queries: a “last 7 days in EU” query only touches one shard.

Consistent Hashing with Virtual Nodes

For the metadata and key-value planes that wrap the index, consistent hashing with virtual nodes remains the standard. Each physical node owns many positions on the hash ring, so adding or removing a node redistributes roughly 1/N of the keys instead of triggering a full reshuffle. This is what powers the replication factor and rebalancing logic in Cassandra-style systems and shows up in vector databases that compose with object storage or metadata stores.

The key insight: sharding decisions should be made on the predicate, not on the vector itself. The vector ID is the worst possible shard key for ANN because similarity and identity are orthogonal.

Index Choices: HNSW, IVF, DiskANN, and the Memory Wall

Sharding decides where vectors live. The index decides how they are searched. The trade-off is almost always memory footprint vs. query latency vs. recall, and there is no free lunch.

HNSW (Hierarchical Navigable Small World)

HNSW, described in the original paper by Malkov and Yashunin (2018), is the workhorse of low-latency semantic search. It builds a multi-layer proximity graph where each layer is a navigable small-world network. Queries greedily descend from a sparse top layer to a dense bottom layer, achieving O(log N) average hops. HNSW delivers excellent recall (>0.95 at 10-recall@10 is common) and predictable sub-millisecond latency.

The catch: HNSW is memory-hungry. Each vector needs its embedding plus ~16–32 graph edges per layer, and the whole graph must fit in RAM to be fast. At 768-dimensional float32 vectors plus graph overhead, a billion-vector index is roughly 4–6 TB of RAM — feasible only with serious hardware. The HNSWlib implementation and Faiss’s HNSW are the two most common production choices.

IVF + Product Quantization (IVF-PQ)

IVF-PQ, the FAISS default for large-scale retrieval, partitions the space into k Voronoi cells (the “IVF” part) and compresses each vector into a short code via product quantization (the “PQ” part). A typical setup is IVF65536,PQ32, which fits a billion vectors into roughly 100–150 GB of RAM at the cost of some recall. The Faiss wiki is the canonical reference.

IVF-PQ shines when you can batch queries and tolerate 5–20ms latency. It is less competitive for very low-latency single-query workloads because the coarse quantizer must be probed across many cells.

DiskANN and SSD-Resident Indexes

When RAM is the bottleneck, DiskANN (Microsoft Research) keeps the full-precision vectors on NVMe SSDs and a slim graph index in memory, achieving near-RAM recall at a fraction of the cost. The DiskANN paper and code report >0.99 10-recall@10 on billion-scale datasets. Qdrant’s built-in memmap and scalar quantization modes and Milvus’s experimental DiskANN integration follow the same principle: trade SSD bandwidth for RAM.

Practical Rule of Thumb

WorkloadRecommended indexWhy
<10M vectors, low latencyHNSW in RAMSimple, fast, fits on one box
10M–500M vectors, mixed loadHNSW with scalar or product quantization2–4x memory reduction, <5% recall loss
500M–5B vectors, batch-heavyIVF-PQ or DiskANNCost-effective, acceptable latency
Streaming / rapidly changingHNSW with delta index + periodic mergeAvoids full rebuilds

Hybrid Retrieval: BM25 + Dense Vectors in One Pipeline

Pure semantic search is overrated. In practice, the highest-recall systems are hybrid — they combine a sparse lexical retriever (BM25 or SPLADE) with a dense ANN retriever, then fuse the rankings via reciprocal rank fusion or a learned reranker. Google’s Vertex AI Search, Elasticsearch’s _knn_search combined with BM25, and Weaviate’s hybrid search all ship this pattern.

The distributed architecture implication: your “vector database” increasingly needs to colocate or co-coordinate with a lexical index. Two patterns dominate:

  1. Unified engine — a single system like Elasticsearch, OpenSearch, or Vespa stores both inverted and HNSW indexes per shard and merges them in the same query plan. Operationally simple, but coupling the two indexes creates contention.
  2. Federated retrievers — separate vector and lexical services, with a thin reranking layer that calls both. Cleaner separation of concerns, more moving parts.

The metadata-filtered vector query is the single most common production pattern: “find the 10 most semantically similar documents in tenant X, category Y, created after date Z.” The system that builds the filtered ANN graph (or pre-partitions by filter) will outperform the one that post-filters a global top-k by an order of magnitude.

Replication, Consistency, and Writes at Scale

Vector writes are awkward. They are append-mostly and idempotent (an embedding for document ID D is just a re-write of D’s vector), but the HNSW graph must be updated under a coarse lock during heavy churn. Naive replication of writes via Raft on every insert is prohibitively slow.

Patterns That Actually Work

  • Write-ahead segments with background indexing — Pinecone’s “pods” and Milvus’s “bulk insert + incremental” both follow this pattern. Writes go to a WAL or segment store; a background process periodically rebuilds or merges the index. Reads see slightly stale data, which is almost always acceptable for semantic search.
  • Per-shard leader-follower with async replication — each shard has one writable leader and 1–2 followers. Writes are synchronous within a shard (Raft/Paxos) and the index update is amortized. Followers serve reads for scale-out QPS. This is Qdrant’s replication mode and Milvus’s architecture.
  • Read-your-writes via routing — send the client’s first read after a write to the leader shard explicitly. Cheap, effective, and avoids global consistency.

Tunable consistency per query ("consistency": "quorum", "majority", or "eventual") is a non-negotiable feature in any production vector database. It lets the application pay for strong consistency only when it matters (e.g., right after an admin ingests a critical document) and accept eventual reads everywhere else.

The Query Path: Scatter-Gather, Merging, and Bounded Latency

A semantic search query in a distributed system looks like this:

  1. Parse and embed — the gateway embeds the query string into a vector (often a separate GPU service).
  2. Route — the coordinator determines which shards are candidates, based on the partition key, filter, and sharding strategy.
  3. Scatter — in parallel, each candidate shard runs its local ANN search and returns its top-k candidates (typically over-fetched by 3–5x).
  4. Merge — the coordinator fuses candidates, applies any reranker, and returns the final top-k.

The latency budget is dominated by step 3: the slowest shard. This is why bounded fan-out matters. If a query hits 20 shards and each takes 30ms, the user sees 30ms (good). If it hits 200 shards, the user sees the slowest one (bad). Smart systems use coarse quantization to prune shards before fanning out: probe nprobe=8 centroids, which usually overlaps 2–4 shards.

Another lever: early termination. If the coordinator already has a strong candidate list after the first few shards respond, it can cancel pending shard queries. This is the RocketQA-style trick adapted to distributed retrieval.

Observability: Measuring What Actually Matters

You cannot operate a vector database without measuring recall, not just latency. The standard operational dashboards are:

  • Recall@k vs. exact brute force — periodically replay a sample of production queries against an exact NN baseline and report recall. A drop from 0.95 to 0.88 is a paging incident.
  • p50/p95/p99 latency per stage — embed, route, scatter, merge. Each stage has its own tail.
  • Index freshness — time between a write and that vector appearing in served results.
  • Sharding skew — vector count and QPS per shard. A 10x skew is a slow disaster.
  • Recall vs. cost — the recall degradation when you turn on aggressive quantization or reduce replication factor.

The community is converging on a standard offline evaluation harness. Anthropic’s context-retrieval benchmarks and the BEIR benchmark suite are useful starting points, though neither captures operational behavior directly.

Key Takeaways

  • Shard by the query predicate, not the vector ID. Tenant key, metadata range, or learned centroid all beat random hashing for ANN.
  • Pick the index to match your scale and budget. HNSW in RAM for <10M, quantized HNSW for 10M–500M, IVF-PQ or DiskANN beyond that.
  • Hybrid retrieval beats pure dense search. BM25 + dense vectors fused via RRF or a reranker consistently wins on recall.
  • Async replication with tunable consistency is the right default. Synchronous consensus on every vector write is too expensive.
  • Bound the fan-out. A scatter-gather across too many shards destroys p99; coarse quantization prunes the search space first.
  • Measure recall, not just latency. A vector database that returns fast but irrelevant results is a liability.
  • Plan for rebalancing. Growth, tenant onboarding, and hardware failures all force shard moves; design your routing layer to handle them without downtime.

Further Reading