TL;DR — Distributed systems fail in predictable ways: networks partition, processes crash, clocks drift. The nine patterns below are the standard toolkit engineers reach for to stay reliable through those failures. Each one trades off consistency, latency, and complexity in a specific way, and most production stacks combine several of them at once.

If you’ve ever debugged a flaky microservice, watched a Kafka consumer rebalance loop, or wondered why your database uses a commit log under the hood, you’ve already brushed against these patterns. They’re not academic — they show up in Amazon’s architecture papers, the design of Cassandra’s consistency model, and the Kubernetes control plane.

Let’s walk through the nine that matter most.

1. Leader Election

In any cluster that needs a single source of truth at a given moment — a primary database, a scheduler, a lock manager — you need to pick one node to be in charge. That’s leader election.

The hard part isn’t picking a leader. It’s agreeing on a leader while the network is misbehaving. Two nodes that can’t reach each other might both think they’re the leader, a classic split-brain scenario. That’s why every serious implementation leans on a consensus algorithm under the hood.

  • etcd and Consul use Raft to elect a leader with a majority quorum. Writes only succeed if a majority of nodes acknowledge them, so two leaders can’t both accept writes.
  • ZooKeeper does the same thing with Zab, a predecessor to Raft.
  • Kafka elects a controller broker using ZooKeeper (or KRaft in newer versions) and ensures exactly one controller at a time.

The trade-off: leader election requires a quorum of nodes to be reachable, which means your cluster’s availability is bounded by the failure tolerance you configured. Three-node Raft tolerates one failure; five-node tolerates two.

2. Quorum and Consistent Reads

Once you have replicas, you face a question on every read: how many replicas must agree before you can return a value? The answer defines your consistency model.

  • Write quorum (W): number of replicas that must acknowledge a write.
  • Read quorum (R): number of replicas that must respond to a read.
  • Replication factor (N): total replicas.

The classic rule is R + W > N, which guarantees that any read quorum overlaps with the most recent write quorum. Set this and you get strong consistency; loosen it and you slide toward eventual consistency.

DynamoDB and Cassandra both expose this knob directly. A quorum read in Cassandra (R = ⌊N/2⌋ + 1) returns the most recent write as long as the write also used a quorum. Configure R=1, W=N for fast writes and stale-tolerant reads; the reverse for the opposite. The Cassandra architecture guide walks through the math.

3. Heartbeats and Gossip

How does a node tell whether its peers are alive? Two patterns dominate.

Heartbeats are explicit, periodic signals: “I’m here.” The leader sends heartbeats to followers; followers respond or time out. If a follower misses too many, the leader assumes it’s gone. TCP keepalives, Kafka’s session timeouts, and Raft’s election timeouts are all heartbeats. They’re precise but chatty — every node talks to every other node it’s tracking.

Gossip is the opposite. Each node, every second or so, picks a peer at random and exchanges a digest of what it knows. Over time, information spreads like a rumor through the cluster. Cassandra uses gossip for failure detection and schema dissemination; Consul uses SWIM-style gossip. The trade-off: gossip is bandwidth-efficient and scales to thousands of nodes, but it converges probabilistically — you can never say “every node knows X right now,” only “most nodes will know X within a few seconds.”

4. The Saga Pattern

Distributed transactions are hard. Two-phase commit gives you atomicity across services but kills availability when coordinators hang. The Saga pattern sidesteps the problem by accepting eventual consistency and modeling business processes as a chain of local transactions with compensating actions.

Consider a travel booking: reserve flight → charge card → reserve hotel → email confirmation. If the hotel reservation fails, you need to refund the card and cancel the flight. Each step has a forward action and a compensating action. Two flavors exist:

  • Orchestrated: a central coordinator tells each service what to do. Easier to reason about, but the orchestrator is a single point of failure (mitigated with retries and persistence).
  • Choreographed: services publish events and react to each other. More decoupled, but the flow is implicit and harder to debug.

Netflix’s Conductor and Temporal are popular orchestrator implementations. The pattern is everywhere in microservices: order fulfillment, payment processing, anything that crosses service boundaries.

5. CQRS (Command Query Responsibility Segregation)

Most systems use the same data model for reads and writes. That’s fine for a CRUD app. It falls apart when read and write workloads diverge — say, a write-heavy OLTP database feeding a dashboard with millions of aggregates.

CQRS says: split them. Writes go to a normalized store optimized for transactions. Reads come from a denormalized store — a search index, a Redis cache, a columnar warehouse — that’s tuned for the queries you actually run. The two are kept in sync via events, often the same event log you’d use for sagas.

The wins are real: you can scale reads independently, use specialized stores (Elasticsearch for search, Druid for analytics), and isolate the write path so a slow query doesn’t tank your transactions. The cost is duplication and the eventual consistency window between the two sides.

6. Event Sourcing

Most systems store current state. Event sourcing flips that: store the sequence of events that led to the current state, and derive state by replaying them.

A bank account isn’t a row with a balance; it’s a list of Deposited, Withdrawn, Transferred events. The balance is fold(events). This gives you a perfect audit log for free, time-travel debugging, and the ability to build new read models by replaying history into a new projection.

It’s not free. Schemas evolve, events are immutable (you have to version them with upcasters), and “current state” queries get slow without snapshots. EventStoreDB, Kafka with compacted topics, and Axon Framework all support the pattern. Pair event sourcing with CQRS and you have the architecture behind many event-driven systems.

7. Bulkhead

The name comes from shipbuilding: a bulkhead is a sealed compartment that contains flooding. In software, a bulkhead isolates workloads so a failure in one doesn’t sink the others.

A concrete example: a single thread pool serving all your downstream calls. If one slow service starts timing out, every thread gets stuck, and your whole API goes down. With bulkheads, you give each downstream its own thread pool, connection pool, or even its own process. One service can be drowning while the rest of your system stays responsive.

Hystrix popularized the pattern in the JVM world, and although Netflix has since deprecated it, the idea lives on in Resilience4j and in service meshes like Istio, where you can apply bulkheads per service via destination rules.

8. Circuit Breaker

Related to bulkheads but distinct: a circuit breaker sits in front of a downstream call and tracks its failure rate. When failures cross a threshold, the breaker “trips” and short-circuits future calls, returning an error immediately instead of letting them pile up and time out.

Three states:

  • Closed: calls flow normally; the breaker counts failures.
  • Open: calls fail instantly; the breaker waits for a cooldown.
  • Half-open: a few trial calls go through; if they succeed, the breaker closes again.

This gives the downstream time to recover instead of being hammered by a thundering herd. It’s an essential companion to retries — retrying a broken service is one of the most reliable ways to make an outage worse. The AWS Builders’ Library has a great writeup on combining circuit breakers with exponential backoff and jitter.

9. Idempotency with Idempotent Receivers

Distributed systems deliver messages at-least-once. Networks duplicate packets, consumers crash after processing but before acknowledging, and Kafka redelivers on rebalance. The receiver must be prepared to see the same message twice — or ten times.

The fix is an idempotent receiver: a consumer that, given the same input, has the same effect as processing it once. Implementations:

  • Dedup table: store seen message IDs in a fast lookup (Redis SET with TTL) and skip duplicates.
  • Idempotency keys: the producer attaches a unique key; the consumer stores the result keyed by it. Stripe’s idempotency layer is the canonical write-up.
  • Natural keys: design operations so the message itself describes the desired end state. set_balance(account=42, value=100) is idempotent; increment_balance(account=42) is not.

Without idempotency, every retry is a risk of double-charging a customer or creating two orders. With it, you can retry aggressively, which lets you combine it with circuit breakers and backoff safely.

Patterns in Production

These nine patterns don’t show up in isolation. A serious production stack layers them:

  • Postgres with a hot standby uses leader election (the primary), synchronous replication (quorum writes for durability), and WAL shipping (an event log feeding replicas).
  • A Kafka-backed order pipeline uses event sourcing (the order topic is the log), CQRS (a normalized write DB and an Elasticsearch read store), the Saga pattern (orchestrated by a service that listens to events), and idempotent consumers (dedup keys in Redis).
  • An API gateway in front of microservices uses circuit breakers per upstream, bulkhead-style thread isolation, retries with exponential backoff, and health checks (heartbeats) that feed into the load balancer.

The art isn’t picking one pattern. It’s knowing which combinations compose well and which fight each other. Event sourcing plus synchronous strong consistency is almost always a mistake; saga plus idempotency is almost always a win.

Key Takeaways

  • Leader election and consensus are inseparable. If you need exactly one coordinator, you need a majority quorum algorithm under the hood.
  • Quorum math defines your consistency story. R + W > N gives strong consistency; looser tunings trade it for latency.
  • Heartbeats are precise, gossip is scalable. Pick based on cluster size and how fast you need failure detection.
  • Sagas replace distributed transactions with eventual consistency and compensating actions.
  • CQRS and event sourcing pair well when read and write workloads diverge or you need a full audit log.
  • Bulkheads, circuit breakers, and idempotency are the resilience trio: isolate failures, fail fast, and make retries safe.

Further Reading