TL;DR — etcd’s Raft implementation provides a battle-tested foundation for fault-tolerant distributed caches by serializing writes through a elected leader, replicating a replicated log across a quorum of nodes, and recovering automatically from failures. Understanding its internals — leader election, log replication, and membership changes — is essential for anyone building reliable infrastructure at scale.

Introduction

Distributed caches have become the invisible backbone of modern cloud-native infrastructure. From session stores and feature flag repositories to configuration centers and rate-limiting backends, the demand for low-latency, highly available key-value stores has never been greater. But availability without consistency is a trap — stale reads, split-brain scenarios, and silent data corruption can cascade into outages that take down entire platforms.

etcd emerged as the de facto standard for distributed coordination, famously powering Kubernetes’ own control plane. At its core, etcd solves the hardest problem in distributed systems: how do N independent nodes agree on a single sequence of operations, even when some nodes crash or networks partition? The answer is the Raft consensus algorithm, which etcd implements in production-grade Go code that has been hardened over more than a decade of real-world use.

This post takes a deep dive into etcd’s Raft implementation — not from a textbook perspective, but from the standpoint of an engineer architecting a fault-tolerant distributed cache. We’ll examine the protocol mechanics, the production-hardened engineering decisions, and the architectural patterns that separate a working prototype from a system that survives real traffic.

The Raft Consensus Engine

How Raft Replaces Centralized Coordination

Traditional distributed caches like Memcached rely on client-side sharding and eventual consistency. Redis Sentinel offers failover but requires careful tuning of quorum thresholds. etcd takes a fundamentally different approach: it uses a replicated state machine driven by the Raft consensus algorithm. Every write is proposed to the cluster, ordered through consensus, and applied identically on every node. The result is a linearizable store where every reader sees the same sequence of updates, regardless of which node they connect to.

The Raft paper, published by Diego Ongaro and John Ousterhout in 2014, decomposed consensus into three subproblems: leader election, log replication, and safety. etcd implements all three with production-specific extensions that we’ll explore below.

Leader Election: The Heartbeat-Driven Approach

When an etcd cluster starts — or when the current leader fails — nodes enter an election state. Each node begins with a randomized election timeout (typically between 150ms and 300ms in etcd’s default configuration). If a node receives no heartbeat from a leader before its timeout expires, it transitions to candidate, increments its term number, and votes for itself.

// Simplified etcd election state transition
func (n *node) campaign(t Term) {
    n.state = StateCandidate
    n.Term = t
    n.Vote = n.ID
    // Send RequestVote RPCs to all peers
    for _, peer := range n.peers {
        go n.sendRequestVote(peer)
    }
}

The first candidate to receive votes from a majority of the cluster becomes the new leader. The randomization of election timeouts is critical — without it, all nodes would time out simultaneously, leading to repeated split votes and indefinite election loops. etcd uses a randomized election timeout between 150ms and 300ms by default, which empirically produces stable leader tenure under typical network conditions.

A key production detail: etcd supports pre-votes (introduced in etcd 3.4). Before entering a campaign state, a node first sends a pre-vote request to check whether another leader might still be active. This prevents a partitioned minority from unnecessarily triggering elections, which reduces unnecessary log churn and improves recovery time.

Log Replication: Safety Through Quorum

Once a leader is established, all client writes flow through it. The leader appends the command to its local log and then sends AppendEntries RPCs to followers. A write is considered committed only when it has been replicated on a majority of nodes. The leader then applies the entry to its state machine and responds to the client.

Client → Leader: PUT /config/feature-x true
Leader → Log: Append entry {term: 5, index: 42, command: PUT /config/feature-x true}
Leader → Followers: AppendEntries RPC with entry at index 42
Followers: Persist entry, reply ACK
Leader: Entry committed once majority acknowledges
Leader → All Followers: Apply entry to state machine
Leader → Client: OK

The critical invariant here is Log Matching: if two logs contain an entry with the same index and term, then every entry preceding that index and term is identical in both logs. etcd enforces this by including the term and index of the previous entry in every AppendEntries RPC, ensuring that followers can detect and correct inconsistencies.

etcd also uses batch append — the leader aggregates multiple client requests into a single AppendEntries RPC, amortizing network overhead across operations. In benchmarks, this batching can improve throughput by 2–5x compared to one-RPC-per-write, depending on the client workload and network latency.

Production Hardening in etcd

Snapshotting and Log Compaction

A naive Raft implementation would accumulate an unbounded log, eventually exhausting disk and making slow followers impossible to catch up. etcd addresses this with snapshotting: the leader periodically takes a snapshot of the current state machine and truncates the log up to the snapshot’s included index.

# Triggering a manual snapshot in etcd
etcdctl snapshot save /var/etcd/snapshot.db

Snapshots are transmitted efficiently using install-snapshot RPCs, where the leader streams the snapshot to lagging followers in chunks. etcd also supports incremental snapshots (added in etcd 3.5), which only transfer the changes since the last snapshot, reducing recovery bandwidth for large clusters.

Membership Changes: Adding and Removing Nodes

etcd supports joint consensus for membership changes, allowing the cluster to reconfigure without downtime. When adding a new node, the cluster transitions to a joint configuration where both old and new members must agree on each log entry. Once the new node has caught up, the cluster removes the old configuration.

# etcd cluster membership update
etcdctl member add node4 --peer-urls=http://10.0.0.4:2380
etcdctl member remove node2

The joint consensus approach guarantees safety during reconfiguration: even if the old and new majorities overlap on only a single node, the cluster cannot lose committed entries. This is a non-trivial property — many naive implementations of dynamic membership allow split-brain scenarios during reconfiguration.

Lease-Based Consistency for Cache Reads

For distributed cache reads, etcd provides leases — time-bound revocable tokens attached to keys. A lease grants a key a TTL, and if the lease expires, the key is automatically deleted. More importantly for cache architectures, leases enable linearizable reads: the leader contacts a quorum of followers to confirm it is still the leader before responding to a read request.

// Grant a 60-second lease
leaseGrantResponse, err := client.Grant(ctx, 60)
// Attach the lease to a cache key
client.Put(ctx, "cache:user:42", profileData, client.WithLease(leaseGrantResponse.ID))

This is critical for cache architectures where stale reads could cause incorrect routing, feature flag evaluation, or authorization decisions. Without linearizable reads, a follower that hasn’t heard from the leader could serve a stale value — a subtle but dangerous failure mode in production systems.

Architecture Patterns in Production

Pattern 1: etcd as a Configuration Store with Local Caching

The most common production pattern is to use etcd as the source of truth for configuration, while each application node maintains a local in-memory cache. The application watches etcd for changes using watch streams and updates its local cache accordingly. This gives you the consistency guarantees of etcd for writes while serving reads from a local cache with sub-millisecond latency.

# Python example: watching etcd for config changes
import etcd3

client = etcd3.client(host='etcd-cluster.local')

def on_change(event):
    if event.key == b'/config/feature-flags':
        update_local_cache(event.value)

watch_iter = client.watch_prefix(b'/config/')
for event in watch_iter:
    on_change(event)

This pattern is used by Kubernetes itself — the kubelet watches etcd for pod specifications and maintains a local pod cache. The key engineering insight is that the watch mechanism uses etcd’s revision-based history, so if a connection drops, the client can resume from the last seen revision without missing updates.

Pattern 2: Service Discovery with Health Checks

etcd’s lease mechanism powers service discovery in production environments. Each service instance registers itself with a lease-backed key. If the instance crashes, the lease expires and the key is deleted, automatically removing it from the discovery registry. Combined with client-side load balancing, this creates a self-healing service mesh without a separate discovery server.

# Service registration with lease
PUT /services/payment-gateway/node-3 {"host": "10.0.1.3", "port": 8080}
Lease: 30s

The 30-second lease TTL provides a balance between failure detection speed and network noise tolerance. Too short, and transient network partitions cause false deregistrations; too long, and failed instances linger in the registry, causing client errors.

Pattern 3: Distributed Locking for Cache Invalidation

When multiple nodes need to invalidate or warm a shared cache entry, etcd provides distributed locks via its CompareAndSwap (CAS) operations. A node acquires a lock by creating an ephemeral key with a unique value, and other nodes wait for the lock to be released.

// Distributed lock for cache invalidation
txn := client.Txn(ctx)
txn.If(
    client.Compare(client.Value(lockKey), "=", ""),
).Then(
    client.OpPut(lockKey, nodeID, client.WithLease(leaseID)),
).Else(
    client.OpGet(lockKey),
)
resp, err := txn.Commit()

This pattern is essential for preventing cache stampede — a scenario where a popular cache key expires and hundreds of nodes simultaneously attempt to regenerate it, overwhelming the underlying data source.

Failure Modes and Mitigations

Split-Brain Prevention

Raft guarantees that at most one leader exists per term. In etcd, a split-brain scenario would require a network partition where both sides have a majority of votes — which is impossible with an odd-numbered cluster (3, 5, or 7 nodes). This is why etcd clusters are always deployed with an odd number of nodes.

Slow Follower Recovery

When a follower falls significantly behind, etcd uses truncation — the leader forces the follower to discard its inconsistent log entries and replace them with the leader’s version. This is why the Log Matching property is so important: the follower’s log is rolled back to the last matching entry, and the leader’s log is grafted on from there.

Disk Latency and Write Stalls

etcd writes every Raft entry to disk before acknowledging it to the client. If disk I/O becomes slow — due to saturation, hardware failure, or kernel issues — the leader’s heartbeat interval can exceed the election timeout, triggering an unnecessary leader change. Production best practices include using SSDs, isolating etcd disk I/O on dedicated volumes, and tuning the --heartbeat-interval and --election-timeout flags based on observed disk latency percentiles.

Key Takeaways

  • etcd’s Raft implementation provides linearizable consistency through leader-based log replication, making it suitable for distributed caches where stale reads are unacceptable.
  • Leader election uses randomized timeouts and pre-votes to prevent split votes and unnecessary re-elections in production environments.
  • Quorum-based commits ensure durability — a write is durable as long as a majority of nodes survive, which is why 3-node and 5-node clusters are the standard deployment sizes.
  • Leases and watch streams enable production cache patterns — local caching with linearizable reads, service discovery with health checks, and distributed locking for cache invalidation.
  • Snapshotting and joint consensus solve operational scaling challenges — log compaction prevents unbounded growth, and membership changes allow cluster reconfiguration without downtime.
  • Disk I/O is the most common production bottleneck — dedicated SSDs and careful tuning of heartbeat/election timeouts are essential for stability.

Further Reading