TL;DR — When a Redis cluster node fails, hot keys can trigger a cache stampede that overwhelms the remaining nodes. By analyzing key access patterns in real time and pre‑warming replicas, you can reduce failover latency and avoid thundering herds. This post shows how to instrument your cluster, detect hot keys, and apply intelligent request routing to keep latency low during failover.

Redis clusters are a cornerstone of modern microservices, offering horizontal scalability and automatic failover. However, the very mechanism that provides resilience—promoting a replica to replace a failed master—can expose a subtle vulnerability: hot keys. When a single key receives a disproportionate share of traffic, its failure or promotion can cause a cache stampede, flooding the remaining nodes with redundant requests and inflating latency. In this article, we’ll explore how to identify hot keys, understand the failover architecture, and apply analytics‑driven optimizations to mitigate stampede risk.

Understanding Cache Stampede in Redis Cluster

What Is a Cache Stampede?

A cache stampede (or thundering herd) occurs when many clients simultaneously request the same key after it expires or becomes unavailable. In a single‑node Redis setup, this is a localized problem; in a cluster, the effect amplifies because the key may be served by multiple replicas, each of which can receive the full request load during a failover.

Why Hot Keys Amplify the Problem

Hot keys concentrate traffic on a specific hash slot. When the master responsible for that slot fails, the cluster promotes a replica, but the replica inherits the same hot key. If the client population is unaware of the promotion, they continue to send requests to the old master’s IP, causing connection resets and retries. The retry storm can saturate the new master, leading to:

  • Increased latency (up to 10× baseline)
  • Elevated CPU usage on the promoted node
  • Potential cascading failures if the node becomes unresponsive

Architecture: Redis Cluster Failover Mechanics

Node Roles and Replication

A Redis cluster consists of master and replica nodes. Each master owns a subset of hash slots (16,384 total). Replicas mirror the master’s data and provide redundancy. The cluster uses a gossip protocol to detect node failures and coordinate failover.

Failover Triggers and Steps

  1. Failure Detection – Nodes exchange PING/PONG messages. If a master is suspected of failure, it is marked as PFAIL and later FAIL.
  2. Leader Election – The cluster’s master nodes vote to promote a replica of the failed master.
  3. Slot Reassignment – The promoted replica assumes ownership of the failed master’s hash slots.
  4. Client Redirection – Clients receive a MOVED error pointing to the new master.

During step 3, the promoted replica must first sync any pending write operations, which can introduce a latency spike. If the key is hot, the sync window becomes a bottleneck.

Patterns in Production: Hot Key Analytics

Collecting Access Metrics

To detect hot keys, you need visibility into per‑key access counts. Redis 6+ provides the INFO stats command, which includes keyspace_hits and keyspace_misses, but these are aggregate values. For per‑key granularity, you can enable Redis Monitoring or use a custom middleware.

import redis
from collections import defaultdict
import time

r = redis.Redis(host='redis-cluster', port=6379, decode_responses=True)

# Sample key access counter
access_counts = defaultdict(int)

def track_key(key):
    access_counts[key] += 1

# Periodically report top 10 hot keys
while True:
    time.sleep(10)
    top_keys = sorted(access_counts.items(), key=lambda x: x[1], reverse=True)[:10]
    for key, count in top_keys:
        print(f"{key}: {count} accesses")

Real‑Time Hot Key Detection

A production‑grade approach leverages Redis Streams or Pub/Sub to publish key access events. For each GET/SET, a lightweight Lua script can increment a counter in a separate “metrics” key.

-- Lua script to increment per‑key counter
local key = KEYS[1]
local counter_key = 'hotkey:' .. key
redis.call('INCR', counter_key)
redis.call('EXPIRE', counter_key, 10)  -- 10‑second sliding window

By consuming these events, a sidecar process can maintain a top‑K heap and trigger alerts when a key exceeds a threshold (e.g., 1,000 accesses per second).

Optimizing Failover with Hot Key Awareness

Pre‑Failover Pre‑Warming

Before a failover, you can pre‑warm the replica’s cache for hot keys. This reduces the read‑miss penalty after promotion. A simple strategy:

  1. Identify top‑N hot keys from the analytics pipeline.
  2. Issue GET requests to the master to load the values into the replica’s memory.
  3. Use WAIT to ensure replication completes before the failover window.
# Example: pre‑warm top 5 hot keys
for key in $(cat hot_keys.txt); do
  redis-cli -h master-node GET "$key" > /dev/null
done

Intelligent Request Routing

During a failover, clients can be directed to the new master using Redis Cluster’s client‑side redirection. However, you can improve the experience by implementing a proxy layer that:

  • Monitors cluster state via CLUSTER NODES.
  • Maintains a mapping of hash slot → current master.
  • For hot keys, routes requests to the replica with the lowest load, falling back to the master only when necessary.

A lightweight proxy can be built with Envoy or HAProxy using the cluster directive and custom Lua scripts.

Key Takeaways

  • Hot keys are the primary driver of cache stampede during Redis cluster failovers.
  • Instrument per‑key access counts using Lua scripts or a sidecar collector.
  • Pre‑warm replicas for top hot keys before a failover to reduce latency spikes.
  • Route requests intelligently based on real‑time load and cluster topology.
  • Monitor failover latency and set alerts for deviations > 20% of baseline.

Further Reading