TL;DR — Static rate limits and hard circuit breakers fail the moment traffic shape changes. Adaptive load shedding — driven by live latency, queue depth, and error budgets — protects backends during overload while degrading gracefully. Pair it with tiered failover (read replicas, cached fallbacks, soft errors) and you turn a cascading outage into a survivable brownout.

Why Static Thresholds Break Under Real Load

Every team has been here: traffic doubles overnight because of a viral post, a partner integration, or a bot campaign. The rate limiter you set to 1,000 RPS a year ago either lets everything through (and the database melts) or rejects legitimate users (and support gets paged for the wrong reason). The threshold was correct when you tuned it, but the system underneath changed — schema migrations added latency, a new dependency became a hot path, the connection pool shrank during a refactor.

The deeper problem is that overload is not a single number. It is a relationship between incoming demand, service capacity, and patience (how long callers will wait before retrying). Static thresholds assume the first two are constant, which is almost never true in production. Adaptive load shedding assumes the third — that you must measure the system to know what it can handle, and update the answer continuously.

The goal is not to never drop traffic. The goal is to drop the right traffic — low-value, retry-heavy, or already-overloaded callers — while keeping the system in a recoverable state. The difference between a brownout and an outage is whether your service has opinions about what to reject.

The Three Signals Worth Shedding On

Most load-shedding systems pick a single signal — CPU, request rate, or error rate. In practice you want a small set of orthogonal signals, each catching a different failure mode. A useful baseline is what I think of as the “3 L” framework: Latency, Loss, Length.

Latency

p99 request latency is the earliest indicator that a service is approaching saturation. CPU and memory utilization lag reality because of buffering, queues, and garbage collection. If your p99 starts climbing while p50 stays flat, you are queueing. The moment tail latency exceeds the client’s timeout, those requests will be retried, multiplying the load you are about to receive. The Tail at Scale by Dean and Barroso remains the definitive writeup on why tail latency dominates user experience in distributed systems.

A practical rule: shed traffic when p99 latency exceeds 2× the steady-state value for more than 30 seconds. The 2× factor absorbs normal noise; the 30-second window filters out GC pauses and batch jobs. Both numbers will be wrong for your system — the point is to have numbers you can argue about.

Loss (Error Rate)

5xx responses are a lagging indicator — by the time errors are climbing, the system is already in trouble. But a bounded error rate is healthy. A target like 0.1% errors is normal; 1% is a warning; 5% is shedding territory. The catch is that some failures are expected during overload and shedding is what produces them. Track the error rate of requests that passed the shedder separately from the total error rate, otherwise your load shedder will be fighting itself.

Length (Queue Depth or Inflight)

The most underrated signal is the number of in-flight requests or queued work items. This is the variable you actually control. Once you exceed the connection pool, the thread pool, or the bounded queue, additional work sits waiting, consuming memory, and adding latency that compounds. A common pattern: set the upper bound equal to the concurrency limit of the slowest downstream, and shed above that.

class AdaptiveShedder:
    def __init__(self, inflight_limit: int, p99_baseline_ms: float):
        self.inflight = 0
        self.inflight_limit = inflight_limit
        self.p99_baseline_ms = p99_baseline_ms
        self.recent_p99 = p99_baseline_ms

    def should_admit(self, request) -> bool:
        # Hard ceiling: never let inflight exceed capacity.
        if self.inflight >= self.inflight_limit:
            return False

        # Soft pressure: as latency climbs, probabilistically reject.
        pressure = self.recent_p99 / self.p99_baseline_ms
        if pressure > 1.5:
            # At 2x baseline, reject ~25% of new traffic.
            # At 3x baseline, reject ~50%.
            reject_probability = min(0.5, (pressure - 1.0) * 0.5)
            if random.random() < reject_probability:
                return False

        self.inflight += 1
        return True

    def on_complete(self, latency_ms: float):
        self.inflight -= 1
        self.recent_p99 = 0.9 * self.recent_p99 + 0.1 * latency_ms

The key trick is the 0.9 / 0.1 EWMA on p99 — it smooths out spikes without lagging too far behind reality. The same idea appears in GCP’s design docs for their load balancing, where regional load balancers use rolling windows to detect unhealthy backends.

Patterns in Production: How Real Systems Shed

Pattern 1: Token Bucket Per Tier (Netflix Hystrix-style)

The classic approach: each upstream caller has a bucket, refilled at a configured rate. When the bucket empties, requests are rejected. The advantage is fairness — a chatty microservice cannot starve a quiet one. The disadvantage is tuning: every consumer-dep pair needs its own bucket.

In practice, teams move to bulkheading — isolating pools of resources (threads, connections, memory budgets) per dependency. If the recommendation service is degraded, the rest of the API still serves. This is exactly what Hystrix (now in maintenance mode, but the patterns survive in Resilience4j) implemented. A bulkhead is just a load shedder with a fixed capacity, scoped to a single dependency.

Pattern 2: CoDel / FQ-CoDel for Queue Management

Borrowed from network engineering, CoDel (Controlled Delay) treats queue length as a function of sojourn time — how long a request has been waiting. When the minimum sojourn time over a sliding window exceeds 5ms for 100ms, packets start being dropped. The system self-tunes because the threshold is in latency space, not queue length.

This pattern translates directly to HTTP services: instead of “drop when queue > 1000,” drop when “the oldest queued request has been waiting > 20ms.” You can implement this with a Redis sorted set keyed on enqueue timestamp, though most production systems use an in-process structure for the hot path.

Pattern 3: Admission Control via Health Endpoints

Many large platforms expose a lightweight /health or /status endpoint that returns 200 OK with a JSON body indicating current load. A central load balancer or service mesh uses this to compute admission decisions. AWS’s Application Load Balancer uses this pattern: target groups report health, and unhealthy targets are removed from the rotation within seconds.

The pattern is powerful because it pushes the decision-making to the service that knows itself best. The cost is the polling interval — typically 5–30 seconds — which is far too slow for millisecond-level overload. Use it as a coarse-grained signal, not a fine-grained one.

Pattern 4: Feedback Loops (PID Controllers)

The most ambitious pattern treats load shedding as a control theory problem. You have a setpoint (e.g., p99 = 100ms), a measured variable (current p99), and an actuator (the reject rate). A PID controller computes the reject rate needed to bring p99 back to the setpoint. The Envoy proxy’s adaptive concurrency filter implements a variant of this, originally based on the TCP BBR congestion control algorithm.

The upside is that the system finds its own operating point. The downside is that you now have a controller with three gains to tune, and a misconfigured controller can oscillate — overshoot, drop everything, recover, overshoot again. Treat it like any other production system: instrument it, alert on it, and have a fallback to static thresholds if the controller goes haywire.

Setting Thresholds That Don’t Lie to You

Threshold-setting is where most load-shedding systems fail. The numbers you write down on a whiteboard are wrong the moment they touch production. Here are the rules I have learned the hard way.

Use a baseline, not a constant. A p99 of 200ms is fine for one endpoint and catastrophic for another. The threshold should be k * baseline_p99 where k is empirically determined during load tests. Recompute the baseline weekly with a job that reads from your metrics store.

Hysteresis prevents flapping. If you start shedding at 1000 RPS and stop at 900 RPS, you will oscillate — traffic drops, shedding stops, traffic returns, shedding restarts, support pages you. Use separate thresholds for entering and exiting the shedding state, with a cooldown window. Uber’s engineering blog has a good writeup of this for their gRPC stack.

Tie thresholds to SLOs, not utilization. “CPU at 80%” is a heuristic; “p99 latency below 200ms” is a user-facing commitment. Use the SLO as the threshold. If you violate the SLO, you are shedding the right amount of traffic when the SLO recovers.

Account for cold starts and diurnal patterns. Your system is more fragile at 3am when the cache is cold and the read replica hasn’t warmed up. Either set thresholds dynamically based on time-of-day, or use absolute tail-latency thresholds that don’t care about time of day.

Failover: What Happens After You Start Shedding

Load shedding and failover are often discussed separately, but they are two halves of the same strategy. Once you have decided to reject some traffic, you need a graceful place for it to land. The options, from best to worst:

  1. Serve from cache. If the requested data is in a CDN or in-memory cache, serve the cached version with a Warning header indicating it is stale. This is the gold standard — the user gets something correct, your backend gets a break, and the freshness story is honest.
  2. Serve a degraded response. Compute what you can without the failing dependency. The recommendations service is down? Return a static “popular items” list. The search index is degraded? Return a substring match from the primary database. Stripe’s graceful degradation patterns are worth studying here.
  3. Fail over to a replica. If you have a read replica and the primary is overloaded, route reads to the replica. The replica may also be struggling, but at least it isolates the failure to a smaller subset of queries. This is a standard pattern in PostgreSQL HA setups.
  4. Queue for later. If the request is non-urgent (analytics events, batch jobs, notifications), push it onto a durable queue and process asynchronously. The user gets a 202 Accepted and a webhook when the work is done.
  5. Return a structured 503 with Retry-After. This is the honest default. Include a Retry-After header in seconds, and a JSON body explaining the failure mode. Do not return a 500 — a 503 tells well-behaved clients (and load balancers) that retrying after a delay is appropriate.

The worst option is the silent fail: returning a 500 with no context, where the client retries immediately, which doubles the load, which trips the shedder again. A well-designed 503 with backpressure is a feature, not a failure.

Architecture: A Reference Adaptive Shedding Layer

Here is a layout I have used on more than one system. It is not the only way, but it is a sane starting point.

                ┌──────────────────────────────────┐
   Client ───▶  │   Edge (CDN / Load Balancer)    │  ← coarse health checks
                │   - rate limit per IP / token   │     (5–30s polling)
                └──────────────┬───────────────────┘
                               │
                ┌──────────────▼───────────────────┐
                │  Service Mesh / API Gateway      │  ← fine-grained
                │  - adaptive shedder (PID-style)  │     admission
                │  - bulkhead per downstream       │     (per-request)
                │  - circuit breaker per dep       │
                └──────────────┬───────────────────┘
                               │
        ┌──────────┬───────────┼───────────┬──────────┐
        ▼          ▼           ▼           ▼          ▼
     Cache     Read         Primary     Async      Replica
   (stale OK)  Path        Write Path   Queue     (fallback)

The edge layer handles coarse-grained abuse (DDoS, scrapers, misbehaving SDKs). The service mesh layer handles fine-grained overload — the kind that happens 100ms before a deployment goes bad. Each downstream has its own circuit breaker, so a failure in one dependency does not cascade.

The async queue is the safety valve: when the synchronous path is shedding >50% of traffic, route the rest to a queue and process it in the background. This is the pattern behind systems like LinkedIn’s Kafka-based overload handling, where the fronting service is sized for steady-state load and the queue absorbs spikes.

Key Takeaways

  • Measure three signals, not one. Latency (p99), error rate, and inflight count together cover most overload regimes. Any one of them alone is a lagging indicator.
  • Use adaptive, not static, thresholds. A baseline-relative threshold (2× p99 baseline) survives schema changes, traffic growth, and refactors. A hard-coded RPS limit does not.
  • Shed the right traffic, not the most traffic. Tier your callers (premium, free, internal, scraper) and shed from the bottom. The same shedder configured differently is the difference between losing a free-tier user and losing an enterprise contract.
  • Pair shedding with failover. A 503 with Retry-After is honest. A cached fallback is better. A silent failure is the worst option in production.
  • Hysteresis and EWMA are not optional. Without them, the system will flap between “shedding” and “not shedding” and produce a worse experience than either steady state.
  • Test the shedder. Chaos-test it: kill a downstream, double the traffic, slow the database by 200ms. Verify that the shedder activates, the system survives, and the recovery is graceful. A shedder that has never fired in production is a shedder that will fail the first time it matters.

Further Reading