TL;DR — Envoy Proxy is one of the most powerful edge proxies available for building resilient API gateways, but its full potential only emerges when you configure circuit breakers, retries, rate limiting, and health checks as a unified resilience strategy. This guide walks through every piece with production-tested configurations you can drop into your own infrastructure today.

Why Envoy for API Gateways

Most teams reach for a gateway solution after watching a cascading failure take down an entire service mesh. Envoy Proxy, originally built at Lyft and now a CNCF graduated project, was designed from the ground up to handle exactly those scenarios. Its architecture — built around a single-threaded non-I/O worker model with multi-threaded listener processors — gives it the throughput and isolation properties that production gateways demand.

Unlike reverse proxies that treat every request as a stateless forwarding operation, Envoy exposes a rich layer of configuration at the route, cluster, and listener level. This means you can apply different resilience policies per upstream service, per endpoint, or even per traffic weight. The result is a gateway that doesn’t just route traffic but actively protects your backend services from one another and from abusive clients.

The key architectural decision that makes Envoy suitable for API gateways is its use of clusters as the unit of upstream connectivity. A cluster defines a set of identical backend hosts along with the load balancing policy, health checking strategy, and circuit breaker thresholds that govern how Envoy interacts with them. Every resilience feature we’ll discuss in this post attaches to the cluster configuration, which makes it straightforward to enforce different policies for different services.

Architecture in Production

A typical production Envoy gateway sits between your clients and your service fleet. The deployment pattern that has worked best for most teams I’ve reviewed places Envoy as a sidecar alongside each service, but for the API gateway use case, you want a centralized edge proxy — a single Envoy instance (or a small fleet behind a load balancer) that terminates external traffic and forwards it to internal services.

Here’s how the pieces fit together:

  1. Listeners accept inbound connections on defined ports and protocols. Each listener can have multiple filter chains, allowing you to apply different logic based on the port or SNI (Server Name Indication).
  2. Routes map incoming requests to upstream clusters. Route matching can be based on path, header, query parameter, or even runtime values.
  3. Clusters define the upstream destinations and all resilience policies: circuit breakers, retry policies, health checks, and load balancing strategies.
  4. Filters process requests as they flow through the proxy. The HTTP connection manager is the workhorse here, providing access to headers, routing, and the ext_authz filter for authorization.
  5. Clusters and listeners are discovered dynamically via the xDS protocol family — specifically, LDS (Listener Discovery Service), RDS (Route Discovery Service), CDS (Cluster Discovery Service), and EDS (Endpoint Discovery Service). This decouples configuration from the proxy binary and lets you update policies without a restart.

The separation of these concerns means you can version your route configuration independently from your cluster configuration. When you need to add a new upstream service, you update CDS and RDS without touching the listener definitions. In production, this is typically driven by a control plane like Istio’s Pilot, Envoy’s own self-hosted xDS server, or a tool like Contour.

Configuring Circuit Breakers

Circuit breakers are the single most important resilience feature in any API gateway. They prevent a overwhelmed upstream service from consuming all available connections, threads, and memory on the gateway side, which would otherwise cascade into a full system outage.

Envoy supports five distinct circuit breaker thresholds per cluster:

  • max_connections — the maximum number of concurrent connections to the upstream cluster.
  • max_pending_requests — the maximum number of requests queued waiting for an available connection.
  • max_requests — the maximum number of concurrent requests to the upstream cluster.
  • max_retries — the maximum number of retries allowed on a single request.
  • retries_per_threshold — a more granular retry budget that works alongside max_retries.

Here’s a production-grade cluster configuration with circuit breakers applied:

static_resources:
  clusters:
    - name: payment-service
      connect_timeout: 0.5s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      circuit_breakers:
        thresholds:
          - priority: DEFAULT
            max_connections: 100
            max_pending_requests: 100
            max_requests: 200
            max_retries: 3
      load_assignment:
        cluster_name: payment-service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: payment-svc
                      port_value: 8080

The critical detail here is that max_pending_requests and max_requests serve different purposes. max_requests limits the total in-flight requests, while max_pending_requests limits the queue of requests waiting for a connection to become available. Without the pending request limit, a slow upstream will accumulate a growing queue of buffered requests, each consuming memory and each likely to time out or fail — a classic resource exhaustion pattern.

In practice, I recommend setting max_requests to a value that reflects your upstream’s actual processing capacity. If your payment service can handle 200 concurrent requests at p99 latency, set the threshold slightly below that — say 180 — to leave headroom for health-check traffic and administrative requests.

Retry Policies with Backoff

Retries are where API gateways can either save your system or destroy it. A naive retry configuration will amplify load on an already-failing service, accelerating the cascade. Envoy gives you fine-grained control over which errors trigger retries, how many times they retry, and the backoff strategy between attempts.

The key configuration elements are:

  • retriable-status-codes — the HTTP status codes that should trigger a retry (typically 503, 504).
  • retriable-headers — response headers that indicate a retry should occur.
  • num-retries — the maximum number of retry attempts.
  • retry-back-off — the strategy for spacing retry attempts. Envoy supports both a fixed base interval and a jitter factor.
clusters:
  - name: user-service
    connect_timeout: 0.5s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    retry_policy:
      retry_on: reset,connect-failure,refused-stream,503,504
      num_retries: 3
      retry_back_off:
        base_interval: 0.1s
        max_interval: 1.0s
      retriable-status-codes:
        - 503
        - 504

The retry_on field is where most teams get this wrong. Including 503 without also including reset or connect-failure means you’ll retry on HTTP 503s but miss connection-level failures that don’t produce an HTTP response at all. Conversely, including x-envoy-retry-on without restricting the set of retriable status codes can cause retries on 429 (Too Many Requests) responses, which are not retryable — the client needs to back off, not the gateway.

A production pattern I recommend is to combine retry policy with a timeout on the route level. This ensures that even if the upstream is slow, the total time spent on any single request (including retries) is bounded:

routes:
  - match:
      prefix: /users
    route:
      cluster: user-service
      timeout: 2s

With timeout: 2s and num_retries: 3, each individual attempt has a 2-second budget, giving you a worst-case ceiling of approximately 6 seconds plus backoff intervals per original request.

Rate Limiting at the Gateway

Rate limiting is the last line of defense against abusive clients and the first step in protecting your upstream services from traffic spikes. Envoy supports rate limiting through the local_rate_limit filter for simple per-proxy enforcement and the ext_authz filter with an external rate-limiting service like Envoy’s own Rate Limit Service (RLS) or a custom implementation.

For most production deployments, I recommend using the local rate limit filter for coarse-grained protection (e.g., “no more than 100 requests per second per client IP”) and an external service for fine-grained, per-user or per-API-key limits.

Here’s a local rate limit configuration applied at the HTTP connection manager level:

http_filters:
  - name: envoy.filters.http.local_ratelimit
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
      stat_prefix: http_local_rate_limiter
      token_bucket:
        max_tokens: 100
        tokens_per_fill: 10
        fill_interval: 1s
      filter_enabled:
        runtime_key: local_rate_limit_enabled
        default_value:
          numerator: 100
          denominator: HUNDRED
      filter_enforced:
        runtime_key: local_rate_limit_enforced
        default_value:
          numerator: 100
          denominator: HUNDRED
      response_headers_to_add:
        - header:
            key: X-RateLimit-Limit
            value: "100"
        - header:
            key: X-RateLimit-Remaining
            value: "%DOWNSTREAM_REQUESTS_REMAINING%"

The token_bucket configuration is where the resilience strategy lives. max_tokens defines the burst capacity — how many requests can be processed in a single second if the bucket is full. tokens_per_fill and fill_interval together define the sustainable rate. Setting max_tokens equal to tokens_per_fill gives you a strict rate limit with no burst capacity. Setting max_tokens significantly higher gives you burst tolerance at the cost of potential upstream overload during spike windows.

A practical pattern for API gateways serving heterogeneous clients is to apply different rate limits based on headers. You can use Envoy’s route level rate_limits to define per-route rate limit descriptors:

routes:
  - match:
      prefix: /api/v1/payments
    route:
      cluster: payment-service
    typed_per_filter_config:
      envoy.filters.http.local_ratelimit:
        "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
        stat_prefix: payments_rate_limit
        token_bucket:
          max_tokens: 50
          tokens_per_fill: 5
          fill_interval: 1s

This allows you to apply stricter limits to expensive endpoints like payment processing while allowing higher throughput on read-only endpoints like product listings.

Health Checking and Active Failure Detection

Circuit breakers protect you from overload, but they don’t tell you whether an upstream is actually healthy. Envoy’s active health checking periodically probes upstream hosts and removes unhealthy ones from the load balancing pool. The configuration is straightforward but the tuning requires care.

clusters:
  - name: inventory-service
    connect_timeout: 0.5s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    health_checks:
      - timeout: 1s
        interval: 5s
        unhealthy_threshold: 3
        healthy_threshold: 2
        http_health_check:
          path: /health
          expected_statuses:
            start: 200
            end: 299
    load_assignment:
      cluster_name: inventory-service
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: inventory-svc
                    port_value: 8080

The unhealthy_threshold and healthy_threshold values are critical. Setting unhealthy_threshold: 1 will remove a host from the pool on the first failed probe — fast, but prone to flapping if your health check endpoint has occasional latency spikes. Setting it to 3 requires three consecutive failures, which adds roughly 15 seconds of detection time but dramatically reduces false positives.

In production, I recommend pairing active health checks with passive health checking via outlier detection. Envoy can automatically eject hosts that exhibit high error rates or latency, independent of the health check endpoint:

clusters:
  - name: inventory-service
    outlier_detection:
      consecutive_5xx_errors: 5
      interval: 10s
      base_ejection_time: 30s
      max_ejection_percent: 50

This configuration ejects any host that returns five consecutive 5xx errors, removing it from the pool for at least 30 seconds. The max_ejection_percent: 50 ensures that at least half your hosts remain available even if all of them are failing simultaneously — a safety valve against total outage.

Putting It All Together: A Complete Gateway Configuration

Here’s a consolidated example that combines listeners, routes, clusters with circuit breakers, retries, health checks, and rate limiting into a single coherent configuration:

static_resources:
  listeners:
    - name: gateway_listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                codec_type: AUTO
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend_services
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: /api/v1/payments
                          route:
                            cluster: payment-service
                            timeout: 2s
                          typed_per_filter_config:
                            envoy.filters.http.local_ratelimit:
                              "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
                              token_bucket:
                                max_tokens: 50
                                tokens_per_fill: 5
                                fill_interval: 1s
                        - match:
                            prefix: /api/v1/users
                          route:
                            cluster: user-service
                            timeout: 1s
                http_filters:
                  - name: envoy.filters.http.local_ratelimit
                  - name: envoy.filters.http.router
  clusters:
    - name: payment-service
      connect_timeout: 0.5s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      circuit_breakers:
        thresholds:
          - priority: DEFAULT
            max_connections: 100
            max_pending_requests: 100
            max_requests: 200
            max_retries: 3
      retry_policy:
        retry_on: reset,connect-failure,503,504
        num_retries: 3
        retry_back_off:
          base_interval: 0.1s
          max_interval: 1.0s
      health_checks:
        - timeout: 1s
          interval: 5s
          unhealthy_threshold: 3
          healthy_threshold: 2
          http_health_check:
            path: /health
      load_assignment:
        cluster_name: payment-service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: payment-svc
                      port_value: 8080
    - name: user-service
      connect_timeout: 0.5s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      circuit_breakers:
        thresholds:
          - priority: DEFAULT
            max_connections: 200
            max_pending_requests: 200
            max_requests: 500
            max_retries: 2
      retry_policy:
        retry_on: reset,connect-failure,503
        num_retries: 2
        retry_back_off:
          base_interval: 0.05s
          max_interval: 0.5s
      health_checks:
        - timeout: 1s
          interval: 5s
          unhealthy_threshold: 3
          healthy_threshold: 2
          http_health_check:
            path: /health
      load_assignment:
        cluster_name: user-service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: user-svc
                      port_value: 8080

Notice how the payment-service cluster has a lower max_requests (200) and stricter retry policy (3 retries, longer backoff) compared to user-service (500 requests, 2 retries, shorter backoff). This reflects the real-world observation that payment endpoints are typically more expensive and less retry-tolerant than read endpoints. The circuit breaker thresholds are calibrated to each service’s actual capacity, not set to a universal default.

Operational Considerations

Deploying Envoy as an API gateway introduces operational complexity that teams should plan for:

  • Configuration management: If you’re using static configuration (as shown above), every change requires a proxy restart. For zero-downtime updates, adopt the xDS model with a control plane. Contour, Gloo, or a custom xDS server are all viable options.
  • Observability: Envoy emits extensive stats in Prometheus format. At minimum, track upstream_rq_pending_overflow (requests rejected due to circuit breakers), cluster_upstream_rq_pending_failure_eject (hosts ejected by outlier detection), and http_local_rate_limiter_* metrics to understand when your resilience mechanisms are actively protecting your system.
  • TLS termination: For production gateways, configure TLS at the listener level using Envoy’s transport_socket configuration. Offload TLS at the gateway rather than at each upstream service to reduce certificate management complexity.
  • Graceful shutdown: Envoy supports a drain sequence that completes in-flight requests before shutting down. Always configure a drain timeout that matches your longest expected request duration to avoid dropping active transactions during deployments.

Key Takeaways

  • Circuit breakers must be tuned per-service based on actual capacity — a universal threshold will either waste resources or leave you vulnerable to cascading failures.
  • Retries without bounded timeouts create infinite retry loops; always pair num_retries with a route-level timeout.
  • Local rate limiting is effective for coarse protection, but per-user and per-API-key limits require an external service like Envoy’s Rate Limit Service.
  • Active health checks should be paired with outlier detection — health checks catch total failures, outlier detection catches partial degradation that health checks miss.
  • The xDS protocol is not optional for production — static configuration creates deployment friction that will eventually lead to configuration drift and outages.
  • Envoy’s stats are your early warning system; instrument upstream_rq_pending_overflow and cluster_upstream_rq_pending_failure_eject before you need them.

Further Reading