TL;DR — Argo Rollouts replaces Deployments with a controller that understands progressive delivery: blue/green, canary, A/B testing, and experiment-driven rollouts. The real leverage comes from pairing it with a traffic router (NGINX, Istio, ALB) and an AnalysisTemplate that gates promotion on real metrics like error rate and p99 latency. The hard part is not the YAML — it’s wiring up metric providers, understanding pauses/aborts, and avoiding traffic-shadow traps.

If you’ve ever shipped a bad release to Kubernetes and watched the pager light up, you already know why progressive delivery exists. The native Deployment controller is a glorified “swap the pods” machine: it terminates old ReplicaSets and spins up new ones with no concept of traffic weighting, health gating, or automatic rollback. That works fine until you have a monolith with 200 pods, a stateful service that can’t tolerate duplicate traffic, or a fintech app where a 0.5% error rate is a compliance event.

Argo Rollouts is a Kubernetes controller and CRD set that fixes this. It ships in the Argo Project family (alongside Argo CD and Argo Workflows) and is one of the most widely adopted progressive delivery implementations in the CNCF ecosystem. It powers deployment pipelines at Intuit, Reddit, LinkedIn, and dozens of other large shops.

This post walks through how to implement it well: the strategies, the traffic-routing plumbing, the analysis loop that makes rollouts self-correcting, and the patterns that actually hold up in production.

Why Native Deployments Aren’t Enough

A Deployment running RollingUpdate does three things: it creates a new ReplicaSet, scales it up, scales the old one down, and lets Kubernetes’ readiness probes gate traffic. That’s it. There is no traffic weighting, no concept of “send 5% of users to the new version and watch the metrics,” and no automatic rollback when things go sideways. You can hack this with two Deployments and a service that flips, but then you own the orchestration, the pod startup races, and the database migrations.

Progressive delivery is the umbrella term for the practice of releasing changes to a subset of users and observing real signals before a full rollout. The two most common strategies are:

  • Canary — Route a small percentage of traffic to the new version, watch metrics, then expand.
  • Blue/Green — Stand up the new version side-by-side, switch traffic over instantaneously, but keep the old version alive for instant rollback.

Argo Rollouts makes both first-class, and crucially, ties them to an analysis loop that can promote, pause, or abort based on metrics from Prometheus, Datadog, CloudWatch, or anything you can wrap in a pluggable provider.

Core Concepts: The Rollout Resource

The Rollout CRD replaces the Deployment you already have. At its core, it looks similar but adds three new sections: strategy, trafficRouting, and an analysis step block.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
spec:
  replicas: 6
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: checkout-api
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: { duration: 2m }
      - setWeight: 25
      - pause: { duration: 3m }
      - analysis:
          templates:
          - templateName: error-rate-check
      - setWeight: 100
      trafficRouting:
        nginx:
          stableIngress: checkout-api
        canaryIngress: checkout-api-canary
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
      - name: checkout-api
        image: registry.example.com/checkout-api:v1.4.0
        ports:
        - containerPort: 8080

That strategy.canary.steps block is the heart of it. Each step either sets a traffic weight, pauses for a duration, or runs an analysis. The trafficRouting block tells the controller where to push those weights — to an Ingress, an Istio VirtualService, an AWS ALB, or something else. Without trafficRouting, the rollout is essentially a “scale-up, scale-down” controller with a fancier UI.

Blue/Green: When You Need a Clean Switch

Blue/green is the right tool when you want zero in-flight mixing between versions. Common scenarios:

  • Schema-breaking database migrations that the old code can’t handle.
  • Long-lived connections (WebSockets, gRPC streams) that you don’t want to share.
  • Compliance-driven switches where you need an audit trail of “version X was live from T0 to T1.”

Here’s a blue/green rollout for a stateful service:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments-ledger
spec:
  replicas: 4
  selector:
    matchLabels:
      app: payments-ledger
  strategy:
    blueGreen:
      activeService: payments-ledger-active
      previewService: payments-ledger-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 600
      abortScaleDownDelaySeconds: 30
  template:
    metadata:
      labels:
        app: payments-ledger
    spec:
      containers:
      - name: payments-ledger
        image: registry.example.com/payments-ledger:v3.0.0

A few details worth noting:

  • autoPromotionEnabled: false is the production-safe default. It means after the new version is healthy, an operator has to manually promote. This is what you want for a payments service.
  • scaleDownDelaySeconds: 600 keeps the previous (blue) version alive for 10 minutes after switchover. If you spot a regression 7 minutes in, rollback is one command: kubectl argo rollouts abort payments-ledger.
  • previewService is a separate Kubernetes Service that points at the new ReplicaSet. You can hit it directly with curl preview.payments.svc.cluster.local to smoke-test before flipping traffic.

The gotcha with blue/green is resource doubling. If your replicas: 4 rollout runs, you temporarily need 8 pods. In a cost-sensitive environment, use abortScaleDownDelaySeconds aggressively or accept that blue/green isn’t the right tool.

Canary: The Default Choice for Most Teams

Canary is where Argo Rollouts earns its keep. Instead of a binary switch, you increment traffic to the new version in steps and run an analysis at each one. The previous example with 5% → 25% → 100% is a typical structure, but the real power is in the analysis step.

An AnalysisTemplate is a separate CRD that defines a queryable check:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
  namespace: checkout
spec:
  metrics:
  - name: error-rate
    interval: 30s
    count: 5
    successCondition: result[0] < 0.01
    failureCondition: result[0] > 0.05
    inconclusiveCondition: result[0] >= 0.01 && result[0] <= 0.05
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc:9090
        query: |
          sum(rate(http_requests_total{
            job="checkout-api",
            status=~"5.."
          }[2m]))
          /
          sum(rate(http_requests_total{job="checkout-api"}[2m]))
  - name: p99-latency
    interval: 60s
    count: 3
    successCondition: result[0] < 0.8
    failureLimit: 1
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc:9090
        query: |
          histogram_quantile(0.99,
            sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout-api"}[3m]))
          )

Read that carefully — it matters. The count: 5 means “collect 5 samples at 30s intervals.” The successCondition says “if the error rate is below 1%, mark this analysis successful.” The failureCondition says “if it ever goes above 5%, abort the rollout.” The inconclusiveCondition lets you distinguish “we don’t know yet” from “we know it’s bad,” which prevents premature promotion on low-traffic services.

Argo Rollouts supports an extensive list of providers: Prometheus, Datadog, New Relic, CloudWatch, Stackdriver, Kayenta (the Spinnaker-style canary analysis engine), and a generic web provider for anything with an HTTP API.

Traffic Routing: Where the Magic Happens

The trafficRouting block is what makes the rollout actually move traffic. Each backend has different semantics:

NGINX Ingress Controller — Most common starting point. You create a canary Ingress with the same host but additional annotations. Argo Rollouts patches the canary weight annotation at each step. The downside is that NGINX only supports 100 fine-grained weight values, not floating point, so 1% increments are the practical floor.

Istio — The heavyweight option. You define a VirtualService and Argo Rollouts patches the weight field on each subset. You get full 0–100 granularity, header-based routing for A/B tests, and the ability to split by user identity. The cost is operational: you now need Istio’s control plane, sidecars (or ambient mode), and the headache of debugging mismatched destinations.

AWS Load Balancer Controller — The right pick for EKS clusters using ALBs. TargetGroup weights are updated via the AWS API. Watch out for the propagation latency — weight changes can take 30–60 seconds to fully converge, which affects how long your pause durations need to be.

Traefik, SMI, Ambassador, Kong, Gloo — All supported. The plugin pattern is open, so adding a new traffic manager is a few hundred lines of Go.

A practical rule of thumb: if you’re already on Istio for service mesh, use Istio routing. If you’re not and don’t plan to be, NGINX Ingress with a canary Ingress is the simplest path. Don’t introduce a service mesh just to get fancy canary weights.

Patterns in Production

After watching several teams adopt Argo Rollouts, a handful of patterns repeat. These are the ones that survive contact with reality.

Tie promotion to a real SLO, not just error rate. A pure “error rate below 1%” check is dangerous on low-traffic services. If your canary is getting 2 RPS, one error in a 30-second window looks like a 16% error rate and the rollout aborts. Add a min-value or use Kayenta for proper statistical confidence. Or gate on absolute request volume: “only analyze if we’ve seen more than 1000 requests in the window.”

Use dryRun on the controller side and dry-run: true on the analysis during initial rollout. The first time you ship a new AnalysisTemplate, the failure mode is usually “the query is wrong” or “the metric name changed.” Run the template against a known-bad canary first to confirm the failure path works before relying on it for a real release.

Pair Argo Rollouts with Argo CD for GitOps. Define the Rollout in Git, let Argo CD sync it, and the controller does the rest. The combination is well-documented and the Argo CD docs cover it explicitly. The temptation to have CI push directly to Kubernetes is strong — resist it. GitOps gives you audit, rollback, and a single source of truth that survives the on-call rotation.

Migrations are not rollouts. This is the most common misconception. If you’re moving from MySQL to Postgres, or from a monolith to 12 microservices, you don’t need canary. You need a migration plan, dual-writes, feature flags, and a much longer timeline. Argo Rollouts is for code changes within a stable architecture.

Handle the SLO feedback loop explicitly. When an AnalysisTemplate fails and the rollout aborts, the new ReplicaSet is scaled to zero. But the old version didn’t change — your service-level error rate may still be elevated because the bug is in shared infrastructure, not the diff you just deployed. Have a runbook for “rollout aborted, do I need to roll back the prior release too?”

Common Gotchas

A few things that bite people repeatedly:

  1. The canary Ingress doesn’t get a TLS cert by default. When you create checkout-api-canary alongside checkout-api, your cert manager (cert-manager, external-dns) needs to know to issue a cert for it. For NGINX with the canary annotation pattern, you can usually share the host and the cert covers both.

  2. pause: {} without a duration waits forever. This is the manual-promotion pattern. It’s useful, but if a junior engineer checks in a rollout with pause: {} and walks away, you’ve got a stuck deploy. Make this an explicit decision in your rollout templates.

  3. HPA and canary conflict. If you have a HorizontalPodAutoscaler on the Rollout, the HPA might scale down your canary pods mid-analysis. The fix is to set HPA behavior to ignore canary ReplicaSets, or to pin replicas during analysis. The Argo Rollouts HPA docs cover this.

  4. Webhook notifications are easy to forget. The notifications field can fire Slack, PagerDuty, or OpsGenie events on promotion, abort, and step transition. Set it up. A canary that auto-aborts in the middle of the night with no alert is a canary that everyone will ignore within a month.

  5. Rollback is a separate concept from abort. kubectl argo rollouts abort stops the rollout but leaves the new version deployed if it was already at 100%. kubectl argo rollouts undo rolls back to a specific revision. Know the difference before you need to.

Key Takeaways

  • Argo Rollouts replaces Deployments with a controller that natively understands blue/green, canary, and experiment-driven rollouts.
  • The strategy block defines the steps; the trafficRouting block defines where those steps push traffic. You need both.
  • AnalysisTemplate is the real power — it gates promotion on Prometheus, Datadog, or any HTTP-based metric provider.
  • Use blue/green for stateful or compliance-sensitive services; canary for everything else.
  • Pair with Argo CD for GitOps, wire up notifications, and write runbooks for the abort path before you need it.
  • Watch out for traffic-routing provider quirks (NGINX weight granularity, ALB propagation latency), HPA conflicts, and the difference between abort and undo.

Further Reading