Progressive delivery — controllers that automate canary and blue/green

The previous lessons covered what you can do with two Deployments and a Service. Production Kubernetes estates running hundreds of services need more: metric-driven promotion, automatic rollback, traffic splitting at the Ingress or Gateway layer, and analysis templates that encode promotion policy. Progressive-delivery controllers — Argo Rollouts and Flagger are the most common — provide this. This lesson covers the gap they fill, how they work, and what they cannot do.

The gap

A native Kubernetes Deployment can roll forward, but it cannot:

• Route 5% of traffic to a new version while leaving 95% on the old.
• Wait 5 minutes and check Prometheus for the error rate before scaling up.
• Roll back automatically when the error rate exceeds 1.5x baseline.
• Maintain a stable Ingress endpoint while swapping the backend Service.

These are the four primitives of progressive delivery. Each requires a controller that watches a CRD (Rollout or Canary), orchestrates the Deployments, splits traffic, and queries metrics.

flowchart TB
    subgraph "Native Kubernetes"
        D[Deployment]
        S[Service]
        I[Ingress]
    end
    subgraph "Progressive delivery"
        R["Rollout / Canary CRD"]
        C[Controller]
        M["Prometheus<br/>metrics provider"]
        T["Traffic provider<br/>Ingress / Gateway / Service Mesh"]
    end
    R --> C
    C --> D
    C --> M
    C --> T
    T --> I

Argo Rollouts in shape

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web
spec:
  replicas: 6
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: {duration: 5m}
      - setWeight: 25
      - pause: {duration: 5m}
      - analysis:
          templates:
          - templateName: success-rate
      - setWeight: 100
      canaryService: web-canary
      stableService: web-stable
      trafficRouting:
        nginx:
          stableIngress: web
      analysis:
        successfulRunHistoryLimit: 5
        failedRunHistoryLimit: 5
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: web:1.28.0

The Rollout replaces the Deployment. It uses two Services (web-stable for the bulk, web-canary for the canary slice) and one Ingress that the controller updates to split traffic according to the current setWeight. After each step the controller pauses; if an analysisTemplate is attached, it queries Prometheus for the configured metrics and either proceeds (success) or aborts (failure).

sequenceDiagram
    participant OP as Operator
    participant AR as Argo Rollouts
    participant D as Deployments
    participant S as Services
    participant P as Prometheus
    OP->>AR: apply new image
    AR->>D: scale canary to 5%
    AR->>S: setWeight=5
    Note over AR: pause 5m
    AR->>P: query success-rate
    P-->>AR: 0.02% (pass)
    AR->>D: scale canary to 25%
    AR->>S: setWeight=25
    Note over AR: pause 5m
    AR->>P: query success-rate
    P-->>AR: 0.05% (pass)
    AR->>D: scale canary to 100%
    AR->>S: retire stable

AnalysisTemplate and metric gates

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
  - name: success-rate
    interval: 60s
    count: 5
    successCondition: result[0] >= 0.995
    failureLimit: 1
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(http_requests_total{job="web",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="web"}[5m]))

The template runs the Prometheus query 5 times, 60 seconds apart. successCondition parses the result and proceeds if the success rate is ≥ 99.5%. failureLimit: 1 aborts on the first failed check.

The power: the promotion decision is encoded in infrastructure, not in operator judgement. Every Rollout goes through the same gate. The hazard: a typo in the Prometheus query blocks every rollout until it is fixed.

Flagger in shape

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: web
spec:
  provider: nginx
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  progressDeadlineSeconds: 60
  metrics:
  - name: request-success-rate
    thresholdRange:
      min: 99
    interval: 60s
  analysis:
    interval: 60s
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    steps: 5

Flagger sits next to the existing Deployment; the operator keeps the Deployment and adds a Canary CRD. Flagger creates the canary Deployment, runs the analysis, and updates the Ingress.

The two systems differ in scope. Argo Rollouts is a CRD replacement for Deployment with full rollout features; Flagger is an add-on that orchestrates a side-by-side Deployment.

Traffic providers

Progressive-delivery controllers do not own the dataplane. They route traffic through a configured traffic provider:

Each provider has its own quirks. NGINX Ingress rounds weights and supports a stable/canary pair of Ingresses. Istio requires a VirtualService and DestinationRule. The controller’s job is to abstract those differences behind a single CRD field.

What progressive delivery cannot do

A controller that automates canary promotion does not eliminate the operator’s responsibility. The remaining gaps:

• Schema migrations. The controller can roll forward and back at the application layer; the database schema is still the operator’s.
• External dependency compatibility. The canary cannot detect that it is breaking a downstream service without that service’s metrics.
• Metric falsification. A bad metric query (typo, wrong label) can make every rollout succeed or every rollout fail. The operator owns the queries.
• Cost. Each Rollout at 5% → 25% → 50% → 100% doubles the resource footprint at peak. Cluster autoscaling may be required.

Inspecting a Rollout

kubectl argo rollouts get rollout web -n prod
# Name:            web
# Namespace:       prod
# Status:          ◌ Progressing
# Strategy:        Canary
#  Step:           2/6
#  SetWeight:      25
#  ActualWeight:   25

kubectl argo rollouts promote web -n prod
# Skip the pause and promote to the next step.

kubectl argo rollouts abort web -n prod
# Roll back to the stable version.

Quiz

Production discipline

• Promote only after success. A Rollout that auto-promotes without an analysis has no safety advantage over a plain Deployment rollout.
• Keep history. successfulRunHistoryLimit and failedRunHistoryLimit are the audit trail. Five runs is the minimum; longer histories surface metric drift.
• Alert on Rollout state. A Rollout stuck in Progressing past progressDeadlineSeconds is the equivalent of a Deployment’s ProgressDeadlineExceeded — alert on it.
• Test the metric gate. Apply a known-bad release to a staging cluster; verify the analysis aborts the rollout. Then verify a known-good release passes.
• Document the rollback path. kubectl argo rollouts abort is the obvious answer; the question is whether the downstream state (database, queues, caches) tolerates the rollback. The CRD does not know.

Progressive delivery is a force multiplier. It encodes the promotion discipline in infrastructure and removes the operator from the hot path. But the discipline encoded is the discipline the team wrote; the controller is a faithful executor. A wrong query, a wrong threshold, or a missing traffic provider makes the system worse than no rollout controller at all.