Canary and progressive delivery — traffic splitting, metric-based promotion

The canary deploy is the pattern for releases where the blast radius must be small. A canary routes a fraction of traffic to the new version, measures outcome against an objective, and either promotes the new version to 100% or rolls it back. The discipline is in the metric gates: the controllers (Argo Rollouts, Flagger) are machinery; the metrics that decide promotion are the policy.

How a canary works

A canary deploy proceeds in steps. Each step is a traffic weight on the new version, an evaluation window, and a metric gate. If the metric gate passes, the rollout advances to the next step. If the gate fails, the rollout halts and the new version is rolled back.

flowchart LR
    A["Canary 5%"] --> B{"Metrics ok?"}
    B -- "yes" --> C["Canary 25%"]
    C --> D{"Metrics ok?"}
    D -- "yes" --> E["Canary 50%"]
    E --> F{"Metrics ok?"}
    F -- "yes" --> G["Stable 100%"]
    B -- "no" --> Z["Abort"]
    D -- "no" --> Z
    F -- "no" --> Z

A typical progression is 5%, 25%, 50%, 100%, with each step holding for a fixed evaluation window (often five to fifteen minutes). The total rollout time is the sum of the windows; the canary’s value is that the largest fraction of users ever exposed to a bad release is the step that failed, not 100%.

Traffic routing on Kubernetes

Plain Kubernetes Deployments cannot shift traffic by percentage between two ReplicaSets. The traffic split requires either a service mesh (Istio, Linkerd) or an ingress controller that supports weighted routing (NGINX, Traefik, Contour). Argo Rollouts and Flagger abstract the underlying router so the Rollout manifest reads as a high-level policy.

kubectl argo rollouts get rollout $NAME --watch

This command shows the current step, the traffic weight, and the metric status. It is the canary equivalent of kubectl rollout status.

sequenceDiagram
    participant U as User
    participant R as Router
    participant S as Stable
    participant C as Canary
    U->>R: HTTP request
    R->>S: 95%
    R->>C: 5%
    C-->>R: response
    R-->>U: response

The router’s job is to maintain the weight. The canary controller’s job is to advance or abort. The metric provider’s job is to decide whether the gate passes.

Metric-based promotion

The metric gate is the heart of the canary. Argo Rollouts calls it an AnalysisTemplate; Flagger calls it a metric template. Both take a query (Prometheus, Datadog, CloudWatch, custom HTTP) and a success condition.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: web-success-rate
spec:
  metrics:
    - name: success-rate
      interval: 60s
      successCondition: result[0] >= 0.99
      failureCondition: result[0] < 0.95
      provider:
        prometheus:
          query: |
            sum(rate(http_requests_total{status!~"5.."}[5m]))
              /
            sum(rate(http_requests_total[5m]))

The template defines what success means. The success condition is the bar; the failure condition is the floor. A metric in between is inconclusive and the rollout waits.

When canary fits

Canary is the right pattern when:

• The release is high-risk and the change is hard to roll back instantly. A bad release that requires a database migration to revert benefits from the canary’s small blast radius.
• The workload serves enough traffic to make metrics meaningful at 1-5%. A service that handles 10 requests per second cannot distinguish a 99% success rate from a 95% success rate at 5% traffic.
• The organisation has observability discipline. A canary without metrics is a rolling update with extra steps.

Canary is not the right pattern when traffic is too low for metrics to be statistically meaningful, or when the rollback cost is low enough that a 100% rolling update with a fast rollout undo is acceptable.

Production discipline

1. Choose canary when blast radius matters. A low-risk change does not need a 30-minute canary; a high-risk change does.
2. Define the metric gates explicitly. A metric not in the template is a metric not evaluated.
3. Guard against empty-data promotion. Require a count condition alongside rate conditions.
4. Observe canaries like production. The metric provider used in the gate must be the metric provider used by SRE.

Cross-course references

• Kubernetes for Production Sysadmins - Part XXIV (Service Mesh) covers the traffic-routing layer canaries depend on.
• This course, Part LVIII-02 (Rolling update) is the pattern canary replaces when blast radius matters.
• This course, Part LIII-05 (SBOM generation in CI) is a complementary safety mechanism - a canary catches behavioural regressions; an SBOM catches supply-chain regressions.

Quiz