Canary — small fraction first, metric-driven promotion

Canary is the pattern in which a small percentage of production traffic is sent to the new version while the majority continues to the old. Promotion is gated on observed metrics — error rate, latency, saturation — and rollback is the natural consequence when the metrics fail. This lesson covers the Kubernetes primitives and the limits of what Deployments alone can express.

The pattern

Two Deployments, one Service, traffic split by replica count or by an explicit traffic-shifting mechanism.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-stable
  labels:
    app: web
    track: stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: web
      track: stable
  template:
    metadata:
      labels:
        app: web
        track: stable
    spec:
      containers:
      - name: web
        image: web:1.27.2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-canary
  labels:
    app: web
    track: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
      track: canary
  template:
    metadata:
      labels:
        app: web
        track: canary
    spec:
      containers:
      - name: web
        image: web:1.28.0
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web       # matches both stable and canary
  ports:
  - port: 80
    targetPort: 8080

The Service has no track selector, so it routes to both Deployments’ Pods. With 9 stable + 1 canary Pods, kube-proxy load-balances roughly 90% / 10% across them.

flowchart LR
    S["Service: web<br/>selector: app=web"]
    S -->|random LB| P1[stable Pod 1]
    S --> P2[stable Pod 2]
    S --> P3[stable Pod 9]
    S --> P4["canary Pod 1<br/>~10%"]

Canary-by-replica-count

The simplest and most common. Set the canary Deployment’s replicas to ceil(stable_replicas * canary_fraction). With 10% canary:

• 9 stable / 1 canary → ~10% canary traffic
• 90 stable / 10 canary → ~10% canary traffic

This is approximate. kube-proxy (and CNI service meshes) distribute load per-Endpoint; the actual fraction can drift based on connection count, Pod readiness, and endpoint churn. For most production canaries it is “close enough.”

Canary-by-traffic-weight

Ingress controllers and service meshes expose an explicit traffic weight. With NGINX Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  rules:
  - host: web.prod.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-canary
            port:
              number: 80

A second Ingress with canary-weight: 0 (or absent) handles the stable traffic. NGINX then routes 10% of HTTP requests to the canary backend. This is precise at the request level but tied to the Ingress implementation.

Promotion criteria

The canary pattern only works if promotion is automated or disciplined. The minimum gates:

flowchart TB
    A["Deploy canary<br/>1% / 5% / 10%"] --> B[Observe N minutes]
    B --> C{Errors &lt; baseline?}
    C -->|yes| D{P99 latency<br/>&lt; baseline?}
    D -->|yes| E{Promote to 50%}
    E --> F[Observe again]
    F --> G{Healthy?}
    G -->|yes| H["Promote to 100%<br/>Retire stable"]
    G -->|no| I["Roll back<br/>scale canary to 0"]
    C -->|no| I
    D -->|no| I

These are the inputs to the human (or the controller) deciding whether to promote. A human-driven canary needs a dashboard that surfaces them; a controller-driven canary needs the same metrics in a Prometheus query.

Canary and Service selectors

The Service in a canary setup must match both versions’ Pod labels. The natural pattern: the selector is app: web without the track discriminator. Both Deployments’ Pod templates carry app: web (and track: stable or track: canary); the Service ignores track and matches both.

A common mistake: the Service’s selector is app: web, track: stable (forgot to remove the track label). Result: canary Pods receive zero traffic, canary metrics are silent, the operator concludes the canary is healthy, and promotes.

Differences from blue/green

Blue/green is the right pattern when the new release is a discrete event. Canary is the right pattern when you want real production traffic to validate the new release before full promotion.

Real-world controllers

Plain Kubernetes cannot express canary promotion gates natively. Production canary systems are typically built on:

• Argo Rollouts: a CRD-based controller with built-in metrics queries, automatic promotion/rollback, and traffic shaping through Ingress, Istio, or NGINX.
• Flagger: a similar system focused on simplicity and Prometheus integration.
• Knative Serving: traffic-splitting between revisions.

For a hand-rolled canary without a controller, the operator follows a runbook: deploy canary Deployment, watch metrics, scale replicas up or down based on observations, retire canary after promotion.

Quiz

Production discipline

• Pick the canary fraction deliberately. 1% for high-risk releases, 10% for routine bumps. The duration of the canary phase should be long enough to capture at least one business cycle of traffic (a peak hour, a backup window).
• Surface canary-specific metrics. The dashboard must distinguish canary from stable. A separate Grafana row or a Prometheus label (canary="true") makes the comparison trivial.
• Roll back on the first signal, not the third. A controller that waits for three failed checks before rollback has already impacted 30% of the user population. Promote fast, abort fast.
• Track every canary outcome. Successful or not, the canary’s metrics and the operator’s notes are post-mortem evidence. Auto-rollback should not delete the record.
• Beware shared dependencies. If canary and stable share a database or a downstream API, a slow downstream will impact both and the canary cannot distinguish its own regression from the shared one. Run canaries on isolated dependencies when risk warrants.

Canary is the workhorse of production rollouts in mature Kubernetes estates. It is not free, and it is not safe without metrics. The operator who can read the metrics and make the promotion decision is the one whose on-call is quiet.