Pre-prod gates and canary fleets — paying down risk before production

A pre-prod gate is the rehearsal that pays down the risk before the change meets production. Canary, blue-green, and progressive delivery are the patterns that make the gate meaningful. A cluster without a pre-prod gate is a cluster that is rehearsing in production.

What a pre-prod gate is

A pre-prod gate is a mirror of production that runs the change before the change meets production. The mirror is not identical to production; it is representative of production. The workload’s shape, the cluster’s version, the CNI’s version, the CSI’s version, the data’s shape — all of these are matched to the production cluster’s current state.

The gate’s job is to catch the failure mode that the plan did not anticipate. The rehearsal is the moment when the “considered improbable” failure becomes “observed in staging.”

flowchart LR
    A[Author writes plan] --> B[Staging rehearsal]
    B --> C{Staging OK?}
    C -->|No| D[Revise plan]
    D --> B
    C -->|Yes| E[Canary in production]
    E --> F{Canary OK?}
    F -->|No| G[Rollback canary]
    F -->|Yes| H[Progressive rollout]
    H --> I[Full rollout]
    I --> J[Validated]

The gate is graduated. A change that fails in staging never reaches production. A change that fails in canary is rolled back at the canary boundary, not at the production boundary.

Canary

A canary is a small fraction of production traffic that runs the new version. The fraction is small (1–5%) so the impact is bounded; the canary is observed (latency, errors, business KPIs) so the rollback is fast.

flowchart TD
    A[100% traffic] --> B[95% stable + 5% canary]
    B --> C{Canary healthy?}
    C -->|No| D[Rollback canary]
    C -->|Yes| E[50% stable + 50% canary]
    E --> F{Canary healthy?}
    F -->|No| D
    F -->|Yes| G[Canary promoted to stable]

The canary is the gate that catches the workload-specific failure: the new image that boots in staging but fails at production’s traffic, the new HPA that scales differently at production’s volume.

In Kubernetes, the canary is typically implemented as a second Deployment that the Service routes to. The Service’s spec.selector matches both Deployments, and the EndpointSlice controller fills the slice with the canary’s pods. The canary is observed; the slice is edited (or the canary is scaled to zero) to roll back.

Blue-green

Blue-green is a topology where two environments — blue (current) and green (new) — run side by side. The traffic is switched from blue to green at the moment of promotion. The switch is atomic at the load balancer (or Service/Ingress); the rollback is the switch back.

stateDiagram-v2
    [*] --> BlueServing
    BlueServing --> GreenDeploying: deploy green
    GreenDeploying --> GreenValidated: tests pass
    GreenValidated --> GreenServing: switch traffic
    GreenServing --> BlueServing: rollback
    GreenServing --> [*]

The blue-green tradeoff is cost: two environments are running simultaneously, so the cluster’s capacity is doubled. The benefit is rollback speed: the switch is a single command, and the rollback is a switch back.

Progressive delivery

Progressive delivery is the gradual, automated rollout of a new version. The rollout is parameterised by a controller (Argo Rollouts, Flagger, Knative) that observes the new version’s metrics and steps the rollout forward or backward based on the metrics.

flowchart LR
    A[Step 1: 1%] --> B[Step 2: 10%]
    B --> C[Step 3: 25%]
    C --> D[Step 4: 50%]
    D --> E[Step 5: 100%]
    A --> F{Metrics OK?}
    B --> F
    C --> F
    D --> F
    F -->|No| G[Rollback]
    F -->|Yes| H[Next step]

The progressive delivery is the gate that catches the failure that scales with traffic. The rollout is the value of the promotion: each step is a new evidence point.

Failure modes of staging

The worst production failure is a staging cluster that is not representative. Common failure modes:

• Stale staging. A staging cluster that has not been updated to match production’s version. The new feature passes in staging because staging is on the old version.
• Synthetic staging. A staging cluster that runs synthetic workloads. The new image passes in staging because the real workload is not there.
• Under-scale staging. A staging cluster that is orders of magnitude smaller than production. The new HPA passes in staging because the scale is small.

The remediation is to represent production in staging. The data shape, the workload’s traffic, the cluster’s version, the CNI’s version, the CSI’s version — all matched.

Production discipline

A pre-prod gate is the cluster’s cheapest insurance. The gate is canary, blue-green, or progressive delivery; the gate is graduated; the gate is observed. The discipline is to never ship to production without a rehearsal that matches the production’s shape.

• Rehearse in staging. The staging must mirror production’s shape, not just production’s version.
• Canary in production. A small fraction of production traffic runs the new version, observed, with a fast rollback.
• Progressive rollout. Each step is a new evidence point.
• Rollback at the gate. The rollback is the gate that stops the failure from propagating.

Quiz