Canary and wave deployments - phased rollout with health checks

A canary deployment exposes a small subset of users to the new version. If health is good, expand. If regression, rollback. This is the safest pattern for production changes.

Canary pattern

1. Deploy new version to 5% of hosts
2. Send 5% of traffic to new version
3. Monitor health for 30 minutes
4. If good: deploy to 25%, 50%, 100%
5. If bad: rollback immediately

Choose canary size

The canary should be:

• Big enough to surface issues (5-10% of production).
• Small enough to limit blast radius.
• Representative of the full fleet (similar load).

For a 100-host fleet: 5 hosts canary. For a 1000-host fleet: 50 hosts canary.

Hold time

After deploying to the canary, hold for:

• 15 minutes: small changes, well-tested.
• 30-60 minutes: typical patches.
• 4-24 hours: major upgrades.

The hold time should be longer than the typical intermittent failure window (cron runs, log rotation, etc.).

Health checks

Monitor the canary for:

• Error rate (HTTP 5xx, application errors).
• Latency (p99, p95, p50).
• Resource usage (CPU, memory, network).
• Synthetic monitoring (transactions per second).
• Custom business metrics (logins, conversions).

If any metric degrades significantly, rollback.

Rollback

Rollback should be fast and automatic when possible:

# Ansible rollback
ansible-playbook -i prod rollback.yml --limit canary_group

# Container redeploy
kubectl rollout undo deployment/myapp

The canary exists to give you a fast, safe rollback path.

Blue-green vs canary

• Blue-green: two identical environments; switch traffic all at once.
• Canary: gradual rollout; traffic shifts over time.

Canary is safer for stateful services. Blue-green is faster for stateless services.

Knowledge check