Choosing a strategy — the decision matrix and the production discipline

The five patterns in this part are not five options to rank. They are five answers to different questions. The decision matrix in this lesson maps the workload profile and the change type to the right pattern, and the production discipline section identifies the metrics that signal a pattern mismatch. Choosing a strategy is not a one-time decision; it is an ongoing calibration against the workload’s evolution.

The decision matrix

flowchart TD
    A["Workload + change"] --> B{"Stateless?"}
    B -- "yes" --> C{"Backward-compatible?"}
    B -- "no" --> D{"Single-instance?"}
    C -- "yes" --> E{"High-risk?"}
    C -- "no" --> F["Recreate or blue-green with data migration"]
    D -- "yes" --> G["Recreate with maintenance window"]
    D -- "no" --> H["Blue-green with per-env database"]
    E -- "yes" --> I["Canary"]
    E -- "no" --> J["Rolling update"]

The matrix is a starting point, not a flowchart to follow mechanically. The dimensions that matter:

• Stateless vs stateful. Stateless workloads default to rolling update; stateful workloads with a single replica default to recreate with a maintenance window.
• Backward-compatible vs breaking. Backward-compatible changes ride rolling update or canary; breaking changes require recreate or blue-green with a data migration.
• High-risk vs routine. High-risk releases benefit from canary or blue-green; routine releases do not need the overhead.
• Data compatibility. The schema and the database layer constrain which patterns are safe regardless of traffic shape.

Patterns by workload profile

• Stateless REST API, additive schema, routine release. Rolling update. The default, no extra infrastructure.
• Stateless REST API, additive schema, high-risk release. Canary with metric gates. The metric gates define what high-risk means in measurable terms.
• Stateless REST API, breaking schema change. Expand-and- contract with sequential rolling updates, or blue-green with per-environment databases.
• Stateful single-instance service. Recreate with a planned maintenance window. The downtime is real and must be communicated.
• Background worker with queue consumption. Rolling update with a preStop drain. Old pods finish in-flight messages before termination.
• High-traffic user-facing service, regulated environment. Canary with strong metric gates, or blue-green with per- environment databases for instant rollback.

Metrics that signal a pattern mismatch

Three metrics indicate that the deployed pattern does not fit the workload or the change:

• Error rate during rollout. A spike in 5xx rate during a rolling update suggests the new version is not backward- compatible or the readiness signal is wrong. A spike during a canary suggests the metric gate is missing the relevant metric.
• Rollback frequency. A high rollback rate means the pattern is allowing bad releases to reach too many users before they are caught. The fix is a stricter pattern (canary instead of rolling) or stricter gates.
• Change failure rate. The DORA metric of percentage of changes causing a production failure. A high rate combined with a high MTTR signals that the deployment pipeline is not catching failures early enough.

flowchart LR
    A["Pattern deployed"] --> B{"Error during rollout?"}
    B -- "yes" --> C["Gate or readiness gap"]
    B -- "no" --> D{"Rollback frequency high?"}
    D -- "yes" --> E["Pattern too permissive; tighten to canary"]
    D -- "no" --> F{"Change failure rate high?"}
    F -- "yes" --> G["Pattern too permissive or gates too lax"]
    F -- "no" --> H["Pattern matches workload"]

The metrics are leading indicators of pattern fitness; a pattern that produces no error spikes and no rollbacks is a pattern that may be too lax (the gate is not catching what it should) or a pattern that fits perfectly. Distinguishing the two requires looking at the gates’ content, not just their outcome.

When to retire a pattern

A pattern that was correct six months ago may be wrong today. The triggers for retirement:

• Rolling update on a data-heavy service. As the service accumulates schema changes, the rolling update window becomes a data-corruption window. Move to expand-and- contract or blue-green.
• Blue-green on a cost-constrained workload. As the workload grows, the steady-state cost of running two environments becomes prohibitive. Move to canary or rolling update.
• Canary on a low-traffic service. As traffic declines, the canary’s metric gates become statistically meaningless at small traffic fractions. Move to rolling update with manual smoke tests.
• Recreate on a user-facing service. As the service matures, the recreate window becomes a customer-visible outage. Move to rolling update or blue-green.

The production discipline

Six rules govern the choice and the evolution of a deployment pattern:

1. Choose the pattern after the data migration is designed. The schema and the database layer are the constraint; traffic shape is secondary.
2. Default to rolling update for stateless services. It is the lowest-overhead pattern that does not produce downtime.
3. Use canary for high-risk releases. The metric gates make the risk measurable; the small initial blast radius makes the consequence bounded.
4. Use blue-green when instant rollback matters. Payments and regulated environments benefit from the atomic switch.
5. Use recreate only when the alternatives are unsafe. The downtime is real and must be measured.
6. Revisit the pattern choice quarterly. Metrics that signal a mismatch (error during rollout, rollback frequency, change failure rate) are the input to the revisit.

Cross-course references

• Kubernetes for Production Sysadmins - Part XIV (Workloads) covers the patterns and their native Kubernetes implementations.
• This course, Part L (DORA metrics) covers the change-failure-rate metric in the broader delivery context.
• This course, Parts LVIII-01 through LVIII-05 cover each pattern in depth.

Quiz