The test strategy decision — choosing depth against the blast radius

Every change that reaches the change-gate pipeline has a different blast radius. A typo fix in a non-production tag is a routine change; a new IAM policy that grants cross-account access is a high-blast-radius change. The testing pyramid exists to scale the cost of verification to the cost of the change: a routine change should pass the cheap layers and skip the expensive ones; a high-blast-radius change should pass every layer the team can afford. The strategy decision is the recurring operational call that maps the change to the layers.

The cost-vs-blast-radius matrix:

The matrix is the policy. Every row is a class of change; every column is a pyramid layer. A “yes” in the cell means the layer runs for that class of change. A “no” means the layer is skipped. A “scheduled” means the layer runs on a cadence rather than on the change itself.

The matrix is not static. A change that is routine today becomes sensitive when the team’s threat model evolves; a sensitive change becomes routine when the cheap layers have proven they catch the mistake class. The matrix is a living document that the team revises as it learns.

Routine changes — the cheap layers are enough

A routine change is a change whose blast radius is bounded and whose failure mode is recoverable. Examples: a tag change on a non-production resource, a label change on a Kubernetes namespace, a comment-only change to a Terraform module, a documentation update. The cost of the change is low and the cost of a mistake is low; the cheap layers are sufficient.

For routine changes:

• Static and policy gates run on every PR.
• Unit and module gates run on every PR.
• Disposable integration is skipped (it runs on a schedule anyway, so it catches the routine change when it runs next).
• Staging and production validation runs because the change is being applied; the smoke test fires either way.

The discipline for routine changes is to keep the per-PR pipeline fast. A five-minute per-PR gate is a gate that engineers will not bypass; a thirty-minute per-PR gate is a gate that produces “I’ll just push it directly to main” exceptions.

Standard changes — the cheap layers plus scheduled integration

A standard change is the bread and butter of the infrastructure pipeline: a new module, a new role, a new manifest, a parameter update. The blast radius is bounded but non-trivial. The cheap layers catch the syntactic and policy mistakes; the scheduled disposable integration catches the cloud-side mistakes; the staging and production validation confirms the deploy worked.

For standard changes:

• Static and policy gates run on every PR.
• Unit and module gates run on every PR.
• Disposable integration runs on a schedule (nightly or pre-merge) and catches the change when it runs.
• Staging and production validation runs because the change is being applied.

The discipline for standard changes is to make the cheap layers rigorous. The schedule of the disposable integration layer is the safety net; the cheap layers are the per-PR defence.

Sensitive changes — every layer, no exceptions

A sensitive change is a change whose blast radius is large enough that a single mistake has material consequences. Examples: an IAM policy that grants cross-account access, a security group rule that opens a port to the internet, a Kubernetes NetworkPolicy that broadens egress, a Terraform backend that points to a different state store, an S3 bucket policy that grants public access.

For sensitive changes:

• Static and policy gates run on every PR.
• Unit and module gates run on every PR, with broader coverage than for standard changes.
• Disposable integration runs on the change itself, not on a schedule. Terratest, Molecule cloud driver, and kubectl apply --dry-run=server against a real cluster fire as part of the change pipeline.
• Staging and production validation runs with manual review. The canary window is longer; the rollback threshold is tighter.

The discipline for sensitive changes is to make the expensive layers non-skippable. The change cannot merge until the expensive layers pass; the apply cannot run until the manual review signs off. The cost is dollars per change and minutes of wall-clock; the value is the catch of a mistake that would otherwise reach production.

Critical changes — every layer plus manual approval

A critical change is a sensitive change whose blast radius extends beyond the team’s immediate production: a change that affects cross-account access, a change that affects production data, a change that affects a multi-tenant boundary, a change that affects the audit trail itself. The cost of a mistake is not bounded by the team’s own blast radius; it extends to other teams, other tenants, or the audit posture of the organisation.

For critical changes:

• Every layer of the pyramid runs as for a sensitive change.
• A second engineer reviews the change in addition to the standard PR review. The reviewer is named, not just required.
• The change is applied in a maintenance window with rollback pre-staged.
• The change is announced to dependent teams before it runs.

The discipline for critical changes is to make the manual review the gate, not the automated tests. The expensive layers are still required; the manual review is the layer that catches the strategic mistake that no automated test can catch.

flowchart TB
    A[Change] --> B{Blast radius}
    B -->|routine| C[Cheap layers only]
    B -->|standard| D[Cheap + scheduled integration]
    B -->|sensitive| E[Every layer, no exceptions]
    B -->|critical| F[Every layer + manual approval]
    C --> G[Merge]
    D --> H[Merge + nightly integration catches it]
    E --> I[Merge + on-demand integration + canary]
    F --> J[Merge + manual review + maintenance window]

The strategy decision as a recurring call

The matrix above is not a one-time policy; it is a recurring decision the team makes on every change. Three signals to read:

• The blast radius of the change. A new IAM policy is sensitive; a new label is routine. The signal is in the change itself.
• The cost of a mistake. A mistake that costs dollars to fix is different from a mistake that costs an outage to fix. The signal is in the change’s failure mode.
• The team’s confidence in the cheaper layers. A team whose fmt gate has caught every mistake for six months has higher confidence in fmt than a team whose fmt gate is new. The signal is in the team’s history.

The strategy decision is the explicit act of mapping the change to the layers, and of documenting the mapping in the change-gate policy. A team that does not make the decision explicitly is a team that runs the same gates for every change, which is over-investing in routine changes and under-investing in sensitive ones.

Production discipline

1. The change-gate policy names the four classes of change and the layers for each. Routine, standard, sensitive, critical — the matrix is documented.
2. The expensive layers are reserved for the changes that justify them. Terratest on a tag change is a budget event, not a safety improvement.
3. Sensitive changes have non-skippable expensive layers. The gate cannot be bypassed; the change cannot merge.
4. Critical changes have a named second reviewer. The reviewer is a person, not a role.
5. The matrix is reviewed quarterly. As the cheap layers prove they can absorb more, the matrix tightens; as the team’s threat model evolves, the matrix loosens.

Cross-course references

• Terraform for Production Sysadmins - Part XXI (ModulePatterns) is where the strategy decision lives for Terraform changes.
• Ansible for Production Sysadmins - Part XXVI (Testing) is the Molecule-specific framing.
• This course, Part XLIX (InfrastructureCI) - lesson git-cicd-gitops-xlix-06-plan-and-review is the broader framing of how the layers compose with the plan-and-review stage.
• Linux for Production Sysadmins - Part XXXVIII (CostControls) is the cost-discipline framing; the strategy decision is part of that discipline.

Quiz