Post-change validation and rollback gates — closing the window with evidence

A window closes when validation passes, not when the change runs. The validation is the evidence: KPIs, PDBs, synthetic traffic, and the operator who is willing to sign. A window that closes without validation is a window that has decided the change succeeded.

The validation gate

A post-change validation is a list of conditions that must be true after the change. The conditions are written before the change; they are not discovered after. A post-change check that is invented on the spot is a check that the operator expects to fail.

flowchart TD
    A[Change executed] --> B[Run validation checks]
    B --> C{KPIs healthy?}
    C -->|No| D[Rollback]
    C -->|Yes| E{PDB satisfied?}
    E -->|No| D
    E -->|Yes| F{Synthetic traffic OK?}
    F -->|No| D
    F -->|Yes| G{Workload Ready?}
    G -->|No| D
    G -->|Yes| H[Sign off]
    H --> I[Close window]
    D --> J[Investigate]
    J --> K[Post-mortem]

Each check is a gate. The window is closed only when every gate passes.

The validation checks

The validation checks are deliberately mechanical. A check is mechanical if it can be expressed as a command and a predicate. The ideal is a CI-style automation that runs the checks and reports “PASS” or “FAIL” with evidence.

# PDB compliance
kubectl get pdb -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.currentHealthy}{"/"}{.spec.minAvailable}{"\n"}{end}'

# Workload readiness
kubectl get deploy -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.status.readyReplicas}{"/"}{.spec.replicas}{"\n"}{end}'

# Synthetic traffic
for endpoint in prod-app billing-api; do
    curl -fsS -o /dev/null -w "%{http_code} %{time_total}\n" \
        https://${endpoint}.prod.example.com/healthz
done

The checks are documented in the runbook. The same checks run in staging after the rehearsal; the same checks run in production after the change. The differentiator is the operator’s pager.

“Looks healthy” vs “validated”

A common failure mode is to close the window because the cluster “looks healthy.” The cluster’s API responds, the kubectl get nodes shows Ready, the dashboards have no alarm. None of that is validation.

The validation is the evidence that the change’s specific goal was achieved. A kubelet upgrade that “looks healthy” has not been validated until the kubelet’s new flag is in --kubelet-version, the workload’s Pods are scheduled by the new binary, and the new feature (--dynamic-config, --max-pods increases) is functional. A network policy that “looks healthy” has not been validated until the synthetic traffic from the affected source lands on the workload and returns the expected status code.

The discipline is to fail closed. If the validation cannot be run, the window is not closed.

The rollback gate

The rollback gate is the trigger that fires when the validation fails. The gate is not “if we’re worried.” The gate is “if the validation check X returns Y.”

The rollback is the procedure documented in the pre-change gate. The procedure is rehearsed in staging; the procedure is executed in production when the gate triggers. The execution is by the same operator who executed the change, with the same pager.

stateDiagram-v2
    [*] --> ChangeApplied
    ChangeApplied --> Validation
    Validation --> Passed: KPIs, PDB, traffic OK
    Validation --> Failed: any check fails
    Passed --> Closed
    Failed --> Rollback
    Rollback --> ReValidated
    ReValidated --> Closed
    ReValidated --> Escalated
    Escalated --> Closed

The Escalated state is the path when the rollback does not restore the cluster. The escalation is the post-incident; the rollback was the immediate response.

The post-incident review

A post-change is paired with a post-incident review (PIR) when the change caused an incident, and with a post-change review (PCR) when it did not. The PCR is the lighter-weight version: the change succeeded, but the question “what did we learn?” gets asked.

The PCR is documented in the same template:

• What was the change?
• What was the risk class?
• What was the actual impact?
• What did the validation check?
• What did the rollback path look like?
• What was the surprise?
• What will the next window do differently?

The PCR is the artefact that pays down the lesson. A cluster that runs PCRs on every change is a cluster that is improving its windows every quarter.

Production discipline

A window closes when validation passes, not when the change runs. The validation is the evidence — KPIs, PDBs, synthetic traffic, and the operator who is willing to sign. The cluster’s discipline is the same scale-free: every change gets the same gate, the same validation, and the same review.

• Validation is mechanical. A check is a command and a predicate; the result is PASS or FAIL.
• Validation runs in staging first. The same checks run in production; the difference is the pager.
• A PCR runs on every change. The PCR is the lighter-weight review that turns the change into an improvement.
• The same operator closes the window that opened it. The signature is the contract.

Quiz