Risk classification and CHG documents — the cost of a change

A change is class 1, 2, or 3 by blast radius and recovery cost. The CHG document is the contract. Misclassification is paid for in incident response: a Class 3 change that is labelled Class 2 gets the review of a Class 2 and the impact of a Class 3.

The three classes

A production change is classified by two axes:

1. Blast radius. What is the worst case if the change goes wrong?
2. Recovery cost. How long does it take to restore the cluster to its pre-change state?

flowchart TD
    A[Change] --> B{Blast radius?}
    B -->|None| C[Class 1]
    B -->|Bounded| D[Class 2]
    B -->|Cluster-wide| E[Class 3]
    A --> F{Recovery cost?}
    F -->|< 5 min| C
    F -->|5–60 min| D
    F -->|> 60 min| E

The class is the maximum of the two axes. A change with a Class 2 blast radius but a Class 3 recovery cost is a Class 3 change.

Class 1 — no impact

A Class 1 change cannot affect the workload’s behaviour. The recovery is to revert the change in a single command.

Examples:

• A label flip on a non-selected resource.
• A documentation update.
• A namespace label that is not selected by any policy.
• A config map update that is not mounted by any workload.

The review is a peer. The window is a routine rollback window. The validation is a kubectl get of the affected resource.

A change that turns out to be Class 2 is a misclassification. The remediation is to re-classify and re-review.

Class 2 — bounded impact

A Class 2 change can affect a single workload, in a single namespace, with a tested rollback. The recovery is to run the rollback procedure.

Examples:

• A Deployment rollout.
• A config push that is mounted by a workload.
• A version bump of a sidecar.
• A HPA tuning.

The review is a senior engineer. The window is a planned maintenance window if the workload is business-critical. The validation is a synthetic traffic check + workload Ready.

A change that turns out to be Class 3 is a misclassification. The remediation is to re-classify and re-review.

Class 3 — cluster-wide

A Class 3 change can affect the control plane, multiple namespaces, or the cluster’s connectivity. The recovery is longer than an hour.

Examples:

• A CNI upgrade.
• An etcd compaction.
• A control-plane upgrade.
• A StorageClass migration.

The review is a CAB. The window is a planned, communicated maintenance window. The validation is comprehensive: KPIs, PDBs, synthetic traffic, control-plane health.

The CHG document

The CHG (change) document is the contract. It is a single document with the following fields:

• ID. A unique identifier.
• Title. A short description.
• Owner. The human whose pager lights up.
• Risk class. 1, 2, or 3 with rationale.
• Summary. A one-paragraph description.
• Pre-change gate. The conditions that must be true.
• Execution. The exact command sequence.
• Validation. The post-change checks.
• Rollback. The procedure if validation fails.
• Communication. The audience, the channel, the timing.
• Reviewer. The human who signed.
• Status. Draft, approved, executing, validated, closed.

stateDiagram-v2
    [*] --> Draft
    Draft --> Approved: reviewer signs
    Approved --> Executing: window opens
    Executing --> Validated: validation passes
    Executing --> RollingBack: validation fails
    Validated --> Closed
    RollingBack --> Closed
    Closed --> [*]

A CHG document that is missing a field is a CHG document that has decided the missing field is not important.

Conservative classification

When in doubt, classify higher. The cost of a Class 3 review on a Class 2 change is a longer meeting. The cost of a Class 2 review on a Class 3 change is an incident.

The discipline is to ask: what is the worst plausible outcome, and what is the recovery cost? If the answer is “two hours,” the change is Class 3.

Production discipline

The classification is the contract. The CHG is the audit trail. The discipline is to classify conservatively, fill every field, and re-classify when the change grows. The cluster’s risk is the sum of its misclassifications.

• Classify by blast radius and recovery cost. The class is the maximum of the two.
• Re-classify when the change grows. A change that started as Class 2 and grew to Class 3 is re-classified, not retroactively approved.

Quiz