Change management as risk governance — the goal of a change

Change management is the discipline of paying down the risk of change in small, well-understood pieces. The goal is not zero changes; the goal is reversible, well-understood change. A cluster that cannot change is a cluster that is decaying; a cluster that changes without governance is a cluster that is an incident away from an outage.

The four pillars

A change is governed by four pillars:

1. Plan. The change is described in a document: what is changing, why, what is the risk class, what is the rollback, what is the validation, what is the communication.
2. Review. A second pair of eyes reads the plan. The reviewer is not the author; the reviewer’s job is to find the thing the author missed.
3. Execute. The change is run during a window, with the steps from the plan, with the rollback in reach.
4. Verify. The validation runs. The window closes. The post-change review (PCR) is written.

flowchart LR
    A[Plan] --> B[Review]
    B --> C[Execute]
    C --> D[Verify]
    D --> E[PCR or PIR]
    E --> F[Improvements to next change]
    F --> A

The cycle is the same for every change. The depth of each pillar scales with the risk class: a one-line config tweak gets a single-paragraph plan and a peer review; a control-plane upgrade gets a multi-page plan, a CAB review, and a staged rollout.

Risk classes

A change is classified by the magnitude of its blast radius:

• Class 1 — No impact. A change that cannot affect the workload’s behaviour. A label flip on a non-selected resource. A documentation update. The review is a peer.
• Class 2 — Bounded impact. A change that can affect a single workload, in a single namespace, with a tested rollback. A deployment rollout. A config push. The review is a senior engineer.
• Class 3 — Cluster-wide. A change that can affect the control plane, multiple namespaces, or the cluster’s connectivity. A CNI upgrade. An etcd compaction. The review is a Change Advisory Board (CAB).

flowchart TD
    A[Change] --> B{Risk class?}
    B -->|Class 1| C[Peer review]
    B -->|Class 2| D[Senior review]
    B -->|Class 3| E[CAB review]
    C --> F[Single window]
    D --> G[Pre-staging rehearsal]
    E --> H[Multi-stage rollout]

The class determines the review depth. A Class 1 review that turns out to be Class 3 is a failure of judgement; the remediation is to re-classify and re-review.

The plan

The plan is the change’s contract. The plan is a single document. It contains:

• Summary. A one-paragraph description of the change.
• Risk class. The class and the rationale.
• Owner. The human whose pager lights up.
• Pre-change gate. The conditions that must be true before the window opens.
• Execution. The exact command sequence.
• Validation. The post-change checks.
• Rollback. The procedure if validation fails.
• Communication. The audience, the channel, the timing.
• PIR/PCR. The template for the review.

A plan that is missing any of these is a plan that has decided the missing part is not important.

The review

The review is the second pair of eyes. The reviewer’s job is to find the thing the author missed: the PDB that the drain would violate, the half-imaged node that the cordon would ignore, the certificate that the upgrade would rotate, the readiness probe that the new version would fail.

The review is not a rubber stamp. The review is a contract: the reviewer has signed that they have read the plan and have not found a fatal error. The signature is the audit trail.

The execution

The execution is the plan in action. The execution is run by the owner, with the same pager, with the rollback in reach. The execution is the moment when the plan is no longer a document; it is the cluster.

The most common execution failure is the “in-flight decision”: the operator notices something the plan did not cover and makes a judgement call. The judgement call is a change that was not in the plan. The remediation is to stop and amend the plan, or to escalate.

The verify

The verify is the post-change validation. The verify is the evidence: KPIs, PDBs, synthetic traffic, the operator’s signature. The verify is the only thing that closes the window.

Production discipline

Change management is the discipline of paying down the risk of change in small, well-understood pieces. The cluster’s discipline is the four pillars: plan, review, execute, verify. The cycle is the same for every change; the depth scales with the risk class.

• Execute within the window. The window is the rectangle that contains the change; an in-flight decision that the plan did not cover belongs outside it, so stop and amend the plan or escalate.
• Verify with mechanical checks. KPIs, PDBs, synthetic traffic, and the operator’s signature are the evidence that closes the window — nothing else does.

Quiz