Change plans and peer review - the discipline of safe changes · Linux

A change plan describes what will be done, why, and how it will be reversed. Peer review catches errors before they happen. This lesson covers the discipline.

What is a change plan

A change plan describes:

What: the change being made.
Why: the reason (CVE, feature, performance).
When: the schedule.
Who: the operator(s).
How: the steps.
Risk: what could go wrong.
Rollback: how to reverse the change.
Validation: how to verify success.

The plan is submitted for review, approved, and executed. A change without a plan is an incident waiting to happen.

What is peer review

Peer review is a second pair of eyes:

The author writes the change.
A peer reviews the change.
The peer checks correctness, edge cases, rollback, and validation.
The peer approves or requests changes.
The change is executed.

Peer review catches:

Typos and configuration errors.
Missing steps (e.g. no validation).
Incorrect rollback (e.g. no plan to reverse).
Misalignment with the goal (the change does not solve the problem).

Why peer review works

A second person:

Sees what the author is too close to see.
Has different mental models and edge cases.
Catches assumptions the author did not articulate.
Forces the author to articulate the plan.

The cost is time (15-30 minutes per change). The benefit is catching errors before they reach production.

Change advisory board (CAB)

For larger changes, a Change Advisory Board (CAB) reviews the change:

Standard changes (low risk): pre-approved; no CAB.
Normal changes (medium risk): CAB review.
Emergency changes (high risk): expedited CAB; usually a security orker.

CABs balance speed (need to deploy) with safety (need to avoid breaking things).

The change record

Every change produces a record:

CHANGE RECORD
============
Date: 2026-08-09 14:00
Change: Update nginx to 1.27.2
Reason: CVE-2024-XXXXX (Critical)
Risk: Medium; nginx may have a regression
Rollback: reinstall from the .deb captured in the pre-check
          (/var/backups/rollback/), then apt-mark hold nginx.
          Fallback: restore the pre-change LVM snapshot
          pre-nginx-1.27.2.
Validation: smoke test, latency, log scan
Approved by: <peer>
Executed by: <operator>
Result: success

`apt install pkg=old-version` is a rollback that usually cannot run

The obvious rollback line - apt install nginx=1.27.1-1 - is the one most change records carry, and at the moment you need it the version is normally gone.

Debian and Ubuntu pockets carry one version per pocket. When a security update lands, the version it replaced is removed from -security and -updates; only the current version and the original release version remain. On this host, right now:

apt-cache madison nginx
ls /var/cache/apt/archives/*.deb | wc -l

Three lines, two distinct versions, and an empty archive cache - the default APT::Periodic::AutocleanInterval clears it. Every intermediate version the fleet has ever run is unfetchable.

So capture the artefact before the change, while it still exists:

# PRE-CHANGE - part of the change, not part of the rollback
OLD=$(dpkg-query -W -f='${Version}' nginx)
sudo install -d -m 0755 /var/backups/rollback
sudo apt-get install --download-only --reinstall -y "nginx=$OLD"
sudo cp /var/cache/apt/archives/nginx_*.deb /var/backups/rollback/
printf '%s\n' "$OLD" | sudo tee /var/backups/rollback/nginx.version

# ROLLBACK - works because the artefact was captured, not assumed
sudo dpkg -i /var/backups/rollback/nginx_*.deb
sudo apt-mark hold nginx    # stop the next upgrade re-applying it

The apt-mark hold matters as much as the downgrade: without it, the next unattended-upgrade run puts the bad version straight back, usually overnight and usually unobserved.

RHEL is better placed - dnf history undo <id> works while the old package is still in the cache or the repository - but the same caveat applies once the cache is cleaned. See linux-package-cache-and-cleanup, which states the constraint directly: apt has no built-in rollback.

A rollback step that has not been rehearsed on a host that has already taken the update is not a rollback plan. It is a sentence in a document.

The record is the audit trail. Compliance and post-incident review depend on it.