Change classification and approval · Proxmox VE

Change management has a poor reputation among engineers, mostly deserved. It is frequently a form that adds three days to a ten-minute task and is approved by somebody who could not evaluate it.

That is a criticism of bad change management, and the answer is not to abandon it. The overwhelming majority of self-inflicted Proxmox outages were approved changes — approved because the record said “add a firewall rule” and nobody asked which chain, on which level, and whether the person running it would still be able to reach the node afterwards.

The purpose of classification is to route each change to the amount of scrutiny it actually needs. Get the classification right and most changes get almost none, which is what makes the process survivable.

Classify by three properties, not by intuition

“Risky” is not a property of a change. These three are, and they can each be answered in a sentence.

Blast radius — how far can this reach if it goes wrong?

Radius	Examples on a Proxmox cluster
One guest	Resize a disk, change a guest firewall rule, add a NIC, snapshot
One node	Kernel update, NIC driver change, node firewall rule, local storage change
All guests on one node	Anything requiring a node reboot; a network change on the node
The whole cluster	`corosync.conf`, `datacenter.cfg`, shared storage definitions, HA rules, cluster firewall, `pveum` roles, certificate changes
The data	Ceph pool parameters, ZFS pool geometry, storage removal, retention policy

The line that catches people is between the third and fourth rows. A change made on one node to a file inside /etc/pve is a cluster-wide change, because /etc/pve is replicated. Editing the datacenter firewall on pve-01 applies it to pve-03 before you have finished reading it back.

Reversibility — can you undo it, and how fast?

Class	Meaning	Example
Trivially reversible	One command, seconds, no data implication	Guest firewall rule, `qm set --description`
Reversible with disruption	Undo requires a restart or migration	Kernel pin and reboot, machine-version change
Reversible with effort	Restore from backup, rebuild	Node reinstall, storage reconfiguration
Irreversible	No path back at all	See the list below

Detectability — how quickly will you know it went wrong?

Class	Meaning	Example
Immediate	Fails visibly within seconds	A guest that will not start
Same session	Visible during verification	A migration that fails
Delayed	Surfaces hours or days later	A backup job that now silently skips a guest
Latent	Surfaces only under failure	A corosync change that works fine until a link drops; a `min_size` change that only matters when an OSD dies

Latent detectability is the property that makes a change dangerous, and it is the one nobody assesses

Blast radius and reversibility get asked about. Detectability almost never does, and it is the property that turns a small change into a large incident.

Consider two changes with identical blast radius and reversibility:

Adding a second corosync link. If misconfigured, it works perfectly. Everything is green. Six weeks later the primary link fails, the secondary does not take over because the address was wrong, and nodes fence.
Changing a Ceph pool from min_size=2 to min_size=1 to clear a warning during a maintenance window. Nothing happens. It clears the warning. Eight months later an OSD fails and the cluster keeps serving from a single copy, and the next error destroys data.

Both are one-line changes, both are reversible in one command, and both were verified as successful at the time — because the verification available at the time could not distinguish success from the failure mode.

A change whose failure mode only appears during a subsequent failure must be tested by inducing that failure. For the corosync link, that means pulling the primary link in a window and confirming failover. For the pool change, it means never making it. If you cannot test it that way, the change does not get approved on the strength of “it worked” — it gets approved on the strength of somebody having reasoned through what would happen.

The changes that cannot be undone

Every platform has a list. This is Proxmox’s, and it is worth knowing by heart because these are the ones where “we can always roll it back” is false.

Operation	Why there is no way back
`volblocksize` on a zvol	Set at creation only. Changing it means creating a new zvol and copying the data
Removing a vdev from a RAIDZ pool	Not supported. The pool must be destroyed and rebuilt
Reducing a Ceph pool’s `size`	The extra replicas are deleted immediately. Raising it back rebuilds from what remains, which is now fewer copies than you had
`pvecm delnode` on a node you want back	The node must be reinstalled; rejoining a previously removed node with its old identity is not supported
Destroying a guest with `--purge`	Removes the configuration, the disks, and the backup job and replication references
Removing a storage that still holds content	The definition goes; whether the data goes depends on the backend, and you will find out afterwards
Reducing a filesystem inside a guest without shrinking the disk first	Ordering error, not a Proxmox one, and it is the classic route to an unbootable guest
Deleting a `pveum` role that ACLs reference	The ACL entries are dropped, not preserved. Recreating the role does not restore them
Pruning a PBS backup group	Snapshots are gone. Retention protects you only until somebody runs a prune with different parameters

The three classes, and what each entitles you to

Class	Definition	Approval	Scheduling
Standard	Pre-approved, repeatedly executed, documented procedure, well-understood outcome	None per instance — the procedure was approved once	Any time
Normal	Everything else that is planned	Approval before scheduling	In an agreed window
Emergency	Required to restore service or remediate an active threat	Retrospective, with a named authoriser at the time	Immediately

What qualifies as standard on a Proxmox cluster

A change is standard when it satisfies all of: blast radius of one guest or smaller, trivially reversible, immediate detectability, and a written procedure that has been executed successfully at least a handful of times.

Realistic standard changes:

Create a guest from an approved template
Extend a guest disk (extending, never shrinking)
Add or remove a guest firewall rule from an approved rule set
Take or delete a guest snapshot
Start, stop or reboot a guest inside its own maintenance window
Live-migrate a guest between nodes on shared storage
Add a guest to an existing backup job

The point of the standard class is that these should be fast. If your process makes creating a VM from a template take two days, engineers will route around the process, and then you will have no change record for anything.

Normal changes, with their real classification

Change	Blast radius	Reversibility	Detectability	Handling
Node kernel or microcode update	All guests on the node	Reversible — kernel pin to the previous version	Immediate at boot, delayed for driver regressions	Rolling, one node, verify before the next
Add a node to the cluster	Cluster	Reversible with effort	Immediate	Normal, in a window
Add or change a storage definition	Cluster	Reversible	Delayed — failures show at the next backup or migration	Normal, verify by exercising it
Change datacenter firewall rules	Cluster	Reversible, if you still have access	Immediate, possibly to your own session	Normal, with console access confirmed first
Change `corosync.conf`	Cluster, and quorum	Reversible with effort	Latent	High-risk normal, tested by inducing failure
PVE minor version upgrade	Rolling, per node	Reversible with effort	Mixed	Normal, one node first, soak, then the rest
PVE major version upgrade	Cluster	Reversible only by restore	Mixed	High-risk normal — own window, own plan
Ceph release upgrade	The data	Effectively irreversible once OSDs are converted	Delayed	High-risk normal
Change Ceph pool `size` or `min_size`	The data	Reducing `size` is irreversible	Latent	High-risk normal, and reducing needs an explicit data-loss review
Certificate replacement on `pveproxy`	Cluster access	Reversible	Immediate	Normal, but hold a shell open

The rule that prevents the most common category of self-inflicted outage

Never make a change that could remove your access, without an independent path to the node already open and verified.

Firewall rules, network reconfiguration, sshd changes, certificate replacement, pveproxy changes — all of them can end your own session, and the recovery for every one of them is a console you either have or do not.

The pre-change step is thirty seconds: open the out-of-band console (iDRAC, iLO, IPMI), log in, confirm you have a prompt, and leave it open. Not “we have iDRAC configured”. A window with a shell in it, right now.

The Proxmox-specific version of this trap is that the firewall is cluster-wide. A datacenter-level rule applies to every node simultaneously, so the mistake does not cost you one node — it costs you all of them at once, and there is no surviving node to fix it from.

What the approver actually needs

Most change records answer “what are you doing”. An approver cannot evaluate risk from that. Six questions, and a record that answers them can be approved in two minutes.

Question	What a bad answer looks like	What a usable answer looks like
What changes, exactly?	“Update the firewall”	“Add one rule to the datacenter security group `mgmt-in`, permitting TCP 8006 from 192.0.2.0/24. No other rule modified”
On which hosts?	“The cluster”	“Datacenter level, therefore all 5 nodes simultaneously”
Who is affected if it goes wrong?	“Nobody, it is low risk”	“All GUI and API access from outside the management network. Guests unaffected. 3 automation systems use the API from that range”
How will you know within 5 minutes?	“We will check it works”	“From an external host: curl the API on each node, expect 200. From inside: confirm the 3 automation systems still poll successfully”
How do you undo it, and has that been run?	“Revert the change”	“`pvesh delete` the rule, tested in the lab cluster on 2026-08-05. Console open on all 5 nodes throughout”
What if the backout fails?	(usually blank)	“Console access to each node; the firewall can be disabled locally with `pve-firewall stop`, which is node-local and does not need the cluster”

The fifth and sixth questions are the ones that separate a real record from a ceremonial one, and the sixth is nearly always blank. It is also the one that matters most, because the incidents that become long incidents are the ones where the backout did not work.

Read-only / Safethe pre-change evidence pack, gathered before you touch anything— Read-only. Take this on every normal change. It is the before picture you will need to prove whether the change caused the thing somebody noticed two hours later.

set -euo pipefail
CHANGE=CHG-2026-0412
OUT="/root/change-evidence/$CHANGE"
mkdir -p "$OUT"

# Cluster state before.
pvecm status                      > "$OUT/pvecm-status.txt"
pvesm status                      > "$OUT/pvesm-status.txt"
ha-manager status                 > "$OUT/ha-status.txt"
pveversion -v                     > "$OUT/pveversion.txt"

# Guest inventory and run state.
qm list                           > "$OUT/qm-list.txt"
pct list                          > "$OUT/pct-list.txt"

# The cluster configuration itself. This is the artefact a backout needs.
tar czf "$OUT/etc-pve.tar.gz" -C /etc pve

# Storage-specific state, where applicable.
ceph -s                           > "$OUT/ceph-status.txt" 2>/dev/null || true
zpool status                      > "$OUT/zpool-status.txt" 2>/dev/null || true

# Copy it OFF the cluster before making the change.
echo "evidence in $OUT - copy it off-cluster now"

Why a rolling change needs a soak period, and how long it should be

The instinct on a rolling cluster change — a kernel update, a minor version upgrade — is to do node one, confirm it comes back, and immediately do node two. It feels efficient and it defeats the purpose of rolling.

The value of rolling is that the first node is a test, and a test needs long enough to fail. What you are waiting for is not “did it boot” — you knew that in three minutes. You are waiting for the classes of failure that only appear under real workload:

Driver and firmware regressions that surface under sustained I/O or network load, not at boot.
Guest incompatibilities that appear when a guest is live-migrated onto the upgraded node, which requires a guest to actually migrate.
Backup interactions, which appear at the next scheduled backup run.
Scheduled job failures, which appear when the job runs.

That last one sets the sensible minimum: soak for at least one full backup cycle. On a nightly backup schedule that is one night. A cluster upgraded across all nodes in a single afternoon has, by construction, not run a single backup on the new version before every node was committed to it.

The counter-argument is that a long soak means running a mixed-version cluster, which has its own risks — live migration between different QEMU versions in particular. That is a real tension and it resolves in favour of the soak, because a mixed-version cluster is a documented, supported and temporary state, whereas a fleet-wide regression discovered at 03:00 on the first backup run has no supported state to return to.

What happens when an emergency change turns out not to have been an emergency?

The emergency class exists so that restoring service is never blocked by paperwork. That is correct and non-negotiable.

Its failure mode is drift. “Emergency” is faster, so under pressure a change that is merely urgent gets labelled emergency, and once that has worked twice it becomes the default route for anything inconvenient. Six months later the normal process is a fiction and every meaningful change is going in under retrospective approval, unreviewed, at speed.

The controls that keep the class honest are cheap and they are all retrospective:

A named authoriser at the time. Not a form afterwards — a person who said yes and is recorded as having said so. This alone stops most self-authorised “emergencies”.
Retrospective review within a fixed period, typically two business days, in which the change is assessed as it would have been assessed normally. The output is a judgement: was this genuinely an emergency?
A visible count. Emergency changes as a proportion of all changes, reported monthly. A healthy platform runs at a few percent. Above roughly fifteen percent, the number is telling you something structural — usually that the normal path is too slow, and the fix is to speed up the normal path rather than to police the emergency one.

The last point is the important one. A high emergency rate is almost never a discipline problem. It is a process design problem wearing a discipline costume, and treating it as the former makes it worse.

Post-implementation review, in five minutes

Not a ceremony. Five questions, written into the change record when it is closed, and reviewed in bulk once a quarter:

Did it do what it was supposed to? Yes, partially, no.
Did the verification actually verify? Would it have caught a failure, or was it “it looked fine”?
Was the classification right? In hindsight, was the blast radius larger than assessed?
Did anything unexpected happen? Including things that turned out harmless.
Should this become a standard change? After the third or fourth uneventful execution of the same normal change, promoting it is what keeps the process from becoming overhead.

Question 5 is the one that makes the whole system sustainable. A change process that only ever adds scrutiny becomes a tax; one that promotes proven procedures into the standard class gets faster over time, and that is what keeps engineers using it.

Common mistakes

Classifying by how hard the change feels, rather than by blast radius, reversibility and detectability.
Forgetting that /etc/pve is cluster-wide. A change made on one node is a change made everywhere.
Never assessing detectability, so latent failures are approved as low-risk.
A backout plan that has never been executed.
No answer to “what if the backout fails”.
Making an access-affecting change without a console already open.
Rolling changes with no soak, so the first backup on the new version runs after every node is committed.
Emergency drift, and treating the resulting rate as a discipline problem.
Never promoting proven changes to standard, so the process becomes a tax and gets bypassed.

Key takeaways

Classify on blast radius, reversibility and detectability. Latent detectability is the property that is never assessed and matters most.
Anything written into /etc/pve has cluster-wide blast radius regardless of which node you typed it on.
Know the irreversible list. For those changes the backout is a restore, and the restore must have been tested.
Standard changes must be genuinely fast, or the process gets bypassed.
The approver needs six answers, and the two usually missing are “has the backout been run” and “what if it fails”.
Soak a rolling change for at least one full backup cycle.
A high emergency rate is a process-design signal, not a discipline problem.

Knowledge check

Why this matters

The value of change classification is not that it prevents mistakes. It is that it puts the effort where the danger is.

Every operations team has finite attention for scrutiny. Spend it uniformly and you get a process that is simultaneously too slow for a template deployment and too shallow for a Ceph upgrade — and engineers, quite reasonably, start treating both as paperwork. Spend it by blast radius, reversibility and detectability and the template deployment takes no approval at all, while the pool parameter change gets the conversation it deserves.

The Proxmox-specific content matters here more than the framework. /etc/pve being cluster-wide, a reduced Ceph size being unrecoverable, a firewall rule reaching every node at once, a corosync change that will not fail until the day it matters — those are the facts that turn a generic risk assessment into an accurate one, and none of them are obvious from the interface you make the change in.

The habit worth building is smaller than any of this. Before a change, ask the sixth question: if the backout does not work, what then? Most of the time there is a good answer and it takes ten seconds to say. The times there is not are exactly the changes worth slowing down.