Responding to a Proxmox security advisory

A security advisory arrives on a Tuesday afternoon, describes something alarming, and gives you no information about whether your cluster is affected. The work between that message and a patched estate is a process, and teams that have not written it down improvise it under time pressure — which reliably produces one of two failures: patching everything immediately and causing an outage, or discussing it until the following month.

This lesson is that process for a Proxmox estate specifically.

Where advisories for a Proxmox estate come from

A PVE node is a Debian system with Proxmox packages on it, running guest workloads, on hardware. Four independent streams, and only one of them is Proxmox’s.

The stream people miss is the fourth. A BMC vulnerability is a management-plane compromise that no amount of host patching touches, and BMC firmware is not in apt. It reaches you through a vendor portal, on the vendor’s schedule.

The stream people over-weight is the second. Most CVEs that matter to a Proxmox estate are Debian CVEs, because most of the attack surface is Debian.

Triage: are you actually exposed?

Three questions in order. Stop at the first “no”.

1. Do you run the affected package, at an affected version?

PKG=qemu-server

dpkg -s "$PKG" | grep -E '^(Package|Version|Status):'
apt list --installed 2>/dev/null | grep -E 'pve-manager|qemu-server|pve-container'
apt-cache policy "$PKG"

Across a cluster, ask every node — version skew after a partial upgrade is exactly the state where one node is exposed and the others are not:

pvesh get /nodes --output-format json | grep -o '"node":"[^"]*"'
pveversion --verbose

Debian’s security tracker is the authoritative answer for whether a specific Debian version is fixed, and it distinguishes fixed, vulnerable and not affected per suite. That distinction matters: a CVE marked “not affected” for Trixie because the vulnerable code path was never compiled in is a different verdict from one awaiting a fix, and only the second needs a mitigation.

2. Is the vulnerable path reachable in your configuration?

This is where most advisories stop being urgent, and it is also where teams talk themselves into a wrong answer.

Legitimate reasons an exposure does not apply:

• The vulnerable feature is not enabled. A flaw in the SPICE proxy does not apply to a cluster where no guest has a SPICE display.
• The vulnerable service is not reachable from any network an attacker is on — but verify that with pve-firewall simulate rather than from the network diagram.
• The affected component is not installed. proxmox-firewall is not installed unless you installed it.

Reasons that sound legitimate and are not:

• “It needs authentication.” Every Proxmox estate has authenticated principals with less than full privilege. An authenticated privilege-escalation flaw is exactly a flaw against them, and against every API token.
• “It is only local.” Guests are local. Containers share the host kernel. A local privilege escalation in the host kernel is a container escape.
• “It is only exploitable from the guest.” That is the threat model of a hypervisor. The guest is where the attacker is presumed to be.

3. What does the exposure actually reach?

A hypervisor advisory has an unusually wide blast radius, and this is the step that sets the priority. Score it against three questions: does it cross the guest-to-host boundary, does it reach the management plane, and does it reach stored data — a flaw reaching the storage backend reaches every VM disk at once.

Record the verdict, including “not exposed” and why. In six months an auditor will ask about this CVE, and a documented not-exposed decision is an answer. Silence is a finding.

Emergency patching under HA

The ordinary rolling procedure applies with one addition: HA will react to the reboot unless you tell it not to.

NODE=$(hostname)

pveversion --verbose
ha-manager status
pvecm status
dpkg-query -W | sort > "/root/pkg-before-$NODE.txt"

NODE=$(hostname)
ha-manager crm-command node-maintenance enable "$NODE"

# Wait until nothing HA-managed is still assigned here before proceeding.
watch -n 5 "ha-manager status | grep -F '$NODE'"

Non-HA guests do not move on their own. Migrate them, or accept the downtime deliberately — the failure worth avoiding is discovering during the reboot that three important VMs were never HA-managed.

PKG=qemu-server

apt update
apt install --only-upgrade "$PKG"

apt-cache policy "$PKG"

Whether to reboot depends on what was patched. A userspace daemon needs its service restarted; a kernel or microcode fix needs a reboot to take effect. needrestart answers the first part and is worth having installed before you need it:

needrestart -r l

NODE=$(hostname)

pveversion --verbose
uname -r
ha-manager crm-command node-maintenance disable "$NODE"
ha-manager status

Then one node at a time. A cluster-wide simultaneous patch removes your ability to compare a patched node against an unpatched one, which is the fastest diagnostic available if the fix misbehaves — and it removes the option of stopping after node one.

Guests are a separate exercise

A patched hypervisor does nothing for a vulnerable guest, and the reverse is also true. Two cases specific to virtualisation are easy to miss:

A QEMU fix requires the guest process to restart. Live migration to a patched node starts a new QEMU process on the destination, so migrating a guest onto an already-patched node is a way to pick up a QEMU fix without guest downtime. A guest that stays put keeps running the old QEMU binary however many times you upgrade the package.

Templates carry the vulnerability forward. A template built before the fix produces vulnerable guests indefinitely. Patching the running estate and leaving the templates alone means the CVE returns with the next provisioning run, and the metric that said “100% patched” was measuring the wrong population.

The evidence trail

Three artefacts, all cheap if you collect them as you go and expensive to reconstruct later.

The triage record. The advisory ID, the date it arrived, the verdict, and the reasoning. Including for advisories you decided did not apply — those are the ones an auditor asks about, precisely because there is no patch to point at.

The before-and-after package state. You captured pkg-before in step

1. Capture the other half and diff it:

NODE=$(hostname)

dpkg-query -W | sort > "/root/pkg-after-$NODE.txt"
diff "/root/pkg-before-$NODE.txt" "/root/pkg-after-$NODE.txt"

/var/log/apt/history.log records every apt transaction with a timestamp independently, and is the artefact to reference when the question is when rather than what.

The timeline. Advisory published, advisory received, triage completed, patch available, first node patched, last node patched. The gaps between those timestamps are the actual measure of your patching capability, and they are the numbers a maturity assessment asks for.

Common mistakes

• Watching only Proxmox announcements. Most of the attack surface is Debian, and the BMC is not in apt at all.
• Dismissing an advisory because it needs authentication. Every estate has authenticated principals with partial privilege, and every API token is one.
• Patching all nodes at once under time pressure. No comparison point, no stopping point.
• Rebooting an HA node without maintenance mode. An unscheduled failover on top of an emergency change.
• Upgrading the package and not restarting the process. A patched library on disk is not a patched process in memory. needrestart -r l answers this.
• Forgetting templates. The CVE returns with the next provisioning run, and the patch metric was measuring the wrong population.
• No record for advisories judged not applicable. Those are the ones the audit asks about.
• apt clean mid-window. The cache is often the only copy of the version you would roll back to.

Key takeaways

• Four advisory streams: Debian, Proxmox, upstream projects, hardware vendors. Each needs a named owner and a destination someone reads.
• Triage is three questions — affected version, reachable path, blast radius — and the verdict gets recorded even when it is “not exposed”.
• Under HA, set node maintenance mode before rebooting; non-HA guests need moving by hand.
• Patch one node, soak it under real traffic, then continue. Emergency is not a reason to lose the comparison point.
• A QEMU fix reaches a guest when its process restarts; migrating onto a patched node achieves that without downtime.
• Patch the templates, or the vulnerability returns with the next provisioning run.
• With no patch available, reduce reachability and blast radius. Never disable a security control as the mitigation.

Knowledge check