Operating a fleet of clusters · Proxmox VE

One cluster has an answer to “where is VM 141”: the cluster. Four clusters do not, and everything that used to be a single lookup becomes a search.

That is the shape of the whole problem. Nothing about operating a fleet is conceptually new — inventory, change, alerting, on-call — but every one of them acquires an extra dimension, and the tooling that was adequate at one cluster quietly stops being adequate at three.

Inventory: one query per cluster, one table

The single most useful cluster-wide endpoint is /cluster/resources. It returns every node, storage, guest and pool in one call, which is exactly the thing you need per cluster to build a fleet view.

Read-only / Safeeverything this cluster knows about, in one call— Read-only. /cluster/resources returns nodes, storages, guests and pools with their status. --type narrows it. This is the endpoint the GUI resource tree is built from.

pvesh get /cluster/resources --output-format json | jq -r '.[] | [.type, .id, .node // "-", .status // "-"] | @tsv' | head -20

pvesh get /cluster/resources --type vm --output-format json | jq 'length'

Read-only / Safe

$ pvesh get /cluster/resources --output-format json | jq -r '.[] | [.type, .id, .node // "-", .status // "-"] | @tsv' | head

node	node/pve-a1	pve-a1	online
node	node/pve-a2	pve-a2	online
node	node/pve-a3	pve-a3	online
storage	storage/pve-a1/shared-rbd	pve-a1	available
storage	storage/pve-a1/local-zfs	pve-a1	available
qemu	qemu/141	pve-a2	running
qemu	qemu/142	pve-a1	running
lxc	lxc/205	pve-a3	running
pool	pool/tenant-blue	-	-

The cluster name is not in the data. Add it at collection time.

/cluster/resources does not return a cluster identifier, because a cluster has no reason to tell you its own name in an answer you asked it directly. The moment you aggregate two clusters, that becomes the most important missing field.

Add it in the collector, not in the consumer. Every row, every metric series and every alert should carry a cluster label from the point of collection onward. The alternative — inferring the cluster from the node name — works until somebody names a node inconsistently, and then produces an alert that sends a responder to the wrong site at 3 a.m.

pvecm status prints the cluster name if you want the cluster to declare it, but it is a per-node command, so the practical pattern is: your inventory configuration lists the clusters, and the collector stamps the name it used.

Read-only / Safea fleet inventory, stamped with the cluster name— Read-only. Iterates clusters defined in a table you control, queries each one with an auditor token, and emits one TSV with a cluster column. Replace the token lookup with your secret store; the shape is the point.

declare -A FLEET=(
[site-a]="192.0.2.10"
[site-b]="192.0.2.60"
[site-c]="192.0.2.110"
)

for cluster in "${!FLEET[@]}"; do
host="${FLEET[$cluster]}"
token="$(cat "/etc/fleet/tokens/${cluster}")"
curl -sS --fail \
  --header "Authorization: PVEAPIToken=fleet-audit@pve!ro=${token}" \
  "https://${host}:8006/api2/json/cluster/resources?type=vm" \
| jq -r --arg c "$cluster" '.data[] | [$c, .vmid, .name // "-", .node, .status] | @tsv'
done | sort

Read-only / Safe

$ ./fleet-inventory.sh | column -t

site-a  141     app-frontend-01   pve-a2  running
site-a  142     app-frontend-02   pve-a1  running
site-b  100141  db-primary        pve-b3  running
site-b  100142  db-replica        pve-b1  running
site-c  200310  build-runner-01   pve-c2  stopped

The receiving identity for this should be an auditor: a user with PVEAuditor on /, a privilege-separated token, and nothing else. It is read-only by construction, so a leak of a fleet inventory token is an information disclosure rather than an outage.

Alerting: two levels, and do not merge them

The instinct at cluster number two is to point everything at one alert pipeline. That is right for delivery and wrong for authorship, because two genuinely different classes of alert exist.

Per-cluster alerts are the ones you already have. A node down, a pool approaching full, a failed backup job, a Ceph health warning, an HA fencing event. Their scope is one cluster, they are meaningful without reference to any other cluster, and PVE’s own notification system can raise many of them.

Fleet alerts are the ones that do not exist at one cluster and cannot be computed inside one:

Fleet alert	Why one cluster cannot raise it
A cluster has stopped reporting entirely	The silent cluster is the one that would have told you
Version skew has exceeded policy	Requires comparing versions across clusters
A configuration difference has appeared since yesterday	Requires the drift report from the previous lesson
Aggregate capacity headroom is below the level needed to lose a site	Requires summing across clusters
A guest exists on two clusters with the same identity	Requires seeing both
The migration campaign has stalled	Requires state that spans the boundary

The most important of these is the first. Absence of signal is the failure mode that a per-cluster alerting design cannot see, because every alert it knows how to raise depends on the cluster being alive enough to raise it. Whatever aggregates your fleet must have a heartbeat per cluster and must alert on its absence, with a threshold shorter than the time it takes for that cluster’s silence to matter.

The quiet weekend

Three clusters, one Prometheus, one Alertmanager, dashboards per cluster. A firewall change on Friday afternoon blocked the scrape path to cluster C.

Nothing fired. Every alert rule was of the form “metric crosses threshold”, and cluster C produced no metrics, so no rule evaluated. The dashboards showed cluster C’s panels as “No data”, which on a Saturday looks identical to a panel nobody is looking at.

On Monday a cluster C node had been down for fifty hours, its HA services had relocated, and one guest on local storage had not been running since Friday evening. The backup job for cluster C had failed twice, silently.

The rule that would have caught it is one line — an up metric absent for more than ten minutes, per cluster — and it is the rule almost nobody writes, because at one cluster the monitoring system going quiet and the cluster going quiet were the same event and you noticed either way.

Staged change across clusters

The single biggest operational gain from having several clusters is that a change no longer has to land everywhere at once. That gain is only realised if you actually stage it, and staging has four components.

An order, written down. Least critical to most critical, with the validation cluster first. Not “we will start with whichever is convenient”.

A soak time between stages, defined in advance. Long enough for the failure mode you are worried about to appear. For a kernel or microcode change that means at least one full business cycle and one backup cycle; for a firewall rule it may be an hour. The number should come from the failure you are guarding against, not from the calendar.

Abort criteria, written before you start. “If any of these is true, stop and do not proceed to the next cluster.” Concrete and observable: a fencing event, a failed backup, a specific error in a specific log, a latency threshold. Criteria authored after a stage has gone wrong are negotiated criteria.

A rollback that has been tested at least once, on the validation cluster.

A staged rollout record

The whole thing fits on a page, and the page is the change record.

Field	Example
Change	PVE 9.2.4 → 9.2.5 on all clusters
Order	site-c (validation) → site-b → site-a
Soak	5 working days between stages; 7 before site-a
Per-stage validation	All nodes online, `pvecm status` quorate, one live migration succeeds, one backup job completes, no HA state changes in 24 h
Abort criteria	Any unplanned fencing; any backup failure; any node failing to rejoin quorum within 10 min of reboot; any Ceph HEALTH_ERR
Rollback	Documented per-node package downgrade path, tested on site-c
Owner	Named person per stage, not a team
End date	The date by which all clusters are on the new version, after which the skew is a finding

The last row is the one that keeps a fleet from acquiring permanent skew. A staged rollout without an end date becomes a fleet with three versions, and the whole argument for staging was that skew is temporary.

Stage the validation cluster on the same shape, not the same size

A three-node validation cluster with local ZFS does not validate a change for a twenty-node Ceph cluster. It cannot reproduce the failure modes you actually care about: backfill behaviour, corosync under load, the interaction between HA and a slow storage layer.

What matters is shape: the same storage type, the same network topology class, the same HA and Ceph usage, the same guest mix in kind if not in count. Fewer nodes is acceptable — different architecture is not, and a validation cluster that is architecturally different gives you the confidence without the coverage.

If the budget only stretches to an architecturally different validation cluster, say so explicitly in the change record: “site-c validates package installation and service restart; it does not validate Ceph behaviour”. That is an honest partial control. Believing it validates everything is worse than not having it.

On-call across a fleet

Three changes to the on-call model, in descending order of how often they are skipped.

1. Every alert names its cluster, first. Not in a label at the bottom, not implied by the node name — in the title. The responder’s first action is opening the right GUI, and an alert that does not say which one costs a minute of confusion at the worst time.

2. Every runbook starts by establishing which cluster. Runbooks written for a single cluster start at step one with a diagnostic command. Fleet runbooks need a step zero: identify the cluster, open its baseline document, note its version and its known differences. Part XXII’s runbook material applies; the addition is small and the omission is expensive.

3. Credentials and access must already exist for every cluster. The responder who can log in to two of three clusters is a responder who will escalate at 3 a.m. for an access problem. Access to every cluster is part of the on-call rotation’s prerequisites, and it should be tested when someone joins the rotation rather than when they are first paged.

Where Proxmox Datacenter Manager fits

PDM (Part XXV) exists for exactly this problem: one place to see inventory, health, tasks and updates across several clusters, without the clusters becoming related to each other.

Two honest boundaries, both covered in that part and worth restating here because this lesson is where the temptation arises.

PDM is not a monitoring system and must not be what pages you. It aggregates collected state, and an unreachable remote shows its last-seen values. Your alerting stack still needs to be independent of it — including, especially, the heartbeat that detects a cluster going quiet.

PDM does not remove the need for the baseline and the drift report. It shows you what each cluster currently is; it does not tell you what each cluster is supposed to be, or which differences are intended.

Used within those boundaries it removes a large amount of daily friction, which is a real benefit and not a small one.

Under the hood

Why is there no “cluster” field in the API responses, and what does that tell you about the design?

Every Proxmox VE API response is scoped to the cluster that answered it, because the API has no concept of a peer. /cluster/resources means “the resources of this cluster”; /nodes means “the nodes of this cluster”. There is no endpoint that returns another cluster’s state, and there is no identifier in the response distinguishing this cluster from any other, because from the API’s point of view there is no other.

That is the same boundary the previous lessons kept running into, seen from a third angle. pmxcfs consistency is defined by corosync membership; remote migration is an API client relationship rather than a federation; and here, the data model itself has no place to put a cluster name.

The operational consequence is a rule worth adopting literally: identity is added by the collector. Whatever queries your clusters — a script, an exporter, Ansible, PDM — is the only component that knows which cluster it is talking to, so it is the only component that can stamp the name. If that stamp is missing at the point of collection, no downstream system can reconstruct it correctly, and every dashboard, alert and inventory row below it inherits the ambiguity.

You can see the boundary directly: pvecm status on a node prints its own cluster’s name and its own membership, and knows nothing whatsoever about the cluster three racks away that you also operate.

What to centralise, and what not to

Centralise	Keep per cluster
Metrics, logs and dashboards, with a `cluster` label	The PVE notification targets, so a cluster can still shout when the aggregator is down
Inventory and CMDB	Backup jobs and windows, which legitimately differ by site
The configuration baseline and drift report	HA rules and groups, which are per-cluster by nature
Change records and the rollout order	Node network configuration, certificates, corosync
On-call rotation, escalation and runbooks	The local console and IPMI path, which must not depend on the central system
Backup verification results	The restore capability itself: each cluster must be restorable without the others

The line running through the right-hand column is independent recoverability. A fleet is only as resilient as its ability to lose the central layer and keep operating. If losing the aggregator means losing alerting, or losing PDM means losing the ability to manage a cluster, you have re-created the single fault domain that splitting was supposed to remove — just one level up.

What happens if the central monitoring or PDM layer is lost?

Test this deliberately, once, before it happens by accident.

Each cluster must remain fully manageable on its own. Its GUI on port 8006, its API, its CLI over SSH and its console over IPMI are all independent of any central system. If an operator cannot manage a cluster without the aggregator, find out why now: usually it is that nobody holds per-cluster credentials any more, which is an access design problem rather than a technical one.

Alerting degrades to per-cluster. This is why the right-hand column above keeps PVE’s own notification targets configured per cluster. A cluster that can still email or webhook on a fencing event, a failed backup or a Ceph health change is a cluster that is not silent while the central layer is rebuilt. It is noisier than the aggregated view and that is the correct trade.

Fleet-level detections stop. Version skew, cross-cluster drift, aggregate capacity and the heartbeat itself are all computed centrally and all stop. Treat the central layer’s own availability as an alert with an owner — the monitor needs a monitor, and the usual answer is a small external check that pages if the aggregator has not reported in.

Migration campaigns pause. Remote migration between two clusters does not technically depend on the central layer, but your record of what has moved probably does. Stopping a campaign at a known point beats continuing without the ledger.

The exercise to run: schedule an hour, stop the central stack, and have the on-call engineer perform a routine task on each cluster — take a backup, migrate a guest between nodes, read the task log. What they cannot do is your list of things to fix.

Key takeaways

/cluster/resources is the one call that returns every node, storage, guest and pool for a cluster, and it is the right basis for a fleet inventory.
No PVE API response names its own cluster. Identity is added by the collector, and inferring it from node names breaks the first time someone names a node inconsistently.
Fleet inventory should run as a privilege-separated PVEAuditor token per cluster, so a leak is disclosure rather than an outage.
Per-cluster alerts and fleet alerts are different classes. The essential fleet alert is a heartbeat per cluster, because absence of signal is invisible to every threshold rule you own.
Stage changes with a written order, a soak time derived from the failure mode, abort criteria authored before you start, a tested rollback, and an end date that stops temporary skew becoming permanent.
A validation cluster must match production’s shape — storage type, topology class, HA and Ceph usage — even if it does not match its size. If it does not, say what it validates and what it does not.
On-call needs the cluster named in the alert title, a step zero in every runbook that identifies the cluster, and credentials for every cluster tested before the first page.
Centralise metrics, inventory, baselines and runbooks. Keep per-cluster notification targets, console access and restore capability, so losing the central layer is a degradation and not an outage.

Knowledge check

Why this matters

At one cluster, operations is about the system. At four, it is about the difference between them — and that is a different job, staffed by the same people, usually without anyone saying so out loud.

The tell is the shape of incidents. Single-cluster incidents are “the storage is slow” or “a node is down”. Fleet incidents are “why did this work at site A”, “which version is site C on”, “who has access to site B”, and “when did that change land there”. None of those are technical questions about Proxmox. They are questions about whether the estate is described anywhere.

Everything in this lesson is an answer to one of them: the inventory answers where, the baseline answers what, the change record answers when, and the on-call model answers who. The tooling that produces them can be modest — a script, a table, a page in a repository — and it will still outperform an excellent monitoring stack that cannot tell you which cluster it is looking at.