The cross-cluster view: inventory, health, and what is actually live · Proxmox VE

This is the lesson where PDM earns its keep. Four clusters, six hundred guests, two backup servers, and one page that tells you the state of all of it.

It is also the lesson where a page that looks authoritative can be wrong in a way that only matters once. A dashboard tile reading “42 running” is a fact about something, and knowing which something — the cluster right now, or the last successful collection run, or the last time PDM could reach that remote at all — is the difference between a useful control plane and a confident liar.

What PDM collects

Collecting data is described in the documentation as a primary function of PDM. The material it gathers from each remote is:

RRD metrics — the round-robin time series behind every graph and gauge, for nodes, guests, storages and PBS datastores, and since 1.1 for the PDM host itself.
Worker task status and logs — which is what makes a cross-estate task log possible.
The resource inventory — nodes, VMs, containers, storages, datastores, SDN zones, and their status.
Update and repository status — what is pending on every remote.
Subscription status — per node, across the fleet.
Ceph cluster state — health, capacity, monitors, managers, OSDs, pools, CephFS, flags — for hyper-converged PVE remotes.

That is a substantial amount of information to hold about a fleet, and holding it is what allows the dashboard to answer a question instantly instead of fanning out to twelve clusters every time somebody loads a page.

Holding it is also exactly why the data can be stale.

Collected versus live: the distinction that matters

Draw a line down the middle of PDM’s behaviour.

On the collected side sit the dashboard, the Views and their widgets, the RRD graphs, the cross-remote guest list, the update overview, the subscription status and the aggregated task list. These are served from PDM’s own store, populated by collection runs.

On the live side sit the actions. Starting a guest, shutting one down, taking a snapshot, rolling one back, migrating, opening a shell on a node, applying updates — each of these is an API call issued to the remote at the moment you click, and it either succeeds against the current state of that cluster or it fails.

This produces the single most useful mental model in the whole part:

The last-seen problem

Here is the property that deserves the most attention, because it is deliberate, sensible, and quietly dangerous.

PDM caches the resource list and its usage metrics in order to provide a quick overview — and to provide the last-seen state for offline or unresponsive remotes. That behaviour has been in the product since the first alpha and it is the correct design choice: a remote that is temporarily unreachable should not blank out your entire inventory, and knowing what a cluster looked like before it went away is genuinely useful during an incident.

But it means a screen full of green tiles is not, on its own, evidence that anything is currently up. It is evidence that PDM either collected recently, or could not collect and is showing you what it last saw.

PDM does distinguish the two. Remote status is surfaced on the dashboard and in the remotes widget, the map widget colours its markers by remote status, and a remote whose connection is failing is offered the certificate re-pin dialog directly. The information is there. The failure mode is human: an operator glances at a familiar dashboard, sees the shape they expect, and reads “everything is fine” from a page that was actually saying “here is what I remember”.

Reading a PDM dashboard during an incident

Four questions, in this order, before you believe any number on the page.

Is the remote currently reachable? Check remote status first, not resource status. A green guest inside a red remote is a memory.
When did collection last succeed? Use the metric-collection status command below rather than inferring it from the graphs — a flat line and a stale line look identical.
Does the aggregate include what I think it includes? Capacity figures sum across remotes, and summation is where subtle bugs live. PDM 1.1 fixed an issue where a shared storage in a cluster was counted multiple times toward storage capacity. Treat cross-cluster totals as indicative and derive anything you will act on from the owning cluster.
Am I about to act on the photograph? If the action matters, confirm the current state on the remote first. The action is live even when your reason for taking it was not.

Read-only / Safewhen did PDM last actually talk to the fleet— Shows metric collection status. Read-only. Run on the PDM host, or against it with the client from elsewhere.

# proxmox-datacenter-manager-client metric-collection status

Configuration changeforce a collection run before you trust the page— Triggers collection for one remote, or for all remotes when --remote is omitted. It reads from the remote and writes to PDM state; it changes nothing on the cluster.

# one remote
proxmox-datacenter-manager-client metric-collection trigger --remote site-a

# the whole fleet
proxmox-datacenter-manager-client metric-collection trigger

The collection interval is not documented — measure it, do not assume it

Searching the 1.1.7 documentation for a polling interval, a cache lifetime or a staleness threshold returns nothing. The roadmap and changelogs describe the collection machinery in detail — concurrent collection, persisted state reused across restarts, delayed tasks being skipped rather than queued, an immediate collection triggered when a remote is added — but no published number.

This course will not invent one. If your runbook needs to say how old a PDM figure can be, establish it on your own installation with metric-collection status over a period, and record the observed value with the PDM version beside it.

Two behaviours from the changelog are worth knowing regardless. Collection runs concurrently across remotes, so one slow remote does not serialise the rest. And a collection task that is delayed is skipped, not queued behind the next one — which means a remote that is intermittently slow can be collected less often than the nominal interval without anything looking broken.

The guest list

The Guests entry in the sidebar is a single cross-remote list of every QEMU virtual machine and LXC container PDM knows about, collected from all configured PVE remotes.

Two presentations of the same data. List is the default: a flat sortable table with guest name, ID, status, remote, node, tags, CPU usage, memory usage, uptime, and the per-guest actions. Tree groups the guests by remote, each remote a collapsible parent with its guest count, dropping the now-redundant remote column.

The filter box is more capable than it looks. A bare word matches as a substring against every visible column. A term qualified with field:value restricts it to one column, and the supported qualifiers are:

Qualifier	Matches
`tag:`	a guest tag
`remote:`	the remote name
`node:`	the node name
`status:`	the run status, such as `running` or `stopped`
`type:`	the guest type, `qemu` or `lxc`

Multiple terms combine with a logical AND, separated by spaces, and matching is case-insensitive. So tag:prod status:running matches only running guests carrying the prod tag, across every cluster. This is the same qualifier syntax as the global search field in the header, so learning it once pays twice.

Tag discipline is what makes this useful

tag:prod status:running type:qemu is a genuinely powerful query across a fleet of six hundred guests — and it is worth precisely as much as your tagging discipline.

PVE guest tags are shown in PDM and filterable in PDM, but they are set on the cluster. If site-A tags production workloads prod and site-B tags them production, the cross-cluster query silently returns half the answer. There is no warning, because there is nothing wrong: both tags exist and both match themselves.

Part XXII’s inventory and tagging material becomes materially more valuable the moment PDM arrives, because a tag vocabulary that was a nicety within one cluster becomes the join key across all of them. Agree the vocabulary, write it down, and audit it — a query that returns a plausible-looking wrong answer is worse than one that returns nothing.

Note also what the guest list shows: the documentation states that it shows the resources the user may audit. Guests on remotes the operator has no audit privilege on are simply absent — not greyed out, not counted, absent. Two operators looking at “the guest list” may be looking at different fleets, which is a feature for delegation and a trap for a conversation about totals.

Views: dashboards you define

The default dashboard covers everything. Views let you build dashboards that cover something specific, and they are the most under-appreciated feature in the product.

A view selects resources with an include/exclude filter system: you define what to consider for inclusion, and then the exclude list removes types back out again. The documented example is the shape to remember — include all virtual machine resources, then exclude any that reside on a specific remote.

The available filter types:

Filter	Selects by
`resource-type`	`datastore`, `lxc`, `node`, `qemu`, `sdn-zone`, `storage`
`resource-pool`	a PVE resource pool name
`tag`	a tag name
`remote`	a remote name
`resource`	a specific resource id

Each filter takes an optional match-behaviour prefix. Currently only exact is available, and it is the default when no prefix is given — so the wildcard syntax you may want is a roadmap item (“wildcards for view filter values”), not a feature.

Views are stored in /etc/proxmox-datacenter-manager/views.cfg with include, exclude, include-all and a JSON-encoded layout, which means they are version-controllable if you want them to be.

The widgets a view can be built from:

nodes — status overview of PVE and PBS nodes, optionally limited to one remote type
guests — status overview of guests, optionally limited to QEMU or LXC
pbs-datastores — usage and status of PBS datastores
remotes — configured remotes and their status, and can offer the add-remote wizard
subscription — subscription status of the remotes
sdn — status of SDN zones
leaderboard — ranks resources by a metric such as guest or node CPU, or node memory, and lists the top consumers
task-summary — recent tasks grouped by a chosen criterion
resource-tree — the selected resources as a hierarchy
node-resource-gauge — one node resource as a gauge chart
map — remotes plotted on an interactive world map

The delegation property: a view can be granted without the remotes

This is the feature to remember from the whole lesson.

Users can be granted access to specific views without being granted access to the underlying remotes or resources directly. Permissions are assigned on the /views/{id} path, separately from /resource/{id}.

That solves a real problem cleanly. A NOC that must see whether anything is on fire, an application team that wants to watch its own guests, a manager who wants a capacity trend — none of them need an account that can reach a cluster. Build a view, filter it to what they should see, grant on the view, and stop.

It is also the one place in PDM where the blast-radius arithmetic goes in the right direction: instead of concentrating authority, a view distributes visibility without authority. Use it deliberately, and prefer it every time somebody asks for “just read-only access” to the estate.

The map widget has one wrinkle worth knowing before somebody files a bug: the locations come from the remotes and cannot be edited in PDM. Set them on the remote — on PVE as a cluster-wide default in the datacenter options, optionally overridden per node in the node options; on PBS under Configuration, in the Other tab’s Location section. The map background is public-domain vector data from Natural Earth, shipped in the proxmox-geojson-data package, so the widget works without any external service.

The other aggregated panels

Tasks. The Remotes area has a Tasks tab giving an overview of all tasks across the entire datacenter, filterable, and the task viewer can be refreshed for remote tasks. This is the panel that makes an incident timeline assemblable in one place, and it is the strongest single argument for PDM in an estate with more than two clusters.

Updates. A panel across all remotes showing repository status and pending updates, with the ability to apply updates from PDM — which it accomplishes by using the remote shell capability, so it needs PVE 9.1 or PBS 4.1 on the target.

Ceph. One row per detected cluster: health with the number of active health checks in parentheses, name and backing remote, raw capacity and available, a threshold-coloured usage meter, OSD counts phrased as up and in out of total, monitor quorum as in-quorum out of total, and an activity string such as “Near full”, “Degraded” or “Recovering”. Read-only, as Lesson 1 established.

Firewall. A list of configured firewall rules and settings per PVE remote cluster, its nodes and its guests. A view, not an editor.

SDN. A status overview showing available, error or unknown for zones across all remotes, plus cross-remote EVPN configuration.

Under the hood

Why does the aggregate view get capacity arithmetic wrong more often than it gets anything else wrong?

Because “total storage across the estate” is not a well-defined quantity, and every implementation has to choose an interpretation.

A shared storage in a PVE cluster is visible from every node in that cluster. Naively walking nodes and summing their visible storages counts a single 100 TB Ceph pool five times on a five-node cluster. PDM 1.1 fixed exactly this: the changelog records a fix for shared storage in a cluster being counted multiple times toward the storage capacity calculations.

The general lesson survives the specific fix. Cross-cluster aggregation involves deduplication decisions — shared versus local storage, thin provisioning, Ceph raw capacity versus usable capacity after replication, PBS deduplicated size versus logical size — and each decision is defensible and produces a different number. Two tools can both be right and disagree.

So use PDM’s aggregates the way they are meant to be used: to notice, to compare, to spot the outlier. When you are sizing a purchase or answering a capacity question that someone will hold you to, derive the number from the owning cluster with the method Part XVII teaches, and cite that.

What happens if you build alerting on PDM?

You build something that cannot page you, on a platform that does not claim it can.

Three verifiable reasons, all from the roadmap — meaning they are absent today, not merely undocumented:

PDM has no notification system for remotes. “Evaluate whether Proxmox Datacenter Manager can act as a notification target for remotes” is a roadmap item. It is being evaluated.
PDM does not even notify about itself. “Standard system, update and task notifications for the Proxmox Datacenter Manager node itself” is also a roadmap item. The PDM host will not tell you its own disk filled up.
PDM has no high availability. Active-standby is under evaluation, with two side-by-side instances as the current workaround. A monitoring system that is itself a single point of failure is not a monitoring system.

Add the two properties from earlier in this lesson — an undocumented collection interval, and a last-seen state that renders an unreachable remote as the inventory it used to have — and the conclusion is not a close call.

The correct architecture is the one Part XVI already describes: metrics to a time-series database from each cluster, alerting rules with defined thresholds and defined staleness handling, notifications configured on each PVE and PBS instance for the things those instances know about, and PDM as the human interface you open after the page arrives, to see the shape of the estate around the alert.

PDM is where you look. It is not what wakes you up.

Key takeaways

PDM collects RRD metrics, task status and logs, the resource inventory, update and subscription status, and Ceph and SDN state from every remote.
Everything you see is collected and cached; everything you do is issued live against the remote. They agree until they matter.
An unreachable remote shows its last-seen state by design. Check remote status before you read resource status.
The collection interval is not published. Measure it with metric-collection status, and note that delayed collection tasks are skipped rather than queued.
The guest list supports tag:, remote:, node:, status: and type: qualifiers combined with AND — worth exactly as much as your cross-cluster tag discipline.
Views can be granted to users who have no access to the underlying remotes. This is the cleanest delegation tool in the product.
Cross-cluster capacity aggregates are indicative. PDM 1.1 fixed shared storage being counted several times, and aggregation has more decisions like that in it.
PDM cannot alert. Notifications for remotes and for PDM itself are both roadmap items.

Knowledge check

Why this matters

Every operator who has run more than one cluster has wanted this page. Being able to answer “where is that VM”, “which cluster is short of memory” and “what happened at 02:14 across the estate” without opening four tabs is a genuine change in how the job feels.

The reason this lesson spends as much time on staleness as on features is that the page is persuasive. A well-laid-out dashboard with green tiles and smooth graphs does not feel like a cache. It feels like the truth, and it feels that way most strongly to the person who has been looking at it every day for six months and knows exactly what normal looks like.

The habit worth building is small and costs nothing: before you conclude anything from a PDM screen, look at remote status first. Not resource status. Remote status. Everything downstream of a reachable remote is roughly as fresh as the last collection run; everything downstream of an unreachable one is a memory wearing the same colours.

That single reflex is what separates using a control plane from trusting one.