What to monitor across the Proxmox stack · Proxmox VE

Why this matters in production

You cannot operate what you cannot see. Monitoring that misses real incidents is as bad as no monitoring; monitoring that floods the team with noise is ignored.

The four observability axes

Axis	What	When
Metrics	Numeric measurements over time	For trends and capacity
Logs	Discrete events with context	For forensics
Events	State changes	For alerting
Alerts	Derived from the above	For human attention

A node being up is not a guest being healthy

Between “the hypervisor is powered on” and “the user can do their work” there are five distinct states. Four of them can be perfectly true while the service is completely unavailable, and every one of them is what somebody somewhere is using as their availability measurement.

Level	What it proves	What can still be broken
1. Node up	The hypervisor booted and is reachable	Every single thing above it
2. Guest `running`	A QEMU or LXC process exists	Guest kernel panic, no network, filesystem read-only, application dead
3. Guest agent responds	The guest kernel is alive and scheduling	The application, its dependencies, its data
4. Port is open	Something is listening on the socket	The application returning errors to every request
5. Endpoint returns the right answer, fast enough	What the user experiences	Only genuinely user-specific problems

Each level is a necessary condition for the one below it and none is sufficient. A node with 400 days of uptime running a guest that has been serving HTTP 502 since lunchtime satisfies levels one through four.

Every layer you monitor below level 5 is a proxy, and proxies fail silently in the same direction

The pattern is consistent and worth stating plainly: infrastructure monitoring fails optimistic. When a proxy metric is wrong, it is almost always wrong in the direction of reporting health.

A guest that has kernel-panicked still shows running in qm list, because the QEMU process is alive and the panic is inside it. A guest whose root filesystem has gone read-only still answers the guest agent. A web application returning 500 to every request still has an open port.

None of those produce a red indicator anywhere in Proxmox, because Proxmox is not wrong about anything — it is accurately reporting the layer it can see.

Which means an availability report built from node uptime or guest state is not a conservative estimate of availability. It is an upper bound that will always look better than reality, and the gap between them is invisible until somebody complains.

At least one signal per service has to come from level 5, measured from where a user is, and that is a probe you build rather than a metric Proxmox exposes.

Read-only / Safefind the guests where levels 2 and 3 disagree— Read-only. A guest that is running with an agent configured but not answering is at level 2 pretending to be at level 3, and its backups are quietly weaker than your restore plan assumes.

set -euo pipefail

for vmid in $(qm list | awk '$3 == "running" {print $1}'); do
configured=$(qm config "$vmid" | awk -F'[ ,=]' '/^agent:/ {print $2}')
if [ "$configured" != "1" ]; then
  printf '%-6s L2 only  (no agent configured)\n' "$vmid"
elif qm guest cmd "$vmid" ping >/dev/null 2>&1; then
  printf '%-6s L3 ok\n' "$vmid"
else
  printf '%-6s L2 only  (agent configured, NOT answering)\n' "$vmid"
fi
done

What to monitor

Hosts

Metric	Why
CPU utilisation (host + per-VM)	Capacity planning
Memory pressure (`/proc/pressure/memory`)	OOM risk
Disk I/O (await, utilisation)	Storage bottlenecks
Network errors and drops	Hardware failures
SMART / NVMe health	Disk failures
Temperature	Hardware failures
NTP offset	Cluster consistency

Cluster

Metric	Why
Quorum state	Cluster health
Corosync latency	Network health
pmxcfs sync status	Configuration consistency
HA state per resource	HA health

Storage (ZFS)

Metric	Why
Pool capacity	Running out
Pool fragmentation	Performance
Scrub status	Data integrity
Disk error counters	Hardware

Storage (Ceph)

Metric	Why
Cluster status (HEALTH_OK / WARN / CRIT)	Health
OSD latency p95, p99	Performance
PG states (active+clean vs degraded)	Recovery
Pool utilisation	Capacity
nearfull warnings	Pre-emptive

VMs and containers

Metric	Why
CPU steal time	Host contention
Balloon / memory pressure	Memory overcommit
Disk I/O await	Storage issues
Guest agent status	Snapshot consistency

Backups (PBS)

Metric	Why
Last backup age per VM	RPO compliance
Last verify status	Data integrity
Datastore capacity	Storage
Sync job status	Off-site replication

Security

Metric	Why
Failed login attempts	Brute force
Configuration changes	Audit
New API tokens	Audit
Outbound connections to suspicious IPs	Exfiltration

Alert design

flowchart LR
  A[Metric source] --> B[Threshold check]
  B --> C[Above threshold?]
  C -->|yes| D[Alert]
  C -->|no| E[No action]
  D --> F[Notification]

Avoid alert storms by:

Grouping related alerts.
Using rate-of-change alerts instead of absolute thresholds.
Suppressing alerts when a known incident is in progress.

Sort every signal into page, ticket or dashboard

Deciding what to monitor is the easy half. The half that determines whether the on-call rota is survivable is deciding what each signal is allowed to do.

Destination	Criterion	Examples
Page	A human can act now, and waiting makes it materially worse	Tier-0 service unreachable; quorum lost; Ceph writes blocked; second OSD down in a failure domain; PBS datastore above 90%
Ticket with a deadline	Days of runway, but it will not fix itself	Single OSD down and recovering; OSD nearfull; deep scrubs behind; certificate expiring in 30 days; a guest whose agent stopped answering
Dashboard only	Useful for understanding, never for waking someone	CPU utilisation; network throughput; memory usage; guest counts

The exercise worth doing once, deliberately, is taking every alert you currently have and assigning it to one of those three. Most estates discover that a substantial share of their pages belong in the second column and a few belong in the third, and moving them is the single largest improvement available to on-call quality.

Leading indicators: the signals that give you time

Not all signals arrive with the same amount of warning, and the ones worth building first are those that fire while you can still do something cheap.

Signal	Warning it gives	What it precedes
SMART reallocated and pending sector counts rising	Weeks	A disk failure
Ceph OSD utilisation spread widening	Weeks	The fullest OSD crossing nearfull, then backfillfull
Backup duration growing run over run	Weeks	A backup that no longer fits its window
PBS datastore growth rate against free space	Weeks	A datastore that cannot prune because it is full
Corosync retransmit counters climbing	Days	A link failure and a fencing event
ZFS pool fragmentation rising past ~50%	Months	Write performance degradation with no obvious cause
Deep scrub age exceeding the interval	Months	Corruption detected outside your backup retention window
I/O pressure rising on a node	Hours	Guests becoming visibly slow
Node CPU at 90%	None	Usually nothing at all

The bottom row is the one worth arguing about. Utilisation is the most commonly alerted metric in existence and it is nearly the worst leading indicator available, because a node running at 90% might be perfectly healthy and a node at 40% might be thrashing. Pressure — the proportion of time tasks spent stalled waiting for a resource, which PVE 9 exposes natively per node and per guest — answers the question utilisation was being used as a proxy for.

The pattern across the table: rate-of-change beats absolute value. A datastore at 60% tells you very little. A datastore at 60% that was at 45% three weeks ago tells you when it will be full, which is the number you can act on.

Production considerations

What happens if the monitoring runs on the cluster it monitors?

It reports perfect health during every outage, and the reports are archived and believed.

The arrangement arrives honestly. The cluster is the compute you have, so Prometheus, Grafana, the log store and the probes all become guests on it. Everything works, the dashboards are good, and the design flaw is invisible because it only manifests during the event the system exists to observe.

Then the cluster has a bad hour. The monitoring guests are on it, so they stop too. No samples are collected, so no failures are recorded, so the availability calculation sees a gap rather than downtime — and a gap in a time series is usually rendered as nothing at all. The monthly report shows 100%.

Worse than the wrong number is the missing forensics. The one hour you most need metrics and logs for is the hour you have none, because the collector was inside the failure.

What is needed is not a second monitoring stack. It is a small amount of observation from outside:

One external probe per user-facing service, from a host that does not depend on the cluster — another site, a small VPS, a hosted synthetic monitoring service. This is what makes the availability number honest.
Log shipping off-cluster, so the forensics survive the event.
A heartbeat, so that silence from the cluster is itself an alarm rather than an absence of alarms.

None of that is expensive, and the test for whether you have it is a single question: if the entire cluster went dark right now, what would tell you? If the honest answer is a user, the monitoring is inside the thing it watches.

Common mistakes

Measuring at level 1 or 2 and calling it service availability.
Alerting on every metric, which produces fatigue and then ignored pages.
Alerting on utilisation rather than on pressure or saturation.
Absolute thresholds where a rate of change would give weeks of warning.
Running the monitoring inside the cluster it monitors.
Watching causes and not symptoms, so a real outage with normal-looking causes is invisible.
Never re-sorting existing alerts into page, ticket and dashboard.

Key takeaways

There are five levels between a node being up and a service working; four of them can be true while the user sees nothing.
Infrastructure monitoring fails optimistic, so at least one signal per service must come from a probe outside the cluster.
Monitor the stack: hosts, cluster, storage, guests, backups, security.
Assign every signal to page, ticket or dashboard, and re-do the exercise when the pager gets noisy.
Prefer leading indicators and rate of change; utilisation is a poor predictor and pressure is a good one.
If the cluster went dark, something outside it has to be what tells you.

Knowledge check

Why this matters

Most Proxmox estates monitor the layer that is easiest to monitor, which is the hypervisor, and then reason about service health from it.

That works until it does not, and the way it stops working is specific: infrastructure signals fail optimistic. The node is up, the guest is running, the port is open, and the service has been broken for two hours. Nothing in the platform is lying — it is answering a narrower question than the one being asked of it.

The correction is not more instrumentation. It is one probe per service at the level a user occupies, run from somewhere that survives the cluster, plus the discipline of sorting everything else into page, ticket or dashboard so that the pager stays meaningful.

And when you do add instrumentation, favour the signals that arrive early. Watching a disk’s sector counts climb for three weeks is a maintenance window. Watching CPU cross 90% is a notification about nothing.