Metrics and notifications: two subsystems, one confusion · Proxmox VE

Why this matters in production

Proxmox has two entirely separate mechanisms for telling you about itself, and a great many clusters have configured one and believe they have both.

	Metrics	Notifications
Answers	How much? For how long? Is it trending?	Something happened, and you should know
Mechanism	`pvestatd` pushes samples to an external server	Events are matched and delivered to a target
Configuration	`/etc/pve/status.cfg`	`/etc/pve/notifications.cfg` (secrets in `priv/`)
Data shape	Time series, continuous	Discrete events, with a type and a severity
Fails by	Going quiet	Discarding silently
Covers	Node, guest and storage statistics	Backups, replication, fencing, updates, mail to root

The overlap between those two columns is zero. A backup failure never appears in a metric. Memory usage never generates a notification. Configuring an InfluxDB export and building a Grafana dashboard gives you a beautiful view of resource consumption and no knowledge whatsoever that last night’s backup failed.

Both subsystems are covered in depth later in this part. This lesson is the map: which one you are looking at, what it is for, and the decisions that belong at this level.

The metric path

pvestatd on every node collects statistics and pushes them to the servers defined in /etc/pve/status.cfg. There is no endpoint to scrape and nothing to install on the node.

PVE 9 supports three backends:

Backend	Transport	Notes
Graphite	UDP (default) or TCP, port 2003	Metric path prefix defaults to `proxmox`
InfluxDB	UDP, HTTP or HTTPS via `influxdbproto`	v2 API needs an organisation, bucket and token
OpenTelemetry	OTLP/HTTP	New in PVE 9; JSON encoding only

Read-only / Safewhat is currently configured, on a cluster you have inherited— Read-only. Worth running first on any cluster you did not build - the frequent answer is that a metric server was configured years ago and has been pointing at a decommissioned host ever since.

set -euo pipefail

# Metric servers. status.cfg is replicated, so this is cluster-wide.
pvesh get /cluster/metrics/server --output-format yaml

# Notification targets and matchers. Separate subsystem, separate file.
pvesh get /cluster/notifications/matchers --output-format yaml

# Is pvestatd complaining? For an HTTP export this is where a failure appears,
# and it appears per node, so check each one.
journalctl -u pvestatd --since '1 hour ago' --no-pager \
| grep -Ei 'metric|influx|graphite|otel' || echo 'no metric errors logged'

Push export against pull-based scraping

Both are in production use and the choice is architectural.

	Native push	`pve-exporter` and Prometheus
Who initiates	Each node, outbound	Prometheus, inbound to the management network
To install	Nothing	An exporter service, with a PVE API token
Failure visibility	Poor on UDP, acceptable on HTTP	Excellent — a failed scrape is data
Data source	`pvestatd`, the same figures the GUI plots	The PVE API
Query language	Whatever the backend offers	PromQL

The deciding argument is usually up is a metric. Prometheus records a failed scrape, so “we stopped receiving from pve-03” is an alertable condition. With push, absence of data is absence of data, and building an alert on it requires deliberate work on the receiving side.

Do not pip install an exporter into the system Python on Debian 13

pve-exporter is a Python application and the instinct is pip install prometheus-pve-exporter. On Debian 13 (Trixie), which PVE 9 is based on, that fails: the system Python is marked externally managed, and pip refuses to write into it.

Forcing past that with --break-system-packages does what the flag name says. It installs into the same site-packages that apt manages, and the next apt upgrade of a Python package can leave you with a hypervisor whose pvestatd, pveproxy or pvedaemon will not start — because those are Perl, but the toolchain around them is not entirely.

Install the exporter into a virtual environment, in a container, or on a separate host. The last option is the right one: the exporter does not need to run on a PVE node at all, it needs an API token and network reachability, and running it off-cluster means the thing that scrapes your hypervisors is not itself a hypervisor workload.

The notification path

Since PVE 8.1 the notification system is a first-class subsystem with two object types:

A target is a destination: sendmail, smtp, gotify or webhook.
A matcher decides which notifications reach which targets, using rules on metadata fields, severity and calendar.

The property that produces most of the trouble: a notification that matches no matcher is discarded. Silently. Delivery is opt-in.

The events a PVE cluster emits are a short and finite list:

Event	`type`	Severity	Metadata
Backup succeeded	`vzdump`	`info`	`hostname`, `job-id`
Backup failed	`vzdump`	`error`	`hostname`, `job-id`
Storage replication failed	`replication`	`error`	`hostname`, `job-id`
Cluster node fenced	`fencing`	`error`	`hostname`
System updates available	`package-updates`	`info`	`hostname`
Mail addressed to root	`system-mail`	`unknown`	`hostname`

Anything not in that list — a full datastore, a degraded ZFS pool, an OSD that has been down for a week — reaches you only if something else generates mail to root, or if an external monitoring system is watching.

The event list is short, and most of what you care about is not on it

This is the fact that determines how much external monitoring you need.

PVE notifies about backups, replication, fencing, available updates and mail addressed to root. It does not notify about a datastore filling up, a Ceph health warning, a degraded ZFS pool, a guest that has stopped, a node running out of memory, or a certificate about to expire.

Several of those do reach you indirectly, through system-mail: smartd, mdadm, ZFS event daemon and cron all mail root when they are unhappy, and that mail becomes a system-mail notification. But it arrives with severity unknown, so a matcher written as match-severity error — the obvious first matcher anybody writes — excludes every one of them.

Two consequences worth acting on now:

Any severity-based safety-net matcher must read warning,error,unknown.
Everything not on that list is the job of your monitoring system, which is the honest reason a Proxmox estate needs one.

Configuration changea target and the matcher that makes it do anything— Configuration change. The second command is the one people omit. A target with no matcher routing to it will test successfully from the GUI and deliver nothing in production.

set -euo pipefail

# 1. The target. mode is insecure, starttls or tls; default tls.
pvesh create /cluster/notifications/endpoints/smtp \
--name relay-ops \
--server smtp.example.com \
--port 587 \
--mode starttls \
--username 'pve-notify@example.com' \
--password 'REPLACE_ME' \
--from-address 'pve-notify@example.com' \
--mailto 'ops@example.com'

# 2. The matcher. Field rules take the form exact:<field>=<value> or
#    regex:<field>=<pattern>; mode all requires every rule to match.
pvesh create /cluster/notifications/matchers \
--name backup-failures \
--mode all \
--match-field 'exact:type=vzdump' \
--match-severity error \
--target relay-ops

# 3. Confirm both exist.
pvesh get /cluster/notifications/endpoints/smtp --output-format yaml
pvesh get /cluster/notifications/matchers      --output-format yaml

Why secrets live in a second file, and what that means for restores

Notification configuration is split across two files inside pmxcfs:

/etc/pve/notifications.cfg — targets and matchers, readable by anyone with audit permission on the notification configuration.
/etc/pve/priv/notifications.cfg — SMTP passwords, Gotify tokens and webhook secrets, readable by root only.

That split is why a webhook body can reference {{ secrets.token }} and be rendered with the real value at delivery time without the token ever appearing in the readable file.

The operational consequence shows up during a restore or a migration. A /etc/pve backup that captured the main tree but not priv/ — or a configuration exported from one cluster and imported into another — produces targets that exist, matchers that route to them, and delivery that fails authorisation. The notification is lost and the symptom is silence.

After any restore, re-create the secrets explicitly and test each target.

The metric path differs here and it is worth knowing: status.cfg holds the InfluxDB token inline, in the readable file. Scope that token to write access on one bucket, and treat any backup containing status.cfg as credential-bearing.

PBS has its own, configured separately

Proxmox Backup Server implements the same target and matcher model with its own configuration, reached in the PBS interface under Configuration → Notifications. Its metric server is configured separately again, under Configuration → Metric Server.

Two things follow, and both are easy to get wrong:

A matcher on the PVE cluster does not route PBS notifications. The two systems share a design and share no configuration. “Notifications are configured” has to be answered twice.
The PBS metric export carries host and datastore statistics only. It does not carry job outcomes, so a PBS dashboard cannot tell you whether a verification job found a corrupt chunk. That comes from the notification side.

What happens if you configure metrics and skip notifications?

You get the failure mode this whole lesson exists to prevent, and it is comfortable enough to persist for years.

The dashboard is genuinely good. CPU, memory, storage and network are graphed per node and per guest, alert rules fire on thresholds, and everyone agrees the platform is well monitored. Capacity conversations are evidence-based. Nobody is wrong about any of that.

Then a backup job starts failing for one guest — a broken guest agent, a storage that was renamed, a snapshot that will not complete. The job reports error. The notification is generated, matches no matcher, and is discarded. The metric path is unaffected, because a failed backup is not a number: the datastore keeps growing at roughly its usual rate as other guests continue backing up successfully, so nothing on the dashboard looks different.

Six weeks later somebody needs a restore for that guest, and the newest snapshot is from before the failure began. Retention has since expired the last good one.

The specific defence is small: a freshness check per backup group. Not “did a job fail”, which the notification system covers, but “when was the newest snapshot for each guest, and is that within the interval I expect”. That question catches both the job that failed and the job that stopped being scheduled, and it is answerable from the PBS side with a scheduled script.

Choosing what to build

Estate	Reasonable configuration
Home lab, one node	Built-in graphs, plus a `sendmail` or `gotify` target with a broad matcher
Small production, one cluster	Built-in graphs, notification targets with per-type matchers, and a backup-freshness check
Multiple clusters or a capacity story to defend	Add an external metric server, external probes for service availability, and alerting on the receiving side
Regulated or contractual availability	All of the above, plus availability measurement from outside the cluster and a heartbeat proving the alert path works

The order matters. Notifications before metrics, every time. A cluster that pages a human when a backup fails and has no dashboards is in much better shape than one with beautiful dashboards and an unrouted notification system, and it takes an afternoon rather than a project.

Common mistakes

Building dashboards and calling it monitoring. Metrics answer how much, never something happened.
Configuring a target with no matcher. It tests successfully and delivers nothing.
A match-severity error safety net, which excludes unknown and therefore excludes SMART, mdadm and cron failures.
Assuming PVE notifies about storage or Ceph health. It does not.
pip install into the system Python on a PVE 9 node.
Configuring PVE notifications and forgetting PBS, where the events that matter most are generated.
Leaving a metric export on UDP and having no way to tell whether it works.
Restoring /etc/pve without priv/ and leaving every target unable to authenticate.

Key takeaways

Two subsystems, two configuration files, no overlap: status.cfg for metrics, notifications.cfg for events.
pvestatd pushes; nothing scrapes PVE natively. Prometheus needs pve-exporter, and it should not run on a PVE node.
Notifications are opt-in through matchers, and an unmatched notification is discarded silently.
The PVE event list is short — backups, replication, fencing, updates, mail to root. Everything else is your monitoring system’s job.
PBS is configured separately for both, and its metric export carries no job outcomes.
Configure notifications first.