The PVE notification system: targets and matchers · Proxmox VE

Every other lesson in this part is about collecting information — metrics, logs, exporters, dashboards. This one is about the last hop, the one where something a computer noticed becomes something a person knows. In Proxmox VE that hop has exactly one mechanism, and since PVE 8.1 it is a first-class subsystem with its own configuration, its own permissions and its own failure modes.

It has two objects, and understanding the relationship between them is most of the lesson:

A target is a place a notification can be delivered to. Four types: sendmail, smtp, gotify, webhook.
A matcher decides which notifications go to which targets. It holds rules and a list of targets.

The critical property, and the one that produces silent failures: a notification that matches no matcher is discarded. Not queued, not logged as undelivered, not surfaced anywhere. Delivery is opt-in through matchers, and a target with no matcher pointing at it is decoration.

What generates a notification

The set is small and finite, which is both a strength and a limitation. These are the events PVE emits, with the metadata each carries:

Event	`type`	Severity	Metadata fields
System updates available	`package-updates`	`info`	`hostname`
Cluster node fenced	`fencing`	`error`	`hostname`
Storage replication failed	`replication`	`error`	`hostname`, `job-id`
Backup succeeded	`vzdump`	`info`	`hostname`, `job-id`
Backup failed	`vzdump`	`error`	`hostname`, `job-id`
Mail addressed to root	`system-mail`	`unknown`	`hostname`

Two things to notice immediately.

job-id is only populated for scheduled backups. A backup you trigger by hand does not carry one. That matters because a matcher rule against a field that does not exist on a given notification fails to match — so a matcher keyed on job-id will silently ignore every manual backup.

system-mail is the catch-all, and it is more important than it looks. Local daemons that mail root — smartd reporting a failing disk, mdadm reporting a degraded array, cron reporting a failed job — become system-mail notifications with severity unknown. If your matchers key on match-severity error, none of them will ever match a system-mail notification, and you will never hear about the failing disk.

A severity-only matcher will miss your failing disk

This is the trap worth naming before anything else, because it looks like good practice.

The obvious first matcher is “send me errors”:

match-severity error

It catches fencing, failed replication and failed backups. It does not catch package-updates (severity info), and it does not catch system-mail, whose severity is unknown — which is where SMART warnings, degraded RAID and failed cron jobs arrive.

The failing disk that smartd has been reporting for three weeks generates a notification every day. Every one of them is discarded, because unknown is not error. There is no error message about this anywhere. The system is doing precisely what it was configured to do.

Include unknown in any severity-based matcher you intend to be a safety net: match-severity warning,error,unknown.

The four target types, and how to choose

Type	Delivery path	Retries on failure	Use it when
`sendmail`	Hands the mail to the local MTA (Postfix)	Yes — the MTA queues and retries	Mail is your channel and the node has a working relay configured
`smtp`	Talks to an SMTP relay directly, no local MTA	No — no queueing, no retry	You want mail without configuring Postfix on every node, and you accept that a relay outage means the notification is gone
`gotify`	Push to a self-hosted Gotify server	No	You want a phone notification without an email round trip
`webhook`	HTTP request to an arbitrary URL	No	Integrating with Slack, Teams, Opsgenie, PagerDuty, or your own handler

sendmail queues, smtp does not

This is the difference that decides which one you should use, and it is easy to miss because the GUI presents them side by side as two ways to send mail.

sendmail hands the message to Postfix. If the relay is unreachable, Postfix holds the message in its queue and retries for days. The notification is delayed, not lost.

smtp opens a connection to the relay itself. The documentation is explicit that it lacks queueing and retry. If the relay is down at that instant — say, because the relay is a VM on the cluster that just had the outage you are being notified about — the notification is gone, permanently, with no retry and nothing in a queue to inspect.

For a hypervisor cluster, where the most important notifications are generated exactly when infrastructure is unhealthy, that distinction is not academic. smtp is convenient; sendmail with a properly configured relay is the one that still works during an incident. Better still: do not depend on a single channel at all, which is the argument for a second target on a different transport.

Configuration lives in two files

Targets and matchers are stored in /etc/pve/notifications.cfg, which is in pmxcfs and therefore replicated to every node. Secrets — SMTP passwords, Gotify tokens, webhook secrets — live separately in /etc/pve/priv/notifications.cfg, which is root-only.

That split is why you can safely include notifications.cfg in a configuration backup or a diff without exporting credentials, and why a restore of /etc/pve that skipped priv/ leaves you with targets that are configured and cannot authenticate.

Configuration changean SMTP target, created from the CLI— Configuration change. Creates a target only - it will deliver nothing until a matcher routes something to it. The password is written to /etc/pve/priv/notifications.cfg, not to the file you can diff.

set -euo pipefail

# mode is one of insecure, starttls or tls. Default is tls.
# Default ports follow the mode: 25 insecure, 465 tls, 587 starttls.
pvesh create /cluster/notifications/endpoints/smtp \
--name relay-ops \
--server smtp.example.com \
--port 587 \
--mode starttls \
--username 'pve-notify@example.com' \
--password 'REPLACE_ME' \
--from-address 'pve-notify@example.com' \
--author 'Proxmox VE prod-cluster' \
--mailto 'ops@example.com' \
--comment 'Primary mail path for cluster alerts'

# Confirm it exists.
pvesh get /cluster/notifications/endpoints/smtp --output-format yaml

Configuration changea webhook target for a chat or paging system— Configuration change. url, header and body all support Handlebars templating; secret values are stored in the root-only file and referenced as secrets.<name> so the token never appears in the body you can read.

set -euo pipefail

# Handlebars placeholders available in a webhook body:
#   {{ title }} {{ message }} {{ severity }} {{ timestamp }}
#   {{ fields.<name> }}   metadata such as fields.type, fields.hostname
#   {{ secrets.<name> }}  values stored in the root-only config
# Helpers: {{ url-encode v }} {{ escape v }} {{ json v }}
#
# The braces below are intentional and are what the target expects.

pvesh create /cluster/notifications/endpoints/webhook \
--name chat-ops \
--method POST \
--url 'https://chat.example.com/hooks/proxmox' \
--header 'name=Content-Type,value=application/json' \
--secret 'name=token,value=REPLACE_ME' \
--comment 'Cluster alerts into the ops channel'

# The body is a templated JSON document. Build it in the GUI if the quoting
# gets awkward from the shell - it is the same field either way.

Why the webhook body uses secrets.token instead of the token itself

A webhook target has to send an authorisation token somewhere in the request — usually a header, sometimes a query parameter, sometimes in the body. If that token were written literally into the url, header or body fields, it would be stored in /etc/pve/notifications.cfg, which is world-readable inside the cluster and which any user with audit permission on the notification configuration can read.

The secret option stores the value in /etc/pve/priv/notifications.cfg instead, and the template references it as {{ secrets.token }}. At delivery time the template is rendered with the secret substituted in, so the token is on the wire and never in the readable configuration.

This is the same separation PVE uses everywhere for credentials, and it has a practical consequence worth knowing: a webhook target exported from one cluster and imported into another will render {{ secrets.token }} as nothing, because the secret did not travel with the configuration. The target will be present, the matcher will route to it, delivery will fail with an authorisation error, and the notification is lost. Re-create the secret explicitly after any migration or restore.

Matchers: the routing layer

A matcher holds rules and a list of targets. The rules are of three kinds and they can be combined:

Rule	Syntax	Matches on
`match-field`	`exact:type=vzdump` or `regex:hostname=^pve-0[123]$`	Notification metadata
`match-severity`	`error` or `warning,error`	Severity level
`match-calendar`	`mon..fri 09:00-17:00`	Wall-clock time

mode decides how multiple rules combine: all requires every rule to match, any requires at least one. invert-match reverses the whole result.

A matcher with no rules matches everything. That is not a bug; it is how the default configuration works.

Configuration changeroute backup failures to the paging channel, and nothing else— Configuration change. mode=all means both rules must hold, so this fires only for vzdump notifications whose severity is error - a successful backup carries severity info and will not match.

set -euo pipefail

pvesh create /cluster/notifications/matchers \
--name backup-failures \
--mode all \
--match-field 'exact:type=vzdump' \
--match-severity error \
--target chat-ops \
--comment 'Failed backups page the on-call channel'

# A second matcher as the safety net: anything that is not routine,
# including system-mail from smartd and mdadm, which carries severity
# unknown and would be missed by an error-only rule.
pvesh create /cluster/notifications/matchers \
--name catch-all-non-info \
--mode all \
--match-severity 'warning,error,unknown' \
--target relay-ops \
--comment 'Safety net - everything that is not routine goes to mail'

pvesh get /cluster/notifications/matchers --output-format yaml

Matchers are independent, not a first-match chain

Coming from firewall rules or mail filters, the natural assumption is that matchers are evaluated in order and the first one wins.

They are not. Every matcher is evaluated against every notification, and a notification is delivered to the union of all targets belonging to all matchers that matched. One event can produce three deliveries.

Two consequences:

There is no “stop processing” rule. You cannot write a matcher that swallows an event so a later one does not see it. To exclude, use invert-match on the matcher that would otherwise catch it, or narrow that matcher’s own rules.
Duplicate alerts are a configuration outcome, not a bug. A vzdump error matching both a targeted backup matcher and a broad severity matcher will be delivered twice. If both point at the same target, you get two messages.

The default configuration, and why it is not enough

A fresh PVE installation ships with a built-in sendmail target named mail-to-root and a built-in matcher named default-matcher that routes to it. The built-ins can be disabled but not deleted, which is deliberate — it guarantees there is always some path for a notification to take.

mail-to-root delivers to the email address configured on the root@pam user. On a freshly installed cluster that address is frequently unset or still root@localhost, which means every notification the cluster has ever generated is sitting in /var/mail/root on a node nobody logs into.

Read-only / Safefind out where your cluster is currently sending things— Read-only. Run this on a cluster you have inherited, before configuring anything - the answer is frequently that notifications are going nowhere and have been for months.

set -euo pipefail

# What targets and matchers exist, including the built-ins?
pvesh get /cluster/notifications/endpoints/sendmail --output-format yaml
pvesh get /cluster/notifications/matchers --output-format yaml

# Where does mail-to-root actually go? Check the address on root@pam.
pvesh get /access/users/root@pam --output-format yaml | grep -i email

# Is there undelivered mail piling up locally? A large mbox here is a
# direct measure of how long notifications have been going nowhere.
ls -lh /var/mail/root 2>/dev/null || echo 'no local mail spool'

# Is the local MTA queueing because the relay is unreachable?
postqueue -p 2>/dev/null | tail -5 || echo 'postfix not installed or no queue'

Per-job overrides

Backup jobs carry a notification-mode option with two values:

notification-system — the job’s notifications go through targets and matchers, like everything else.
legacy-sendmail — the job sends mail directly through the system sendmail, to an address configured on the job, bypassing the notification system entirely. This reproduces the pre-8.1 behaviour and is deprecated.

Read-only / Safefind backup jobs that bypass the notification system— Read-only. Any job showing legacy-sendmail is outside every matcher you have configured. Jobs with no notification-mode set at all use the current default, which is the notification system.

set -euo pipefail

# The notification-mode column is what you are reading. A job showing
# legacy-sendmail is outside every matcher; a job with the column empty is
# using the current default, which is the notification system.
pvesh get /cluster/backup \
--output-format json-pretty \
| grep -E '"(id|notification-mode|mailto)"'

# Or the whole job list in a readable form, if you prefer to eyeball it.
pvesh get /cluster/backup --output-format yaml

Testing that an alert actually arrives

Configuration that has never delivered a message is a hypothesis. The GUI provides a Test button per target — Datacenter → Notifications → select the target → Test — which sends a test notification through the full delivery path for that target. Using it requires Mapping.Use together with Mapping.Audit or Mapping.Modify on the ACL path /mapping/notifications.

But a target test only proves the target works. It does not exercise the matcher, and the matcher is where the routing mistakes are. To test the whole path you need to generate a real notification of the type you care about.

Service impact possibleend-to-end test: make a real backup fail and confirm the page arrives— Service impact is minimal but non-zero - this runs a real backup job against a real guest. Do it in a maintenance window on a guest you have chosen, not on production at random.

set -euo pipefail
VMID=9999

# The honest test is a genuine failure of the type your matcher keys on.
# Backing up a VMID that does not exist produces a vzdump notification with
# severity error and type vzdump - exactly what the matcher is written for.
vzdump "$VMID" --storage pbs-main --mode snapshot || true

# Then verify, in this order:
#   1. The notification was generated. Check the task log and the journal.
journalctl -u pvedaemon --since '5 min ago' --no-pager | grep -i vzdump

#   2. It was delivered. Check the target's own logs - the mail log for
#      sendmail and smtp, the Gotify server, the webhook receiver.
journalctl -u postfix --since '5 min ago' --no-pager | tail -20

#   3. A human received it. This step cannot be automated and is the only
#      one that actually matters. Ask the person on call whether their
#      phone lit up.

What happens if the notification path itself fails?

Nothing tells you. This is the structural weakness of every alerting system and Proxmox is not exempt: the component that reports failures cannot report its own failure, because the report would travel by the broken path.

The realistic failure modes, in rough order of how often they occur:

The relay rejects the sender. SPF, DKIM or DMARC on the receiving domain rejects mail from a hypervisor that is not an authorised sender for example.com. Postfix queues, retries, and eventually bounces to a local mailbox. Silent from the operator’s point of view.
The Gotify token was rotated and the target still has the old one. Delivery fails with a 401 that appears in the journal and nowhere else.
The webhook receiver changed its URL during an unrelated migration. HTTP 404 per notification, no alert about the alerts.
The smtp target’s relay is down — with no queueing, each notification is simply lost.
A matcher was narrowed during an unrelated change and now excludes the event class you care about.

There is exactly one defence and it is not clever: a heartbeat. Arrange for something to notify successfully on a schedule, and alert on its absence from a system that is not the cluster. A cron job on a PVE node that mails root once a day becomes a system-mail notification, flows through the same targets and matchers as everything else, and its arrival proves the whole path end to end. When it stops arriving, the monitoring system that expected it — external to the cluster — is what raises the alarm.

Without a heartbeat, the honest description of your alerting is “it worked the last time somebody checked”.

PBS has the same system, configured separately

Proxmox Backup Server implements the same target and matcher model with its own configuration and its own CLI. It is a separate system: a matcher on the PVE cluster does not route PBS notifications, and PBS generates event types that PVE does not — garbage collection, verification jobs, sync jobs, prune, and datastore health.

This matters because the events most worth being paged about are on the PBS side. A failed verification job means a backup that exists and may not restore. Configure both, and treat “PBS notifications are configured” as a separate line item from “PVE notifications are configured”.

Common mistakes

Configuring a target and never writing a matcher. The target is reachable, tests pass, and nothing is ever routed to it.
Keying a matcher on job-id. Manual backups do not carry the field, so the rule fails to match and manual jobs are never reported.
match-severity error as the safety net. It excludes unknown, which is where SMART and RAID warnings live.
Assuming matchers stop at the first hit. They do not, and duplicate alerts are the result.
Leaving default-matcher enabled alongside real matchers, then wondering why every alert arrives twice.
Testing the target but not the path. The Test button does not exercise matcher rules.
No heartbeat. A silent alerting system and a healthy cluster look identical.

Key takeaways

Targets are destinations, matchers are routing, and an unmatched notification is silently discarded.
sendmail queues and retries through the local MTA; smtp does neither. During an incident that difference decides whether the alert arrives.
Severity unknown covers system-mail, which carries SMART, mdadm and cron failures. Include it.
Every matcher is evaluated against every notification; there is no first-match-wins.
Backup jobs set to legacy-sendmail bypass matchers entirely — audit for them after an upgrade.
Test the full path with a real event, and run a heartbeat so that silence is detectable.

Knowledge check

Why this matters

Everything else in this part is instrumentation. This is the part that decides whether instrumentation matters.

A cluster can have perfect metrics, a beautiful Grafana dashboard, structured logs shipped to a central store and an SLO document signed by a director, and still fail catastrophically because a backup job stopped working in March and the notification about it went to a sendmail target pointed at root@localhost on a node nobody logs into.

The specific thing worth carrying away is that this system fails quietly by design. An unmatched notification is not an error; it is the documented behaviour. Which means the question “are we alerting correctly?” cannot be answered by looking at the configuration — the configuration will look fine. It can only be answered by generating a real event and confirming a human received it, and then by arranging a heartbeat so that the next time the path breaks, its silence is itself an alarm.