Cluster Health Metrics

It is 02:30. The on-call engineer gets paged: “Checkouts failing”. Three seconds later, a second page: “VM CPU at 99% on pve-04”. A third page: “API latency to pve-04 above 5 s”. The on-call engineer reaches for the per-VM dashboard and finds nothing; the panel has stopped updating across the whole cluster. The actual cause is a corosync split: four of twelve nodes visible to the GUI, eight frozen because they no longer have quorum. The “API latency” alert was correct but symptom-level; the “VM CPU” alert was noise; the “checkout failure” was customer-visible. A single pve_cluster_quorum alert two minutes earlier would have named the cause before any customer saw impact.

This lesson is about cluster health metrics: corosync membership, quorum, and the per-node state the Proxmox cluster publishes. These are the metrics that distinguish a cluster-wide incident from a per-host accident.

What it is

“Cluster health” is a small set of signals that describe whether the Proxmox cluster is a single coherent system or a split one:

• Quorum - the boolean fact that more than half of the configured cluster nodes can see each other through corosync. Without quorum, the cluster refuses to start, migrate, or fence VMs.
• Membership - the list of nodes corosync currently believes are alive, plus the IPs it uses to reach each.
• Node status - per-node health: online, offline, left-the-cluster, or unknown. Drives the pve_node_up series.
• Service state - the high-level “is pve-cluster service healthy”, used as a guard against silent agent death.

Proxmox publishes these through three channels:

1. pvestatd per node, polled by pve-exporter and consolidated into pve_cluster_info, pve_cluster_quorum, pve_node_status, pve_node_up.
2. pmxcfs, the shared cluster filesystem that holds /etc/pve/cluster.conf and is the cluster’s source of truth for membership.
3. The corosync ring itself, which exposes its own counters over the network to other cluster tooling.

Why a sysadmin cares

The cluster is the unit of operation. A single Proxmox node can be rebooted, but a cluster loss is a customer-visible event. The metrics that answer “is the cluster OK?” must:

• Page on quorum loss in under 60 seconds. Quorum loss freezes the cluster’s VM operations; the customer sees it inside 60 seconds, not inside five minutes.
• Distinguish a corosync split from a single node failure. The two have very different remediation paths.
• Stay silent during a planned rolling restart. The alert that pages every time a node leaves the cluster for 30 seconds is a alert that gets muted by 03:00.

The metrics also answer a more subtle question that dashboards fail to ask: which node is currently the elected master for the PMXCFS filesystem? That election changes which pveproxy you should target when API calls return stale reads.

How it works

          corosync ring
       (UDP multicast on a
        dedicated network)
              |
   +----------+----------+----------+----------+
   |          |          |          |          |
 pve-01    pve-02    pve-03    pve-04     pve-05
   |          |          |          |          |
   +---- pvestatd (per-host) reads corosync quorum
              |
              +----- PMXCFS publishes /etc/pve/cluster.conf
              |
              +----- pve-exporter polls /cluster/status
              |
              v
        Prometheus
              |
              v
        Alertmanager
              |
              v
           Page

The arrow that matters is the one out of corosync: every node has the same view of “who is in the ring and can we have quorum”, and that view is what Proxmox publishes. A node that has been silently disconnected from corosync for 90 seconds is treated as offline whether or not pveproxy still serves HTTP - and that is the right distinction for the alerting model.

Under the hood

How to configure it

The exporter side requires only that the cluster and node modules are enabled (the default in most forks).

The Prometheus side is where the alerting model lives:

# /etc/prometheus/rules/cluster-health.rules.yml
# SEVERITY: CONFIGURATION (rule reload; no scrape impact)
groups:
- name: proxmox-cluster
  interval: 30s
  rules:

  # Cluster has lost quorum. Pages on cause, not on symptom.
  - alert: PVEClusterQuorumLost
    expr: pve_cluster_quorum == 0
    for: 30s
    labels:
      severity: page
      team: virtualization
      component: pve
    annotations:
      summary: 'Proxmox cluster {{ $labels.cluster }} lost quorum'
      description: |
        Cluster `{{ $labels.cluster }}` is no longer quorate.
        VMs cannot start, stop, migrate, or fence. Inspect
        `pvecm status` on every node and check corosync on the
        ring network. Last good quorum: {{ query (with
        (.cluster == "{{ $labels.cluster }}"))
        last_over_time(pve_cluster_quorum[24h]) }} ago.
      runbook_url: 'https://runbooks.example.com/pve/quorum-loss'
      dashboard_url: 'https://grafana.example.com/d/pve-cluster'

  # Node has been out of the ring for > 60 seconds. The cluster
  # still has quorum (this rule fires independently of PVE
  # ClusterQuorumLost); the lost node is the alert target.
  - alert: PVENodeDown
    expr: pve_node_up == 0
    for: 60s
    labels:
      severity: page
      team: virtualization
      component: pve
    annotations:
      summary: 'PVE node {{ $labels.node }} down for 60s'
      description: |
        Node `{{ $labels.node }}` has not responded via the
        cluster API for over one minute. SSH to the node and
        inspect `pvestatd`, `corosync`, and the ring network.
      runbook_url: 'https://runbooks.example.com/pve/node-down'

  # Corosync counter has stopped incrementing. Distinct from API
  # down: the node answers API but is excluded from the ring.
  # The right disposition is corosync, not pveproxy.
  - alert: PVECorosyncRingStuck
    expr: rate(pve_node_info_seen[5m]) == 0
    for: 2m
    labels:
      severity: ticket
      team: virtualization
      component: pve
    annotations:
      summary: 'pve-exporter cannot see corosync updates for {{ $labels.node }}'

The order matters. PVEClusterQuorumLost is the page. PVENodeDown is the page when there is still quorum. PVECorosyncRingStuck is the ticket to investigate the silent-fail case.

How to validate it

Side-by-side with the canonical Proxmox CLI:

# SEVERITY: READ-ONLY
# The pvesh output is the truth; the exporter must agree.
pvesh get /cluster/status --output-format json \
  | jq '. | {quorum: .quorum, nodes: [.[] | .name]}'

Expected: a quorum: 1 field and a list of every node name. The exporter’s pve_cluster_quorum series should report 1 and pve_node_up{node="..."} should be 1 for every named node.

# SEVERITY: READ-ONLY
# Corosync's own view, in case the API is lying.
pvecm status
corosync-cmapctl | grep -E '^totem\.|^members\.|^quorum\.'

A cluster that has lost a node reports:

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      4
Quorum:           3   (3 active + 1 without votes)
Flags:            Quorate

Members:
  nodeid: 1 votes: 1   addr: 10.0.0.11
  nodeid: 2 votes: 1   addr: 10.0.0.12
  nodeid: 3 votes: 1   addr: 10.0.0.13
  nodeid: 4 votes: 1   addr: 10.0.0.14
  nodeid: 5 votes: 0   addr: 10.0.0.15 (offline)

Validate that Prometheus sees what the CLI shows:

# SEVERITY: READ-ONLY
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=pve_cluster_quorum' | jq
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=pve_node_up' | jq

How it can fail

Five failure modes recur in production cluster-health alerting:

1. Quorum lost on a three-node cluster during a planned upgrade. The operator reboots node 2; the cluster drops to 2-of-3 votes and the PMXCFS filesystem goes read-only. Symptom: pve_cluster_quorum flips to 0; alerts page; the cluster correctly refuses to start new VMs.
2. Silent node ejection by corosync. A cabling fault causes one node to drop from the corosync ring for 15 seconds, rejoin, drop again. Symptom: pve_node_up flaps at sub-minute intervals; alerting is noisy; the underlying fault is the switch on the dedicated ring network.
3. Stale pve_cluster_info after a node rename. The exporter caches the node list briefly. Symptom: dashboards show the old name for an hour; alerts that interpolate $labels.node send the wrong name.
4. False-positive PVE NodeDown during API maintenance. The cluster API is being restarted for a TLS renewal on a single node. Symptom: the rule fires for that node; correlation with the pveproxy journal shows the planned restart.
5. Quorum recovery doesn’t propagate to PMXCFS. corosync reports quorate but PMXCFS keeps the read-only flag. Symptom: pve_cluster_quorum == 1 while write operations still error; this is a Proxmox bug class; recovery requires systemctl restart pve-cluster.

How to troubleshoot it

The order is: confirm the symptoms, confirm the source of truth, inspect corosync, inspect the ring.

1. Confirm the symptom. Open the cluster-health dashboard; read pve_cluster_quorum and the per-node pve_node_up together.
2. Confirm against the source of truth. pvecm status on every node. The CLI is authoritative; the exporter is a translation.
3. Inspect corosync. journalctl -u corosync -n 500 on every node. Look for “lost”, “split”, or “token” lines.
4. Inspect the ring. A ping to the dedicated ring subnet from a node that is supposedly excluded will reach 100% loss even while the management network still answers.
5. Decide. If quorum is 1 and a single node is 0, restart that node’s corosync. If quorum is 0, find the partition and recover the larger side first.

Security implications

The cluster-health metrics expose cluster topology: node names, ring subnets, and node IDs. A token with PVEAuditor is enough to read them. Treat the exporter’s /metrics endpoint as reconnaissance-grade information for an attacker preparing to exploit a corosync misconfiguration.

Two operational precautions:

• Do not export pve_node_info or pve_cluster_info to a public Prometheus. Keep them on the internal network.
• Do not commit node names from these metrics to issue tickets or wiki pages; the names are stable for the life of the cluster and can fingerprint the deployment.

Performance implications

Cluster-health series are among the cheapest the exporter publishes. Cardinality is 1 * nodes. There is no need to alert on cardinality; there is no need to scrape more often than 30 seconds. The risk is not the exporter - it is the alerts themselves.

Three alert-tuning costs:

• A for: clause shorter than the rolling-restart time. A 30-second for: will fire on every planned maintenance. 60 seconds is the minimum for a sensible signal-to-noise ratio.
• Duplicate alerts. The pair PVEClusterQuorumLost and PVENodeDown may both fire on the same incident; inhibit one under the other, or merge them.
• Group by node, fire by cluster. Alert aggregation should be per cluster, not per node, when quorum is the cause.

Production guidance

• Run pvecm status against the live cluster before trusting the exporter. The CLI is the source of truth; the exporter translates.
• Wire PVEClusterQuorumLost first, PVENodeDown second, and use Alertmanager inhibition so a partition does not flood the on-call.
• Scrape at 30 seconds. Faster is noise, not signal.
• Document the ring network in the runbook. The cluster is only as strong as the dedicated network that connects its nodes.
• Run the cluster health alert rules under a chronic-noise budget; if the alert is muted weekly, redesign it.

Verification

You should now be able to answer:

• What is the difference between a corosync membership failure and an API reachability failure, and which metric exposes each?
• Why is pve_cluster_quorum == 0 the alert to wire before any per-node rule?
• What is the right for: clause for PVENodeDown to suppress rolling-restart noise?
• Why does a partition that touches 8 of 12 nodes merit a single page, not nine?

Quiz