The pve_exporter

A dashboard panel reads “pve_cluster_info missing” for one cluster but shows correct values for the others in the same Prometheus instance. The exporter is up; /pve/health returns 200; nothing in the logs looks fatal. The cluster that is missing is the one that runs twelve nodes, 240 VMs, and a ?full=1 URL on a 15-second scrape interval. The exporter has tripped the Proxmox rate-limit on that cluster and is silently shedding requests.

This lesson is about pve-exporter - the community Prometheus exporter for Proxmox VE. The exporter is the bridge between the Proxmox API covered in lesson 01 and a Prometheus / Grafana stack. It is stateless, polling, and tunable per-cluster; getting it right is the difference between a working cluster view and a panel of “no data”.

What it is

pve-exporter (also published as prometheus-pve-exporter) is a small HTTP service that exposes Prometheus metrics on /metrics. On each scrape it authenticates to the Proxmox API with an API token, fetches a fixed set of endpoints, and translates the JSON into typed metrics. The exporter is stateless: each scrape is independent. There is no storage, no aggregation, no local database.

A single exporter instance serves the whole cluster. The exporter is not deployed on every node (that would multiply the API call volume by N). One exporter, running on the Prometheus host or on a peer, talks to pveproxy on one of the cluster nodes (it does not matter which), and returns metrics that already carry cluster, node, and vmid labels.

Why a sysadmin cares

Proxmox ships its own metrics through pvestatd (the agent that collects per-host stats) and the cluster status endpoints. Neither yields a Prometheus-native surface that answers cluster-wide questions without glue code. The exporter provides that surface for free.

A working pve-exporter gives the operator:

• Cluster membership and quorum state as a single time series.
• Per-node CPU, memory, IO, and uptime.
• Per-VM CPU, memory, disk IO, network IO, and status.
• Per-storage capacity, used, and content type.
• Backup job last-run timestamp and duration.

A misconfigured pve-exporter gives the operator a panel of identical, useless series that all show “no data”, or worse, an exporter that silently causes the cluster API to rate-limit every other consumer (proxmox-backup-proxy, the Terraform provider, the GUI itself).

How it works

        Prometheus               pve-exporter                  Proxmox API
        (scrape)                  (stateless)               (pveproxy + pvedaemon)
            |                          |                              |
            |   GET /metrics           |                              |
            +------------------------->|                              |
            |                          |  Auth + GET cluster/resources |
            |                          +----------------------------->|
            |                          |<-----------------------------+
            |                          |  GET nodes                    |
            |                          +----------------------------->|
            |                          |<-----------------------------+
            |                          |  GET storage                  |
            |                          +----------------------------->|
            |                          |<-----------------------------+
            |                          |  GET per-VM status (per guest) |
            |                          +----------------------------->|
            |                          |<-----------------------------+
            |   200 text/plain;         |                              |
            |   version=0.4              |                              |
            +<-------------------------+|
            |                          |

Each scrape is one or more API calls. The exporter groups the calls so a 200-VM cluster is ~250 calls per scrape, not ~250 * 3. The default module set covers cluster, nodes, storage, and guests; extra modules can be enabled per-deployment.

Under the hood

The exporter itself is a small Python or Go binary (depending on fork). Both forks read a YAML config file, expose the same /metrics shape, and consume the same API tokens. Choose the upstream that matches the deploy pattern you maintain.

How to configure it

The exporter reads a YAML config that maps directly to the API parameters. The real production config looks like this:

# /etc/pve-exporter/pve.yml
# SEVERITY: CONFIGURATION
default:
  # API endpoint; the exporter talks to one pveproxy and the proxy
  # handles cluster-wide requests via pmxcfs.
  api_url: https://pve-01.example.lan:8006/api2/json
  api_token: "monitoring-pve@pam!prometheus"
  api_token_value: "${PVE_EXPORTER_TOKEN}"
  # Validate the cluster's own CA. Disable only for an internal
  # network where the CA is pinned out-of-band.
  verify_ssl: true
  # Modules: pick what your dashboards consume.
  modules:
    cluster: true
    node: true
    storage: true
    guests: true
    backup: true
  # Filter which VMs/CTs are exported. Empty filter = every guest.
  # In a 500-VM cluster, restricting by tag reduces per-scrape API
  # call volume by 80 percent with no loss of dashboard value.
  guest_filter: "production=true|environment=prod"
  # Per-cluster scrape module timeout. 10s is comfortable for a
  # 12-node / 240-VM cluster on a healthy API.
  timeout: 10

The secret itself is held outside the YAML; the exporter reads from PVE_EXPORTER_TOKEN. A systemd unit loads the secret from a EnvironmentFile= directive owned by root, mode 0400.

A systemd unit for the exporter:

# /etc/systemd/system/pve-exporter.service
# SEVERITY: CONFIGURATION (drop-in; reload required)
[Unit]
Description=Prometheus pve-exporter
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=pve-exp
Group=pve-exp
EnvironmentFile=/etc/pve-exporter/pve-exporter.env
ExecStart=/usr/local/bin/pve_exporter \
  --config.file=/etc/pve-exporter/pve.yml \
  --web.listen-address=0.0.0.0:9221
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/pve-exporter

[Install]
WantedBy=multi-user.target

The Prometheus scrape configuration is conventional:

# /etc/prometheus/prometheus.yml (job entry)
# SEVERITY: CONFIGURATION
scrape_configs:
  - job_name: pve
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['pve-exporter.internal:9221']
    # The exporter is stateless; honor the API rate-limit by
    # scraping no more often than every 30 seconds.
    scrape_interval: 30s
    scrape_timeout: 25s

How to validate it

The exporter is up and reachable:

# SEVERITY: READ-ONLY
curl -s http://pve-exporter.internal:9221/metrics | head

Expected: pve_up series with value 1, plus the pve_cluster_info series for at least one cluster name.

Specific series for a smoke test:

# SEVERITY: READ-ONLY
curl -s http://pve-exporter.internal:9221/metrics \
  | grep -E '^pve_(up|cluster_info|node_info|guest_info) ' \
  | head

A cluster with twelve nodes and two hundred VMs produces:

• pve_up{cluster="prod"} 1
• pve_cluster_info{cluster="prod",cluster_id="..."} 1
• 12 pve_node_info series
• ~200 pve_guest_info series
• Storage and backup series consistent with the cluster contents

Side-by-side validation against pvesh:

# SEVERITY: READ-ONLY
# The exporter should agree with pvesh on node count and CPU.
pvesh get /nodes --output-format json | jq '.[] | .cpu'
curl -s http://pve-exporter.internal:9221/metrics \
  | grep '^pve_node_cpu_usage_ratio'

Disagreement between pvesh and the exporter means one of: the exporter is mid-scrape and reporting stale numbers, the token scope was tightened, or the cluster joined a new node and the exporter’s cache has not refreshed (some forks cache briefly).

How it can fail

Five failure modes appear regularly in production pve-exporter deployments:

1. Exporter version drift after a Proxmox upgrade. A major PVE minor release removes or renames a JSON field. Symptom: the exporter container exits non-zero with a key error in the logs; pve_* series stop appearing; only pve_up continues.
2. Token expired or revoked upstream. pveum removed the token, or the user’s password age triggered a re-issue. Symptom: pve_up remains 1 (the exporter process is up), but pve_cluster_info and the guest series drop to missing; the exporter logs fill with HTTP 401.
3. Cardinality blowup from a guest_filter left blank. A single test cluster with five hundred VMs creates five hundred series per dimension and floods Prometheus. Symptom: Prometheus TSDB ingest stalls; prometheus_tsdb_head_series climbs past the cardinality budget; up{job="pve"} for other jobs reports scrape_error.
4. Rate-limit hit by a too-aggressive scrape interval. A team sets scrape_interval: 15s and the exporter starts returning HTTP 595 on internal calls. Symptom: random gaps in metric series; the exporter logs “rate limit exceeded”; other Proxmox API consumers (PBS, Terraform) also slow down.
5. Scrape timeout on a busy cluster. The cluster is large and the exporter cannot finish the module set inside Prometheus’s scrape_timeout. Symptom: Prometheus reports scrape_duration_seconds{job="pve"} near timeout; up{job="pve"} flips between 0 and 1; panel data drops.

How to troubleshoot it

The order is: exporter is up, exporter can authenticate, exporter’s modules are correct, the cluster is not overloaded.

1. Exporter is up. curl /metrics | head from the Prometheus host. If empty or refused, restart the exporter service and read its journal.
2. Exporter can authenticate. The exporter logs at debug level show every API call. Run with --log.level=debug (or fork equivalent) and confirm a 200 response on /access/ticket (if using ticket) or that the Authorization header is being accepted.
3. Modules are correct. A series you expect to see (for example pve_storage_used_bytes) is missing: confirm the module is enabled in the exporter config and the relevant endpoint is reachable for the token role.
4. Rate-limit. Inspect exporter logs for “rate limit exceeded”. If present, raise scrape_interval on the Prometheus side or disable a module.
5. Cluster overloaded. Inspect pvestatd health on every node; confirm the API daemon is responsive from the exporter host. pvesh get /cluster/resources should return in well under a second.

Security implications

• The API token is the credential. Read it from a EnvironmentFile or secret store. Never embed the value in pve.yml or in a config map. Mode 0400 on the secret file; user ownership set to the exporter’s service user.
• The exporter has no auth on /metrics. This is by convention, not by accident; Prometheus expects to scrape without authentication. The exporter should bind to the management network only. Restrict /metrics with a reverse proxy if the exporter must run on a multi-tenant network.
• The token’s role is the blast radius. A leaked token with PVEAuditor exposes the cluster layout. A leaked token with PVEAdmin enables VM deletion. Restrict the role, audit the access log on the cluster.

Performance implications

Three budgets matter:

• Cardinality. Per-cluster series scale with (nodes + guests + storage + tasks). A 500-guest cluster on defaults is roughly 1500 series, well inside Prometheus’s comfort zone. A guest_filter that does not bind is a 10,000-series incident waiting to happen.
• Scrape latency. A scrape that takes longer than scrape_timeout causes Prometheus to mark the target down. The default 10s scraper timeout is tight for a 1000-guest cluster; set scrape_timeout: 25s and scrape_interval: 30s for any cluster above 500 guests.
• Exporter process footprint. The Python fork uses ~50-80 MiB RSS at idle; the Go fork uses ~20 MiB. Both are negligible on a Prometheus host.

Production guidance

• One exporter per cluster. The exporter is stateless and the API returns cluster-wide data.
• Set scrape_interval to 30s or 60s by default. The cluster state changes on a human time-scale.
• Restrict guest_filter to the dashboards you actually consume. Do not let “everything” be the default.
• Run the exporter as its own service user, with /etc/pve-exporter owned by root and the secret file mode 0400.
• Alert on the exporter up == 0 and on the absence of pve_cluster_info. Both are first-class signals in Prometheus rules.
• Pin the exporter to a Prometheus scrape config that includes scrape_timeout comfortably below scrape_interval.

Verification

You should now be able to answer:

• Where does pve-exporter live in the request path between Prometheus and the Proxmox API?
• What is the relationship between the exporter’s modules and the API’s rate-limit?
• Why is one-exporter-per-cluster the right shape, not one-per-node?
• What is the right scrape interval and timeout for a 500-VM cluster?

Quiz