Proxmox APIs and Metrics

A Grafana panel reads “VM 104 CPU at 99%” but the VM is not slow. The metric is from the host that runs the VM, not the VM itself; the host’s pvestatd counter has drifted after a memory balloon event. The engineer clicks through to the same answer three different ways and gets three different numbers. That ambiguity is what the Proxmox REST API exists to remove - one authoritative surface for cluster, node, VM, storage, and backup state - and what observability for Proxmox requires you to master.

This lesson establishes the API. It covers the REST surface (/api2/json), the pvesh CLI that wraps it, the three credential models (ticket, token, user) and which one a monitoring integration should choose, the rate limits you only discover at 03:00, and the failure modes that take a metrics pipeline offline silently.

What it is

The Proxmox VE API is a JSON-over-HTTPS interface served by pveproxy (the per-node HTTP frontend) and pvedaemon (the per-node privileged daemon). The URL prefix is /api2/json, and the authoritative endpoint catalog lives in the pvesh CLI and in the API viewer at pve.proxmox.com. Every Proxmox GUI action calls the same API; the GUI is a thin client that calls it.

Three surfaces matter for monitoring:

• /api2/json/cluster/resources - aggregated cluster-wide resources. The endpoint every cluster-level exporter consumes.
• /api2/json/nodes - per-node status, including CPU, memory, storage, and the corosync membership state.
• /api2/json/access/users and /access/domains - identity and authentication surface; the API is where tokens live.

The APIs are documented per-form. A path that is not in the catalog returns HTTP 501.

Why a sysadmin cares

Without the API, Proxmox observability reduces to SSH-per-node plus pvestatd scraping. That approach makes cluster-wide questions hard: total VMs across a 12-node cluster, the storage used per datastore, the backup health across all VMs. The API is the canonical answer to “what does the cluster look like right now”, and a metrics exporter is just a polling loop over it.

The security and operational price of using it badly is high. A token issued with the wrong role gives the exporter permission to migrate VMs or delete containers. A token left unrevoked after a decommissioned exporter remains a valid credential until expiry. A scraper that exceeds the rate-limit starts returning HTTP 595 responses, the metrics pipeline shows gaps, and the on-call engineer has no panel to look at during the incident.

How it works

The mental model has four moving parts: the API protocol, the authentication model, the rate-limit, and the per-node proxy.

    HTTP client                              pveproxy
    (exporter, pvesh,                       (per-node TLS
     curl, Proxmox GUI)                      + auth + ACL)
        |                                        |
        |   HTTPS POST /api2/json/...            |
        +--------------------------------------->|
        |                                        |
        |   ticket OR token                       |
        |   in Authorization header              |
        |                                        |
        |                                        +-- pvedaemon (privileged,
        |                                        |  talks to kernel, qemu,
        |                                        |  LVM, ZFS, Ceph)
        |                                        |
        |   200 OK (JSON)                         |
        +<---------------------------------------+
        |
   poll loop (15s scrape typically)

pveproxy is a Perl daemon that terminates TLS, validates the credential, applies the ACL, and forwards to pvedaemon over a local Unix socket. pvedaemon does the privileged work (snapshot, migrate, start VM). Both daemons run per node; the cluster API /api2/json/cluster/... is satisfied by the proxy on whichever node the request lands at, which reads the shared cluster filesystem (pmxcfs) for cluster-wide state.

Under the hood

The rate limit is undocumented but real. Proxmox applies a per-source-IP cap on the API daemon; sustained high call rates return HTTP 595 with the message “rate limit exceeded”. The cap is generous for human use (a GUI user issues a few calls per minute) and sharp for an exporter that polls every node, every VM, every storage, every task, every 15 seconds. A naive exporter that requests /cluster/resources?full=1 on every scrape will trip the cap inside a busy cluster.

How to configure it

Create a dedicated user and token for the monitoring exporter. Use pveum (the user-manager CLI) so the change is auditable in /etc/pve/user.cfg.

# SEVERITY: CONFIGURATION (writes user.cfg; no service restart)
# Realms assume the default pam authentication.
pveum useradd monitoring-pve@pam --comment 'Prometheus exporter'

Grant the user a role that matches what the exporter calls. The default PVEAuditor role is the minimum that exposes the read paths the exporter needs.

# SEVERITY: CONFIGURATION
pveum aclmod / --user monitoring-pve@pam --role PVEAuditor

Issue an API token. Tokens do not need a password; the secret is displayed once at creation.

# SEVERITY: CONFIGURATION
# Output line: "│ full token id: monitoring-pve@pam!prometheus"
# Output line: "│ value: 1a2b3c4d-..."
# Capture the value; it cannot be retrieved later.
pveum tokenadd monitoring-pve@pam prometheus \
  --comment 'pve-exporter token'

Propagate the token to the exporter using whatever secrets manager the rest of your stack uses. Never commit the secret to git.

Validate from the exporter host before pointing the scraper at it:

# SEVERITY: READ-ONLY
curl -sk -H "Authorization: PVEAPIToken=monitoring-pve@pam!prometheus=$TOKEN" \
  https://pve.example.lan:8006/api2/json/cluster/resources | jq '.data[0]'

Expected output is a JSON object with fields type, id, node, status, cpu, mem, maxmem, disk, maxdisk, uptime. The presence of these fields proves the token is valid, the role is sufficient, and the cluster is reachable from the exporter host.

How to validate it

Beyond the curl above, the export-side validation is the same as for every Proxmox API integration:

# SEVERITY: READ-ONLY
# Confirm the user still exists and the role is unchanged.
pveum userlist
pveum acllist --user monitoring-pve@pam

# SEVERITY: READ-ONLY
# Time the call. Healthy pveproxy + pvedaemon respond in < 50 ms for
# cluster/resources on a 12-node cluster. Sustained calls over 200 ms
# indicate pveproxy is contending for the per-process accept queue.
time curl -sk -H "Authorization: PVEAPIToken=..." \
  https://pve.example.lan:8006/api2/json/cluster/resources

# SEVERITY: READ-ONLY
# Verify each node answers. A partial cluster response (one node
# missing) indicates a corosync split, not an API failure.
for N in pve-01 pve-02 pve-03; do
  curl -sk -o /dev/null -w "%{http_code} %{time_total}s $N\n" \
    -H "Authorization: PVEAPIToken=..." \
    "https://$N.example.lan:8006/api2/json/nodes/$N/status"
done

# SEVERITY: READ-ONLY
# Confirm pvestatd is publishing values on every node. The agent
# populates the RRD files that back many of the metrics the exporter
# reports; without it, per-VM metrics read zero.
systemctl status pvestatd
ls -l /var/lib/rrdcached/db/ | head

How it can fail

Five failure modes appear repeatedly in production:

1. Token deleted from the GUI. A colleague removed the token while cleaning up credentials. Symptom: HTTP 401 from every API call; exporter logs fill with auth errors; every cluster-level metric series disappears at the same timestamp.
2. Role downgraded. pveum aclmod / --role PVENoAccess (a common mis-type) silences the exporter silently. Symptom: HTTP 200 responses with empty arrays; dashboards show “no data” but no errors.
3. Token secret leaked into a config file. The exporter’s YAML is world-readable or committed to a public mirror. Symptom: nothing visible until the metrics show clusters you never owned; the token must be revoked and rotated.
4. Rate-limit hit. The exporter polls too aggressively for a busy cluster. Symptom: HTTP 595 responses from pveproxy; random gaps in metric series; the exporter retries and compounds the load.
5. pveproxy stuck. A long-running TLS connection in CLOSE_WAIT state exhausts the accept queue. Symptom: every new request hangs; established connections still return data; restart of pveproxy clears the queue.

How to troubleshoot it

The order is: token, role, network, rate-limit, daemons.

1. Token. pveum userlist shows the user; pveum tokenlist monitoring-pve@pam shows attached tokens. If the token is gone, recreate it.
2. Role. pveum acllist --user monitoring-pve@pam shows the ACL. The expected rule is /(ACL) PVEAuditor. If empty or wrong, run pveum aclmod / --user ... --role PVEAuditor.
3. Network. From the exporter host, openssl s_client -connect pve:8006 confirms TLS. tcpdump -i any host pve and port 8006 on the exporter host shows whether packets leave.
4. Rate-limit. journalctl -u pveproxy -n 200 for the “rate limit exceeded” string. If present, raise the scrape interval or reduce the modules the exporter pulls.
5. Daemons. systemctl status pveproxy pvedaemon on every node. A daemon crash is visible in journalctl -u pvedaemon -n 200. Restart with systemctl restart pveproxy; pvedaemon restart affects VM operation, not just metrics.

Security implications

Three rules apply:

• Token scope equals exporter scope. The token’s role must match what the exporter reads. PVEAuditor is sufficient for every pve-exporter metric; PVEAdmin is not. Issuance is cheap via pveum, and the audit log records who created the role.
• Treat the secret as a credential. Rotate on personnel change, on suspect compromise, and at the cadence your compliance regime requires. The pve-exporter reads the secret from the environment; no file should be world-readable.
• Restrict the API surface. pveproxy listens on the internal network by default. A reverse proxy that faces the public internet must terminate TLS, validate the source IP, and apply WAF rules that reject credential-stuffing patterns.

Performance implications

The Proxmox API is not a database; it is a request-response daemon that serialises work through pvedaemon. Two constraints matter:

• Concurrency limit per node. pvedaemon is single-threaded per node; a slow API call blocks the daemon. Long-running endpoints (/nodes/{node}/vzdump, /cluster/backup/{id}) compete with read calls.
• Scrape interval cost. A pve-exporter scrape that pulls /cluster/resources?full=1, /nodes, /storage, and per-VM /status/current is 1 + N + M + V*x API calls. For twelve nodes, four storage pools, and 200 VMs, that is hundreds of calls per scrape. At a 15-second interval this is the difference between stable and saturated.

Mitigations:

• Use ?full=0 for the metrics exporter; full=1 adds guest network and disk details the exporter does not need.
• Raise the scrape interval to 30s or 60s for the cluster exporter; cluster state changes on a human time-scale.
• Disable exporter modules per cluster if a dashboard does not consume them.

Production guidance

• Treat the token like a database password. Vault, sealed secrets, or a least-readable file. Rotate quarterly.
• Use pveum useradd and pveum aclmod. The GUI looks the same but the audit trail is weaker.
• Run a single exporter per cluster, not per node. The API returns cluster-wide data.
• Alert on exporter up == 0, not on the rate of HTTP 5xx. The scrape failure is the symptom; the upstream exporter being down is the cause.
• For very large clusters, prefer prometheus-pve-exporter with module filtering, and split the cluster into groups of nodes scraped by independent exporters.

Verification

You should now be able to answer:

• What are the three credential models Proxmox exposes, and which should a monitoring integration use?
• What is the relationship between pveproxy, pvedaemon, and the /api2/json URL prefix?
• What is the rate-limit symptom, and what exporter behaviour produces it?
• What is the right user-role combination for a metrics-only exporter?

Quiz