External metric servers: InfluxDB, Graphite and OpenTelemetry

Proxmox VE has a native metric export. It is not an exporter and it is not scraped: pvestatd on every node pushes the statistics it has just collected to whatever servers are defined in /etc/pve/status.cfg, on its own schedule, whether anybody is listening or not.

That single architectural fact — push, not pull — decides most of what follows. It is why there is no /metrics endpoint to curl, why a broken export produces no error on the collector side, and why the failure mode of this feature is silence rather than a red dashboard.

Three backends are supported in PVE 9: Graphite, InfluxDB and OpenTelemetry.

Where the configuration lives

/etc/pve/status.cfg holds every metric server definition. It is inside pmxcfs, so a definition added on one node is active on all of them within milliseconds, and every node begins pushing independently.

# cat /etc/pve/status.cfg

influxdb: metrics-prod
server influx.example.com
port 8086
influxdbproto https
organization ops
bucket pve
token REDACTED
timeout 5
max-body-size 25000000
verify-certificate 1

graphite: graphite-legacy
server graphite.example.com
port 2003
proto tcp
path proxmox.prod
timeout 2
disable 1

InfluxDB

The most common target, and the one with the most options because it has two quite different transports.

set -euo pipefail

# influxdbproto https selects the v2 write API. The token must have write
# permission on the named bucket and nothing beyond it.
pvesh create /cluster/metrics/server/metrics-prod \
--type influxdb \
--server influx.example.com \
--port 8086 \
--influxdbproto https \
--organization ops \
--bucket pve \
--token 'REPLACE_ME' \
--timeout 5 \
--verify-certificate 1

# Confirm it is present and enabled.
pvesh get /cluster/metrics/server --output-format yaml

Graphite

Simpler and older, and still the right answer if that is what your organisation runs.

path is the prefix under which everything lands, and it is worth setting deliberately. Two clusters both pushing under the default proxmox prefix will interleave their metrics into one namespace, and node names are the only thing distinguishing them. proxmox.prod and proxmox.dr cost nothing at setup and save a confusing afternoon later.

The timeout option applies to TCP and exists for a specific reason: pvestatd is a single-threaded loop, and a TCP connection to an unreachable Graphite server that blocks for thirty seconds blocks statistics collection for the whole node. Keep it low — two seconds is plenty on a LAN.

OpenTelemetry

New in PVE 9, and the one to reach for if your organisation has standardised on OTel collectors. It pushes over OTLP/HTTP to a collector endpoint.

Two limits worth knowing before you commit to it:

• JSON encoding only. Protobuf is not currently supported. Most collectors accept both, but a collector configured to require protobuf on its OTLP/HTTP receiver will reject everything PVE sends.
• The path matters. OTLP/HTTP metrics are posted to /v1/metrics on the collector. If your collector is behind a reverse proxy that rewrites paths, that rewrite has to preserve it.

set -euo pipefail

pvesh create /cluster/metrics/server/otel-collector \
--type opentelemetry \
--server otel.example.com \
--port 4318 \
--proto https \
--path /v1/metrics \
--timeout 5

pvesh get /cluster/metrics/server --output-format yaml

Push export against pull-based scraping

Both approaches are in use on real Proxmox estates and they are not equivalent. The decision is architectural, not a matter of taste.

The strongest argument for pull, and the one that usually decides it: up is a metric. Prometheus records a failed scrape as data, so “we stopped receiving metrics from pve-03” is a condition you can alert on. With push, the absence of data is the absence of data, and building an alert on absence requires deliberate work.

The strongest argument for push: it needs nothing installed, nothing to maintain, no inbound access into the management network, and it exports the exact numbers the GUI is showing — so an operator and a dashboard never disagree about what a figure means.

Cardinality: estimate it before you point it at a shared platform

PVE exports metrics per node, per guest and per storage, on pvestatd’s collection loop. That loop runs on the order of every ten seconds, which makes the write rate a function of your guest count rather than something you configure.

A rough estimate for planning:

Three thousand series is unremarkable for a dedicated InfluxDB and can be noticeable on a shared platform that is already carrying a large estate. The number that surprises people is not the total but the churn: every VM you create adds series, every VM you destroy leaves series that never receive another point, and a cluster used for CI or ephemeral test guests can generate a great deal of dead cardinality over a year.

If your platform team charges by series or enforces a cardinality budget, have that conversation before turning on the export for a cluster with a thousand guests.

set -euo pipefail

# 1. What is configured, and is anything disabled?
pvesh get /cluster/metrics/server --output-format yaml

# 2. Is pvestatd complaining? This is the only place an HTTP failure appears.
journalctl -u pvestatd --since '1 hour ago' --no-pager \
| grep -Ei 'metric|influx|graphite|otel|opentelemetry' || echo 'no metric errors logged'

# 3. Is traffic leaving the box at all? Count packets to the target port.
#    Useful for the UDP case, where nothing is logged either way.
METRIC_HOST=influx.example.com
METRIC_PORT=8086
timeout 30 tcpdump -ni any "host $METRIC_HOST and port $METRIC_PORT" -c 20 || true

# 4. The only authoritative check is on the receiving side: query the
#    backend for the most recent point per node and compare it to now.

Step 4 is the one that matters, and it is deliberately the one this lesson cannot give you a command for — it belongs to your metrics platform, not to Proxmox. Whatever form it takes, write it down as a check that runs on a schedule rather than one somebody remembers to run.

Common mistakes

• Leaving influxdbproto at the UDP default for an export you intend to rely on, then having no way to tell whether it works.
• A thirty-second timeout against a flaky server, which turns a monitoring problem into a node statistics problem.
• No staleness alert. A node that stops pushing simply vanishes.
• A broadly scoped InfluxDB token in a replicated, readable file.
• Two clusters sharing the default Graphite path, interleaved into one namespace.
• Setting verify-certificate 0 as the first response to a TLS error rather than fixing the trust store.
• Assuming the export backfills after an outage. It does not; the data is gone unless you pull it out of RRD by hand.

Key takeaways

• PVE pushes from pvestatd on every node; nothing scrapes it. /etc/pve/status.cfg is the whole configuration and it is replicated.
• Three backends: Graphite, InfluxDB, and OpenTelemetry over OTLP/HTTP with JSON encoding only.
• UDP transports fail silently and identically to succeeding. HTTP or HTTPS makes failure observable from the node.
• timeout protects pvestatd; a large value lets a monitoring outage damage local statistics collection.
• Push gives you no equivalent of Prometheus’s up, so an explicit staleness alert on the receiving side is mandatory rather than optional.
• Estimate cardinality from guest count before pointing a large cluster at a shared platform.

Knowledge check