Prometheus node_exporter: collecting host and VM metrics

node_exporter is the canonical way to expose PVE host metrics to Prometheus. With the qemu-guest-agent on each VM and the built-in PVE exporter, you have full visibility from the host kernel all the way into the guest.

Installing node_exporter on every node

apt install -y prometheus-node-exporter
systemctl enable --now prometheus-node-exporter
systemctl status prometheus-node-exporter
# Active: active (running)
# Listening on :9100
curl -s http://localhost:9100/metrics | head -5
# # HELP go_gc_duration_seconds A summary of the GC invocation durations.
# # TYPE go_gc_duration_seconds summary
# go_gc_duration_seconds{quantile="0"} 1.6e-05

The default configuration exposes over 600 metrics covering CPU, memory, disk, network, filesystem, kernel, and more. The full list is at http://localhost:9100/metrics.

For most clusters, the defaults are fine. Tune only if you have specific concerns:

# /etc/default/prometheus-node-exporter
ARGS="--collector.filesystem.mount-points-exclude='^/(sys|proc|dev|host|run)($|/)' \
      --collector.netclass.ignored-devices='^(veth|docker|br-).*' \
      --collector.diskstats.device-exclude='^(loop|ram|dm-).*'"

systemctl restart prometheus-node-exporter

The mount-points-exclude and device-exclude reduce noise from container and LVM devices that aren’t useful to monitor at the host level.

Collecting VM metrics with qemu-guest-agent

The host can see CPU and memory for a VM but not what’s happening inside the guest. qemu-guest-agent runs inside the VM and reports guest-side metrics.

Inside the VM

# Debian/Ubuntu
apt install -y qemu-guest-agent
systemctl enable --now qemu-guest-agent

# RHEL/CentOS
dnf install -y qemu-guest-agent
systemctl enable --now qemu-guest-agent

# Alpine
apk add qemu-guest-agent
rc-service qemu-guest-agent start
rc-update add qemu-guest-agent

On the PVE host

Enable the guest agent on the VM:

qm set 100 --agent enabled=1
# Or via the GUI: VM → Options → QEMU Guest Agent → Enable

# Verify the agent is connected
qm agent 100 ping
# (no output = success)
qm agent 100 get-fsinfo
# Lists mounted filesystems from inside the guest

With the agent enabled, PVE’s pvestatd automatically collects guest metrics and exposes them via the PVE API:

pvesh get /nodes/pve-01/qemu/100/status/current
# {
#   "cpu": 0.05,
#   "cpus": 2,
#   "disk": 0,
#   "diskread": 12345,
#   "diskwrite": 67890,
#   "maxdisk": 10737418240,
#   "maxmem": 1073741824,
#   "mem": 536870912,
#   "netin": 12345,
#   "netout": 67890,
#   "status": "running",
#   "uptime": 3600,
#   ...
# }

The pve-exporter for Prometheus

The pve-exporter (community-maintained) scrapes the PVE API and exposes per-VM and per-container metrics in Prometheus format:

# /opt/pve-exporter
wget https://github.com/inovex/pve-exporter/releases/latest/download/pve-exporter.linux-amd64
chmod +x pve-exporter.linux-amd64
mv pve-exporter.linux-amd64 /usr/local/bin/pve-exporter

# /etc/pve-exporter/pve-exporter.yaml
default:
  user: metrics@pve
  password: REPLACE_ME
  realm: pve
  verify_ssl: false

clusters:
  - url: https://pve-01.cluster.example.com:8006/api2/json
    name: production-cluster

Run as a service:

# /etc/systemd/system/pve-exporter.service
[Unit]
Description=PVE Exporter
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/pve-exporter \
  --config.file /etc/pve-exporter/pve-exporter.yaml \
  --web.listen-address=:9221
Restart=always

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now pve-exporter

Scrape from Prometheus:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: pve-nodes
    static_configs:
      - targets:
        - pve-01:9100
        - pve-02:9100
        - pve-03:9100

  - job_name: pve-cluster
    static_configs:
      - targets:
        - pve-exporter:9221

The essential PromQL queries

The fundamentals of PromQL for a PVE cluster:

CPU usage (per node)

# Total CPU usage, 5-minute average
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Per-core CPU usage (which core is hot)
rate(node_cpu_seconds_total{mode!="idle"}[5m]) * 100

Memory pressure

# Available memory in bytes
node_memory_MemAvailable_bytes

# Memory used percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

Disk pressure

# Disk space used percentage
(1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} /
      node_filesystem_size_bytes{fstype!~"tmpfs|overlay"})) * 100

# Disk I/O utilisation per device
rate(node_disk_io_time_seconds_total[5m]) * 100

Network throughput

# Receive rate per interface
rate(node_network_receive_bytes_total{device!~"lo|veth.*|docker.*"}[5m]) * 8

# Errors per interface
rate(node_network_receive_errs_total[5m])

Per-VM (via pve-exporter)

# Per-VM CPU usage
pve_guest_info{name="..."} * on(name) group_left(vm_name) pve_vm_info{name="..."}
# Or, with the standard exporter:
pve_cpu_usage_ratio{name="vm-name"}

# Per-VM memory usage
pve_memory_usage_bytes{name="vm-name"} / pve_memory_total_bytes{name="vm-name"}

# Per-VM disk IOPS
rate(pve_disk_read_bytes{name="vm-name"}[5m])

Building a Grafana dashboard

The standard “cluster overview” dashboard has:

Top row — cluster health

Cluster status (online / degraded)
Total nodes online / offline
Total VMs running / stopped
Ceph health (if applicable)

Second row — host metrics

CPU usage per node (line graph, last 24h)
Memory usage per node (stacked area)
Disk usage per node (gauge per critical filesystem)
Network throughput per node (stacked area)

Third row — VM metrics

CPU usage top 10 VMs (bar chart)
Memory usage top 10 VMs
Disk I/O top 10 VMs
Network throughput top 10 VMs

Fourth row — alerts

Active alerts (alerts that are currently firing)

Save the dashboard as JSON in version control so it’s reproducible.

Common mistakes

Missing qemu-guest-agent. The PVE API returns host-side metrics but not guest CPU/memory/IO inside the VM. Install the guest agent for full visibility.
Collecting too many metrics. node_exporter exposes 600+ metrics. If you scrape at 1s intervals, that’s 36 GB/day of metric data per node. Scrape at 15s for production, 5s for troubleshooting.
Not setting up recording rules. Aggregating metrics on every query is slow. Use recording rules for common aggregates:
```
- record: cluster:cpu_usage:avg5m
  expr: avg(100 - (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))
```

Production considerations

High availability for the monitoring stack itself. If Prometheus is on a single node and that node fails, you have no monitoring. Run Prometheus on a separate small cluster or use thanos / cortex for cross-cluster aggregation.
Metric retention. Prometheus default is 15 days. Long-term storage (thanus, Mimir, VictoriaMetrics) keeps months or years of metrics for trend analysis.
Alert routing. Prometheus alerts go to Alertmanager. Route alerts to PagerDuty, Slack, email based on severity.
Dashboard discipline. A dashboard for every service, owned by the team that owns the service. Stale dashboards get ignored.

Key takeaways

node_exporter on every host, qemu-guest-agent in every VM, pve-exporter for VM/container metrics.
Scrape at 15s for production, 5s for troubleshooting.
Build standard dashboards: cluster health, host metrics, VM metrics.
Set up recording rules for common aggregates.