VM Resource Metrics

A dashboard panel reads “VM 104 CPU at 99%” but the customer reports the VM is responsive and serving requests in normal time. The panel target is wrong: it is showing the host’s CPU (the qemu process is busy serving block I/O to the guest) not the guest’s CPU. The right metric - what the guest’s operating system believes about its own CPU

• is on a different panel. An engineer who does not know the distinction chases a phantom performance regression for an hour.

This lesson is about the per-VM metrics Proxmox publishes and where they come from. The exporter reports host-side series and guest-side series side by side, and the difference matters for every per-VM investigation.

What it is

Per-VM metrics are the time-series the exporter attaches to a single virtual machine or container:

• pve_guest_cpu_usage_ratio{vmid="104",node="pve-02"} - the host’s share of physical CPU consumed by the qemu process, as a ratio of one core.
• pve_guest_mem_usage_bytes and pve_guest_mem_max_bytes - the current allocated memory and the configured maximum, in bytes.
• pve_guest_disk_read_bytes and pve_guest_disk_write_bytes - the block-IO delta the host sees from qemu’s virtio or SCSI driver.
• pve_guest_network_in_bytes and pve_guest_network_out_bytes - the host-side tap interface counter.

These are host-side series. They describe what the host believes about the VM, not what the VM believes about itself. The latter is available only when the guest runs an exporter of its own (node exporter, cAdvisor, or the OTel collector with a host metrics receiver).

Container (LXC) and VM (QEMU) guests use the same series; the VMID label distinguishes them.

Why a sysadmin cares

Per-VM metrics answer the questions that drive capacity and incident work:

• Capacity planning. Which VM is approaching its memory ceiling, and by when?
• Noisy neighbour. Which VM is consuming disproportionate IO on a shared datastore?
• Chargeback. Which team owns which VM, and what did it consume?
• Migrate window. Which VM is the right candidate to live-migrate given its current CPU profile?

Without per-VM metrics, the operator can answer “is the host OK” but not “which VM is hot”. The investigation falls back to logging into the guest and using top; that scales to two or three VMs, not to two hundred.

How it works

   qemu-kvm process              LXC runtime
        |                              |
        +-- virtio / SCSI driver       +-- cgroup v2 controller
        |                              |
   pvestatd (per-host agent, reads cgroup + libvirt)
        |
        +-- /var/lib/rrdcached/db/pve2-vm/<vmid>
        |   (RRD database, last 24 h at 1-minute resolution,
        |    last week at 1-hour resolution)
        |
        +-- /api2/json/nodes/{node}/qemu/{vmid}/status/current
        |
   pve-exporter translates JSON to Prometheus series
        |
        v
   Prometheus + Grafana

pvestatd is the per-host agent; it polls the qemu and LXC runtimes and writes into RRD files. The exporter reads those values through the API and re-exposes them as pve_guest_* series.

Under the hood

How to configure it

Most exporter forks enable guests by default. The production consideration is the cardinality budget.

# /etc/pve-exporter/pve.yml (excerpt)
# SEVERITY: CONFIGURATION
default:
  api_url: https://pve-01.example.lan:8006/api2/json
  api_token: "monitoring-pve@pam!prometheus"
  api_token_value: "${PVE_EXPORTER_TOKEN}"
  verify_ssl: true
  modules:
    cluster: true
    node: true
    storage: true
    guests: true
    backup: true
  # Restrict exported VMs to the production tag. Wildcard export
  # on a 500-VM cluster is 1500+ Prometheus series per scrape
  # and risks a cardinality incident; the tag filter keeps the
  # metrics surface to the dashboards the team owns.
  guest_filter: "production=true"
  timeout: 10

A Prometheus rule that uses per-VM labels for routing:

# /etc/prometheus/rules/vm-resource.rules.yml
# SEVERITY: CONFIGURATION
groups:
- name: pve-vm-resource
  interval: 30s
  rules:
  # Balloon driver absent: VM reports max memory as live, host
  # cannot reclaim. Pages on the cause, not on a memory metric.
  - alert: PVEGuestBalloonMissing
    expr: |
      pve_guest_mem_max_bytes > 0
      and on(vmid, node)
      pve_guest_mem_usage_bytes == pve_guest_mem_max_bytes
    for: 30m
    labels:
      severity: ticket
      team: virtualization
    annotations:
      summary: 'VM {{ $labels.vmid }} on {{ $labels.node }} balloon driver absent'
      description: |
        The host sees the VM using all its configured memory for
        over 30 minutes. The balloon driver is usually disabled
        or the agent inside the guest is stale. Open a ticket to
        re-enable ballooning or to right-size the VM.
      runbook_url: 'https://runbooks.example.com/pve/balloon-missing'

How to validate it

A specific VMID returns specific series:

# SEVERITY: READ-ONLY
curl -s http://pve-exporter.internal:9221/metrics \
  | grep -E '^pve_guest_(cpu_usage_ratio|mem_usage_bytes|mem_max_bytes|disk_(read|write)_bytes|network_(in|out)_bytes)\{.*vmid="104".*\}' \
  | head

The expected lines on a healthy VM:

pve_guest_cpu_usage_ratio{cluster="prod",node="pve-02",type="qemu",vmid="104"} 0.34
pve_guest_mem_usage_bytes{cluster="prod",node="pve-02",type="qemu",vmid="104"} 4.21e+09
pve_guest_mem_max_bytes{cluster="prod",node="pve-02",type="qemu",vmid="104"} 8.59e+09
pve_guest_disk_read_bytes{cluster="prod",node="pve-02",type="qemu",vmid="104"} 1.23e+10
pve_guest_disk_write_bytes{cluster="prod",node="pve-02",type="qemu",vmid="104"} 4.50e+09
pve_guest_network_in_bytes{cluster="prod",node="pve-02",type="qemu",vmid="104"} 9.87e+10
pve_guest_network_out_bytes{cluster="prod",node="pve-02",type="qemu",vmid="104"} 6.10e+10

Side-by-side with the guest-side view when the VM runs node_exporter:

# SEVERITY: READ-ONLY
# pve-exporter shows 8.6 GiB max; the guest sees only 4.6 GiB live.
ssh vm-104 "free -m | awk 'NR==2 {print \$3, \$2}'"
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=pve_guest_mem_usage_bytes{vmid="104"}' \
  | jq '.data.result[].value[1]'

The two numbers must reconcile: pvestatd’s number is the host’s view; the guest’s free is the guest’s view; the gap, if any, is the balloon driver at work.

How it can fail

Five failure modes recur in production per-VM metrics:

1. Stopped VM reporting zero for every series. A VM is off, pvestatd has stopped polling it, and the exporter returns pve_guest_status == 0 plus zeros for every other metric. Symptom: dashboards show flat-zero lines for the VM; an alert on pve_guest_cpu_usage_ratio == 0 for 1h fires falsely if the VM is intended to be off.
2. Balloon driver absent. The guest kernel has no virtio_balloon module, or the qemu commandline disabled it. Symptom: pve_guest_mem_usage_bytes == pve_guest_mem_max_bytes sustained; the host cannot reclaim memory; real pressure becomes harder to detect.
3. Stale values after a live migration. The VM moved from pve-02 to pve-03 with qm migrate. Symptom: the exporter briefly reports the VM on both nodes for one scrape, then only on the new node; an alert with the wrong node label fires during the gap.
4. High cardinality from a missing tag filter. A 500-VM cluster without guest_filter produces 500 * (cpu + mem + disk + network) * 2 (R + W) = 4000+ series. Symptom: Prometheus TSDB head growth stalls; older series stop ingest.
5. Mismatch with guest-side exporter. The VM runs node_exporter and reports 50% CPU while the host reports 2% CPU. Symptom: the panels contradict; the investigation must reconcile pve_guest_cpu_usage_ratio (host share of one core) against the guest’s node_cpu_seconds_total (guest-side busy time).

How to troubleshoot it

The order is: is the VM running, is pvestatd publishing, is the exporter scraping, is the guest reporting.

1. Is the VM running? pvesh get /nodes/\{node\}/qemu/\{vmid\}/ status/current | jq .status returns “running” or “stopped”. Stopped VMs export zero for every metric by design.
2. Is pvestatd publishing? ls -l /var/lib/rrdcached/db/pve2-vm/{vmid} on the VM’s node. The RRD file exists for any VM that has ever run on this node; check that the timestamp is recent: rrdtool lastupdate /var/lib/rrdcached/db/pve2-vm/{vmid} reports now-ish.
3. Is the exporter scraping? curl http://pve-exporter: 9221/metrics | grep vmid=... shows the series. Missing means the role or filter is wrong.
4. Is the guest reporting? If the guest runs node_exporter, ssh \{vm\} -- curl http://localhost:9100/metrics | grep node_ cpu returns the guest-side truth. Mismatch with the host’s series implies balloon state or virtio misbehaviour.
5. Reconcile. A two-pane dashboard that puts host and guest CPU side by side is the operator’s friend. The pve_guest_cpu_usage_ratio panel and the guest’s node_cpu_ seconds_total panel on the same chart make the difference obvious.

Security implications

• The exporter exposes the VM name (pve_guest_info{vmid=...}) for every exported VM. A token with PVEAuditor is enough to see who owns what. Restrict the exporter’s /metrics to the internal network.
• VMIDs are predictable integers in a known range. Treat them as identifiers but not as secrets.
• The node= label on per-VM series exposes the topology. An attacker scanning the exporter can map the cluster from / metrics alone.

Performance implications

Per-VM metrics are the largest single contributor to exporter cardinality. The order of magnitude:

• pve_guest_info: 1 series per guest (constant).
• CPU: 1 series per guest.
• Memory: 2 series per guest.
• Disk: 2 series per guest (read + write).
• Network: 2 series per guest (in + out).

Total: ~8 series per running guest. A 200-VM cluster is ~1600 series; a 500-VM cluster is ~4000; a 2000-VM cluster is ~16,000 series just from pve_guest_*.

Three mitigations:

1. guest_filter by tag. Production-only by default; everything else on demand.
2. scrape_interval of 30 seconds or 60 seconds. The VM CPU signal does not need 5-second resolution.
3. Recording rules. Pre-aggregate noisy series (sum by (service)(pve_guest_cpu_usage_ratio)) so dashboards do not pull thousands of raw series every refresh.

Production guidance

• Default guest_filter to a meaningful tag. “Everything” is a cardinality and rate-limit incident in waiting.
• Pair host-side pve_guest_* with guest-side metrics (node exporter or OTel) on the same dashboard when the VM is critical.
• Alert on pve_guest_status first; CPU, memory, and IO second; user-facing SLI third.
• Reconcile by VM. Build the runbook around “the host says X, the guest says Y, the reconciliation is Z”.

Verification

You should now be able to answer:

• What does pve_guest_cpu_usage_ratio actually measure - host, guest, or both?
• Why is a flat-zero panel not necessarily an incident?
• What is the cardinality cost of an unbounded guest_filter, and what is the right default?
• Why does a balloon-disabled VM inflate the host’s reporting?

Quiz