Reading the built-in graphs: RRD, retention and limits

The graphs on a node’s Summary tab are the most-viewed piece of monitoring in any Proxmox estate and the least examined. They are free, they are always there, they need no exporter, and they are the first thing anyone opens when a user says the word “slow”.

They are also a specific data structure with specific properties, and most of the mistakes people make with them come from not knowing what that structure does to the data on its way in.

What is actually stored

pvestatd runs on every node, collects statistics, and writes them into round-robin databases through rrdcached. The files live under /var/lib/rrdcached/db/, one per node, per guest and per storage.

PVE 9 changed both the layout and the schema. The directory naming moved from pve{version}-{type} to pve-{type}-{version}:

# ls -la /var/lib/rrdcached/db/

total 24
drwxr-xr-x  8 root root 4096 Aug 12 09:14 .
drwxr-xr-x  3 root root 4096 May 21 11:02 ..
drwxr-xr-x  2 root root 4096 Aug 12 09:14 pve-node-9.0
drwxr-xr-x  2 root root 4096 Aug 12 09:14 pve-storage-9.0
drwxr-xr-x  2 root root 4096 Aug 12 09:14 pve-vm-9.0
drwxr-xr-x  2 root root 4096 May 21 11:02 pve2-node
drwxr-xr-x  2 root root 4096 May 21 11:02 pve2-storage
drwxr-xr-x  2 root root 4096 May 21 11:02 pve2-vm

Resolution and retention, and why it matters

PVE 9 aligned all three object types on the same aggregation schema that Proxmox Backup Server uses. Four windows, each holding both an average and a maximum series:

Read that table as a statement about what you can prove. A CPU spike that lasted four minutes is fully visible today, is one thirty-minute average by tomorrow, and is one six-hour average by next month. It has not been deleted — it has been averaged into near-invisibility.

set -euo pipefail
NODE=$(hostname -s)
RRD="/var/lib/rrdcached/db/pve-node-9.0/$NODE"

# Confirm which data sources and consolidation functions this file holds.
rrdtool info "$RRD" | grep -E '^ds\[|rra\[[0-9]+\]\.cf' | head -40

# Thirty days of six-hour AVERAGEs - what the GUI shows you.
rrdtool fetch "$RRD" AVERAGE --start end-30d --resolution 21600 | head -20

# The same window as MAX - what actually happened inside each bucket.
rrdtool fetch "$RRD" MAX --start end-30d --resolution 21600 | head -20

What PVE 9 added, and why it is the most useful change

The schema gained columns, and the new ones are the interesting part.

Nodes gained memfree, arcsize, and four kernel pressure-stall metrics: pressurecpusome, pressureiosome, pressureiofull, pressurememorysome.

Guests gained memhost — memory consumption as the host sees it, which is not the same number the guest reports — and six pressure metrics of their own, including pressurecpufull and pressurememoryfull.

Storage is unchanged: total and used.

arcsize is a small addition with a large payoff on ZFS: the ARC is frequently the largest single consumer of host memory and its size against your VM allocation is now a native graph rather than something you had to go and measure.

The pressure metrics matter more. Pressure Stall Information is a kernel facility that reports the proportion of time tasks were stalled waiting for a resource, rather than how much of the resource was consumed. some means at least one task was waiting; full means every runnable task was waiting.

The migration that gets skipped

New columns and finer aggregation mean bigger files: a node RRD grows from roughly 79 KB to 1.4 MB, a guest RRD from 66 KB to 1.3 MB. On a cluster with a few thousand guests that is a real amount of disk, which is why the migration is a separate, explicit step rather than something the package upgrade does silently.

pve8to9 checks that there is space for it. After the upgrade, a separate tool converts the old files into the new schema.

set -euo pipefail

# Dry run first. No arguments beyond this: it reports what it would convert.
/usr/libexec/proxmox/proxmox-rrd-migration-tool

# Then the real thing. Threads default to between 1 and 4 based on core
# count; roughly 10,000 guest files take a few minutes single-threaded.
/usr/libexec/proxmox/proxmox-rrd-migration-tool --migrate

# Confirm the new tree is populated and the old sources are marked .old.
ls /var/lib/rrdcached/db/pve-node-9.0/
ls /var/lib/rrdcached/db/pve2-vm/ | head

What each graph actually measures

The labels are shorter than the definitions, and three of them are routinely misread.

The four questions built-in metrics cannot answer

This is the section that decides whether you need anything else. The built-in graphs are excellent at what they do and structurally incapable of four things:

1. “What was the 99th percentile latency?” RRD stores averages and maxima of gauges. There are no histograms, so there are no percentiles, and an average latency figure hides exactly the tail that users notice.
2. “Which process caused that spike?” The data is per-node, per-guest and per-storage. There is no dimension below a guest, so the answer to “what inside VM 104 did that” is not in this data set at any resolution.
3. “Show me all guests where memory exceeded 90% last week.” There is no query language. The files are per-object and the GUI shows one object at a time. Answering a fleet-wide question means iterating over files yourself.
4. “What did this look like eighteen months ago at 14:05?” The archive window holds weekly averages after a year. The shape of a Tuesday afternoon is not recoverable.

Each of those is a reason to export to an external system, and each is a better reason than “we want nicer dashboards”. If none of the four questions matter to you, the built-in graphs are genuinely sufficient and adding a Prometheus stack is work you do not need.

set -euo pipefail
NODE=$(hostname -s)

# Peak memory fraction per guest over the last week, from the node's own API.
for vmid in $(qm list | awk 'NR>1 {print $1}'); do
peak=$(pvesh get "/nodes/$NODE/qemu/$vmid/rrddata" \
         --timeframe week --cf MAX --output-format json \
       | grep -o '"mem":[0-9.]*' | cut -d: -f2 | sort -g | tail -1)
maxmem=$(qm config "$vmid" | awk '/^memory:/ {print $2}')
printf '%-6s peak_mem_bytes=%s configured_mib=%s\n' "$vmid" "${peak:-unknown}" "$maxmem"
done

Note the --cf MAX in that call. The API exposes the consolidation function, and defaulting to AVERAGE here would answer a different and less useful question.

Common mistakes

• Reading the year graph as if it had the resolution of the day graph. It is six-hour and then weekly averages.
• Plotting only AVERAGE. MAX is stored for every window and is where saturation is visible.
• Skipping the PVE 8 to 9 RRD migration, then deleting pve2-* as cleanup and losing the history permanently.
• Reading load average without dividing by core count.
• Trusting guest memory on a VM with no balloon driver.
• Trying to derive per-process or percentile answers from a data set that contains neither.
• Assuming /var/lib/rrdcached is backed up. It usually is not, and RRD history exists in exactly one place.

Key takeaways

• PVE 9 stores 1-minute data for a day, 30-minute for a month, 6-hour for a year and weekly for a decade, with AVERAGE and MAX for each.
• Consolidation happens on write. A spike that has aged out of the fine window cannot be recovered from a coarse one.
• The new pressure and arcsize series answer “is it suffering”, which utilisation never could.
• The RRD migration after an 8 to 9 upgrade is a separate step and silently costs you all pre-upgrade history if skipped.
• Built-in metrics cannot give percentiles, sub-guest detail, fleet-wide queries or multi-year fidelity. Those four, and not aesthetics, are the reason to export.

Knowledge check