Storage Metrics

The capacity alert fires at 04:00: “datastore rpool at 92%”. The on-call opens the dashboard. The capacity number is fine - 92% is well below the team’s alert threshold - and yet the page says 92%. The real reason for the page is elsewhere. The capacity number was right an hour ago, but the underlying NFS export has been returning 4-second latency on writes for two hours; the customer started seeing slow disk inside the VMs thirty minutes ago and the dashboard for the datastore still shows a thin green line. Capacity is healthy; latency is not.

This lesson is about storage metrics. Proxmox exposes a capacity- shaped surface through the API; the latency and IOPS metrics live underneath, in the storage stack itself. A monitoring stack that reads only the API is blind to 80% of the storage incidents an operator will see.

What it is

“Storage metrics” in Proxmox covers two surfaces:

• The Proxmox API surface. pve_storage_* series from pve-exporter. Datastore-level capacity, used, content type (image, backup, ISO, etc.), and reachable state. The capacity number and the up/down signal.
• The under-storage surface. Latency and IOPS from the filesystem or block layer that backs the datastore. ZFS has zpool_* from the zfs_exporter; Ceph has ceph_* from the Ceph mgr Prometheus module; NFS has client-side latency from node_nfs_dircache_* or prometheus-node-exporter’s mount metrics; iSCSI has multipath and per-LUN latency from the kernel.

Proxmox does not unify these. A capacity alert can pass while a latency alert screams. The dashboards must combine them.

Why a sysadmin cares

Storage is the most common cause of customer-visible degradation in Proxmox environments. A network glitch is visible at the network panel; a CPU regression at the host panel; a storage regression typically appears first as customers reporting slow applications, with the per-VM metrics showing nothing obviously wrong.

The metrics that answer “is storage OK?” must:

• Show latency, not just throughput. A datastore that has not yet saturated its capacity can be saturated at the latency level already.
• Distinguish sync from async write latency when sync writes are on the critical path (database logs, NFS with sync).
• Cover all four stack choices. ZFS, Ceph, NFS, and iSCSI have different failure shapes.

How it works

   VM (qemu)
     |
     +-- virtio / SCSI driver
     |
   qemu on Proxmox node
     |
     +-- storage plugin (zfspool, ceph-rbd, nfs, iscsi)
     |
   filesystem / block layer (ZFS pool, Ceph RBD, NFS client,
                            iSCSI multipath)
     |
   network / local disk
     |
   backing store (disk array, NFS server, Ceph cluster)

The exporter reads the Proxmox API call /storage/\{storage\}/ status and emits pve_storage_total_bytes, pve_storage_used_ bytes, pve_storage_avail_bytes, and pve_storage_status. None of those is a latency number. The latency lives below.

   +-- zfs_exporter:           zfs_pool_*, zfs_arc_*
   +-- ceph mgr prom module:   ceph_osd_op_*, ceph_pg_*
   +-- node_exporter:          node_filesystem_*, mount metrics
   +-- custom iSCSI exporter:  per-LUN latency, multipath paths

A production stack combines both.

Under the hood

How to configure it

The Proxmox exporter side:

# /etc/pve-exporter/pve.yml
# SEVERITY: CONFIGURATION
default:
  api_url: https://pve-01.example.lan:8006/api2/json
  api_token: "monitoring-pve@pam!prometheus"
  api_token_value: "${PVE_EXPORTER_TOKEN}"
  verify_ssl: true
  modules:
    cluster: true
    node: true
    storage: true
    guests: true
    backup: true
  # Datastores. A small cluster exposes a handful; a thousand-
  # VM deployment can expose fifty. Trim what no dashboard
  # consumes.
  storage_filter: ""
  timeout: 10

The under-storage exporters (representative; each has its own form):

# /etc/prometheus/prometheus.yml (excerpt)
# SEVERITY: CONFIGURATION
scrape_configs:
  - job_name: zfs_exporter
    static_configs:
      - targets: ['pve-01.internal:9123', 'pve-02.internal:9123']
  - job_name: ceph_mgr
    metrics_path: /metrics
    static_configs:
      - targets: ['ceph-mgr-01.internal:9283']
  - job_name: nfs_client_textfile
    static_configs:
      - targets: ['pve-01.internal:9100', 'pve-02.internal:9100']
        labels: {collector: 'nfs_client'}

The Prometheus rules that turn latency into alerts:

# /etc/prometheus/rules/storage-latency.rules.yml
# SEVERITY: CONFIGURATION
groups:
- name: storage-latency
  interval: 30s
  rules:
  # NFS read p99 above 200 ms for 10 minutes.
  - alert: NFSReadLatencyHigh
    expr: |
      histogram_quantile(0.99,
        sum by (mountpoint) (rate(node_nfs_client_read_latency_seconds_bucket[5m]))
      ) > 0.2
    for: 10m
    labels:
      severity: ticket
      team: virtualization
      component: nfs
    annotations:
      summary: 'NFS read p99 above 200 ms on {{ $labels.mountpoint }}'
      description: |
        Check the NFS server for I/O pressure; check the network
        path between pve host and NFS server. Sustained high
        read latency means the customer-visible application
        latency is degraded.
      runbook_url: 'https://runbooks.example.com/storage/nfs-latency'

  # Ceph OSD apply latency p99 above 100 ms.
  - alert: CephOSDApplyLatencyHigh
    expr: |
      histogram_quantile(0.99,
        sum by (osd) (rate(ceph_osd_op_apply_latency_ms_bucket[5m]))
      ) > 100
    for: 10m
    labels:
      severity: page
      team: virtualization
      component: ceph
    annotations:
      summary: 'Ceph OSD {{ $labels.osd }} apply latency above 100 ms'

  # ZFS pool fragmentation above 60% is a future-state alert.
  - alert: ZFSPoolFragmented
    expr: |
      zfs_pool_fragmentation_percent > 60
    for: 1h
    labels:
      severity: ticket
      team: virtualization
      component: zfs
    annotations:
      summary: 'ZFS pool {{ $labels.pool }} fragmented above 60%'

How to validate it

Top-level capacity from the Proxmox exporter:

# SEVERITY: READ-ONLY
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=pve_storage_total_bytes - pve_storage_used_bytes' \
  | jq

Expected: a non-negative series per datastore.

Match against the source of truth:

# SEVERITY: READ-ONLY
pvesm status
df -h /rpool
zpool list
ceph status

The values should reconcile to within a small margin (the exporter is polled; pvesm status is live).

Latency validation against the underlying stack:

# SEVERITY: READ-ONLY
# ZFS write latency at the vdev level, in nanoseconds.
zpool iostat -v rpool 5 2

# NFS client read latency on the mount used for VM disks.
cat /proc/self/mountinfo | grep -E 'nfs'
nfsstat -c

# Ceph OSD apply latency, in milliseconds.
ceph osd perf

How it can fail

Six failure modes recur in production storage metrics:

1. NFS stall from hard mount. Network blip causes the kernel to keep retrying; latency metrics show seconds-to-minutes; the VM’s qemu process blocks inside the kernel. Symptom: every metric for VM disks reads zero because pvestatd cannot sample; a remount by umount -f && mount -a clears it.
2. Ceph PG degraded with healthy capacity. A disk fault triggered backfill; the capacity read is correct but the write-path is degraded. Symptom: ceph_pg_state{state="degraded"} > 0; ceph_health_status is not “OK”.
3. ZFS fragmentation above 60% on a write-heavy pool. Long writes become slow; the latency grows by an order of magnitude without capacity changes. Symptom: zfs_pool_fragmentation_percent > 60; the application sees slow disk; the capacity alert does not fire.
4. iSCSI multipath failover storm. A switching fault causes traffic to flip between paths at sub-second intervals. Symptom: a custom exporter reports multipath_path_changes rising; per-VM disk read latency oscillates.
5. Stale metrics after a path failover. The NFS or iSCSI exporter does not refresh the latency counters for the new path. Symptom: the latency panel shows a flat line at the old value; new samples are off by the failover gap.
6. Capacity drift between Proxmox and the underlying stack. A thin-provisioned datastore over-reports capacity to Proxmox. Symptom: the dashboard says 90% used; the underlying ZFS pool or Ceph OSD is at 95%; alerts fire late; the customer feels it earlier.

How to troubleshoot it

The order is: confirm the latency with a ground-truth tool, identify the storage stack, work the stack.

1. Latency with ground truth. iostat -x 1, zpool iostat -v 1, ceph osd perf, NFS client metrics from nfsstat. These run independently of the exporter.
2. Identify the storage stack. Proxmox’s per-VM disk target reports the stack in the VM configuration (qm config \{vmid\}); the storage level is documented.
3. Work the stack. Each stack has its own first response: ZFS, scrub and reclaim; Ceph, recovery and backfill; NFS, check network path; iSCSI, check multipath.
4. Check for staleness. If the ground truth disagrees with the exporter, the exporter is stale; restart it.
5. Reconcile the dashboards. Capacity in Proxmox, latency in the under-storage exporter, end-to-end latency from the guest. Three panes, one incident.

Security implications

• The exporter exposes datastore names; an attacker who gets the exporter’s /metrics can map the storage topology.
• The NFS and Ceph API tokens must be in scope only for read metrics; a token with write privilege on Ceph can corrupt the cluster.
• iSCSI exports belong on a dedicated network. Storage metrics that expose iSCSI target names are reconnaissance-grade.

Performance implications

• pve_storage_* cardinality is 1 * datastores. Tiny.
• The under-storage exporters vary. zfs_exporter per host is ~50 series per pool. ceph_mgr is ~6000 series for a hundred-OSD cluster. NFS and node_exporter are small.
• The heavy one is Ceph at scale. 6000 series is fine in Prometheus; 60,000 is not. Filter Ceph by osd_id or aggregate through recording rules for very large clusters.

Production guidance

• Latency first, capacity second. The customer cares about latency; the planner cares about capacity; the alert that pages first is latency.
• Reconcile three panes: Proxmox capacity, under-storage latency, guest-side end-to-end. The first disagreement is the symptom; reconcile to find the cause.
• Wire alerts per stack: ZFS on pool fragmentation and sync write latency; Ceph on OSD apply latency p99; NFS on client read latency p99; iSCSI on multipath changes.
• For sync-write-sensitive workloads (databases), alert on sync latency specifically. The async number hides the problem.

Verification

You should now be able to answer:

• Why does pve_storage_used_bytes alone under-report storage incidents?
• Where do ZFS, Ceph, NFS, and iSCSI latency metrics originate
  • all from the Proxmox API, or from the underlying stack?
• Why does the alert threshold for ZFS sync-write latency differ from NFS read latency?
• What three-pane reconciliation does an operator need to make when a storage incident starts?

Quiz