Monitoring Ceph and PBS

Node monitoring is well understood: CPU, memory, disk, network, and you know roughly what bad looks like. Ceph and PBS are different, and they are different in the same way — both degrade for a long time before they fail, and neither degradation is visible in the metrics an operator habitually watches.

A Ceph cluster that is one disk away from blocking writes has normal CPU, normal memory and normal network. A PBS datastore that has not produced a restorable backup for one guest in six weeks has normal disk usage and a green dashboard. In both cases the useful signal is a specific number that nobody looks at unless somebody told them to, and this lesson is that somebody.

Ceph: the numbers that predict rather than report

HEALTH_OK is not a monitoring strategy. It is a summary of health checks that have already tripped. The useful monitoring is upstream of it.

# ceph -s

  cluster:
  id:     8f2c1b44-9d3a-4e57-b1c0-2a7e5d9f0c31
  health: HEALTH_WARN
          1 nearfull osd(s)
          Degraded data redundancy: 4127/1893456 objects degraded (0.218%)

services:
  mon: 3 daemons, quorum pve-01,pve-02,pve-03 (age 4w)
  mgr: pve-01(active, since 4w), standbys: pve-02
  osd: 12 osds: 11 up (since 22m), 12 in (since 3w)

data:
  pools:   2 pools, 289 pgs
  objects: 631.15k objects, 2.4 TiB
  usage:   7.3 TiB used, 10.5 TiB / 17.8 TiB avail
  pgs:     271 active+clean
           18  active+undersized+degraded

io:
  client:   14 MiB/s rd, 42 MiB/s wr, 812 op/s rd, 1.902k op/s wr
  recovery: 128 MiB/s, 31 objects/s

Reading it line by line:

• 11 up (since 22m), 12 in. One OSD is down but still marked in, meaning Ceph still expects it to hold data. Once mon_osd_down_out_interval elapses (10 minutes by default, unless noout is set) it is marked out and recovery begins in earnest. The gap between up and in is where you still have a choice about whether to fix the disk or let the cluster heal around it.
• active+undersized+degraded. These PGs have fewer replicas than the pool requires. They are still serving I/O. undersized means fewer OSDs than size; degraded means fewer copies than required. Neither is an outage. inactive, incomplete or down in that list would be.
• usage: 7.3 TiB used, 10.5 TiB avail. This is the number that will mislead you, and the next section is about why.
• MON quorum age 4w. A quorum age that keeps resetting means monitors are flapping, which is a network or clock problem and a much more serious finding than a nearfull OSD.

set -euo pipefail

# Per-OSD utilisation, with the spread visible. The STDDEV line at the
# bottom is a direct measure of how unbalanced the cluster is.
ceph osd df tree

# The single number to alert on.
ceph osd df --format json \
| grep -o '"utilization":[0-9.]*' \
| cut -d: -f2 | sort -g | tail -1

# What the thresholds are set to on this cluster.
ceph config get mon mon_osd_nearfull_ratio
ceph config get mon mon_osd_full_ratio
ceph config get osd osd_backfillfull_ratio

# Per-pool usage and, more usefully, MAX AVAIL - what a pool can still take
# given its replication and the fullest OSD in its CRUSH tree.
ceph df detail

MAX AVAIL in ceph df detail is the honest capacity figure. It already accounts for replication and for the fullest OSD in the pool’s failure domain, and it is usually a good deal smaller than the number an operator computes by dividing raw free space by three.

The health checks worth alerting on individually

HEALTH_WARN is too coarse to page on and too important to ignore. Alert per check:

set -euo pipefail

# Per-OSD commit latency. Sort and look at the tail.
ceph osd perf

# Which health checks are currently firing, with detail rather than summary.
ceph health detail

# Deep scrub age - how long since each PG was fully verified. Falling behind
# here means bit rot would not be detected.
ceph pg dump pgs 2>/dev/null | awk 'NR>1 {print $1, $23, $24}' | head -20

# Confirm the suspect disk from the OS side before condemning it.
OSD_DISK=/dev/sdf
smartctl -a "$OSD_DISK" | grep -Ei 'reallocated|pending|uncorrect|error rate'

PBS: the metrics do not measure what you care about

PBS supports the same external metric server model as PVE, configured under Configuration → Metric Server. It exports host metrics — memory, network, disk activity — and datastore usage.

It does not export job outcomes.

set -euo pipefail
DATASTORE=pbs-main
MAX_AGE_HOURS=36

# Every backup group and the timestamp of its most recent snapshot.
proxmox-backup-manager datastore list

# Per-group freshness. Anything older than the threshold is a finding,
# whether or not any job ever reported an error.
proxmox-backup-client snapshot list \
--repository "root@pam@localhost:$DATASTORE" \
--output-format json \
| grep -o '"backup-time":[0-9]*' | cut -d: -f2 | sort -n | tail -1 \
| while read -r newest; do
    age=$(( ( $(date +%s) - newest ) / 3600 ))
    echo "newest snapshot is ${age}h old (threshold ${MAX_AGE_HOURS}h)"
    [ "$age" -le "$MAX_AGE_HOURS" ] || echo 'STALE - investigate'
  done

What to watch on the PBS host

set -euo pipefail
DATASTORE=pbs-main

# Datastore usage from the PBS side.
proxmox-backup-manager datastore list

# Did garbage collection run, and did it succeed?
proxmox-backup-manager garbage-collection status "$DATASTORE"

# Recent tasks, including failures. This is where verify and prune outcomes
# are visible from the CLI.
proxmox-backup-manager task list --limit 50

# Bytes and inodes. On ext4 or XFS the second one can run out first.
df -h  /mnt/datastore
df -i  /mnt/datastore

# The underlying pool, if the datastore is on ZFS.
zpool status -x
zpool list -o name,size,alloc,free,capacity,fragmentation

Setting thresholds that give you time

The recurring theme across both subsystems is that the default instinct — alert when the thing is nearly broken — leaves no room to act. Both Ceph and PBS have a remediation step that itself needs resources, and if you alert at the point of exhaustion, the remediation is unavailable.

Common mistakes

• Alerting on HEALTH_WARN as a single condition. It is either constantly firing and therefore ignored, or suppressed and therefore useless. Alert per health check.
• Using cluster-average capacity. The fullest OSD is the constraint.
• Reading SLOW_OPS as load. It is usually one dying disk.
• Treating PG_NOT_DEEP_SCRUBBED as cosmetic. It sets the size of the window between corruption and detection.
• Believing a PBS metrics dashboard reports backup success. It reports host and datastore statistics only.
• Monitoring only for job failure, so a job that stopped being scheduled is invisible.
• Monitoring bytes and not inodes on a non-ZFS chunk store.
• Leaving a pool at size=2, min_size=1 and never auditing for it.

Key takeaways

• Ceph’s usable capacity is set by the fullest OSD, not the average, and writes stop cluster-wide at 0.95 on any one OSD.
• The gap between backfillfull (0.90) and full (0.95) is where the cluster loses the ability to heal itself.
• Alert on individual health checks with individual urgencies; HEALTH_WARN alone is not actionable.
• SLOW_OPS plus ceph osd perf finds the one bad disk behind “everything is slow”.
• Deep scrub lag determines whether corruption is caught inside your backup retention window.
• PBS metrics carry host and datastore statistics and no job outcomes. Backup success comes from notifications; backup presence comes from checking snapshot freshness yourself.
• Set thresholds so that the remediation is still possible when they fire.

Knowledge check