The per-OSD dashboard

Advanced⏱ ~17 mingrafanaprometheus

What you'll learn

Design a dashboard that surfaces OSD outliers
Distinguish device from host problems visually
Correlate OSD metrics with host metrics
Support the replace-or-investigate decision

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The overview says there is a slow OSD; this dashboard says whether it is the device, its host, or its DB device, which determines the action.

Surfacing outliers

Latency ranking     topk(15, ceph_osd_apply_latency_ms)
Latency heatmap     ceph_osd_apply_latency_ms — heatmap over time
Utilisation ranking topk(15, ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)
PG count ranking    topk(15, ceph_osd_numpg)
Ops ranking         topk(15, rate(ceph_osd_op_w[5m]) + rate(ceph_osd_op_r[5m]))

The heatmap is the panel that most repays the effort: it shows the whole fleet’s latency distribution over time as a density, so an outlier appears as a persistent band separated from the mass, and a cluster-wide event appears as the whole band shifting.

Grouping by host

sum by (hostname) (
  ceph_osd_apply_latency_ms
    * on(ceph_daemon) group_left(hostname) ceph_osd_metadata
) / count by (hostname) (
  ceph_osd_apply_latency_ms
    * on(ceph_daemon) group_left(hostname) ceph_osd_metadata
)

A per-host average latency panel is what distinguishes one bad device from one bad host in a single glance: the first shows one OSD elevated, the second shows every OSD on a host elevated together.

Grouping by device class

avg by (device_class) (
  ceph_osd_apply_latency_ms
    * on(ceph_daemon) group_left(device_class) ceph_osd_metadata
)

This gives the expected level per class, which is the reference an individual OSD should be compared against — an HDD at 12 ms is normal and an NVMe at 12 ms is not.

Correlating with host metrics

node_cpu_seconds_total, node_memory_MemAvailable_bytes,
node_disk_io_time_seconds_total, node_network_receive_errs_total

Placing these on the same dashboard, filtered by the same host variable, is what makes the correlation immediate rather than requiring a second dashboard and a second time-range selection.

Variable: osd     label_values(ceph_osd_metadata, ceph_daemon)
Variable: host    label_values(ceph_osd_metadata{ceph_daemon=~"$osd"}, hostname)

Supporting the decision

Pattern	Action
One OSD elevated, host metrics normal	device — check SMART, plan replacement
All OSDs on a host elevated	host — CPU, HBA, memory
All OSDs of one class elevated	class-wide load or a configuration change
One OSD elevated with high PG count	placement imbalance, not a fault
Latency correlates with `node_disk_io_time`	the device is saturated, not failing

Quiz

Knowledge check · 4 questions

Q1. What does a latency heatmap show that a topk ranking panel cannot?

Whether the rest of the fleet moved too — an outlier appears as a separated band, a cluster-wide event as the whole mass shifting The exact latency figure for each OSD, since a heatmap labels every cell while a ranking only names the worst fifteen The device class each OSD belongs to, because the heatmap buckets its series by the class label carried in ceph_osd_metadata rather than by daemon The PG count carried by each OSD, which is what separates an over-assigned OSD from one whose device is genuinely failing
Q2. An OSD with elevated latency and an unusually high PG count is likely to have a failing device.

True False
Q3. Characterise a latency outlier.

The overview shows osd.44 with elevated apply latency. The team is preparing to replace its device.
Q4. Why should an OSD's latency be compared against its device class average rather than the cluster average?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Include a latency heatmap alongside the topk ranking; it is the only panel that distinguishes a single outlier from a cluster-wide shift. Add per-host and per-device-class groupings and a PG count panel, which together separate a failing device from a host fault and from a simply over-assigned OSD.

Cross-course references

Kubernetes: node-grouped pod metrics distinguish node faults from workload ones
Linux: comparing a device against its class peers rather than a global average