The host and network dashboard

Advanced⏱ ~17 mingrafanaprometheusnode_exporter

What you'll learn

Design a host-level dashboard for Ceph investigation
Include the network panels that matter for Ceph
Correlate host metrics with Ceph metrics
Identify the host-level causes of Ceph symptoms

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A large share of Ceph problems are host or network problems presenting as storage symptoms. This dashboard is where that connection is made.

The host panels

CPU
  Utilisation      1 - avg by (instance)(rate(node_cpu_seconds_total{mode="idle"}[5m]))
  Steal            avg by (instance)(rate(node_cpu_seconds_total{mode="steal"}[5m]))
  Load average     node_load1, node_load5, node_load15
  Run queue        node_procs_running

Memory
  Available        node_memory_MemAvailable_bytes
  Swap used        node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes
  OOM kills        rate(node_vmstat_oom_kill[5m])
  Page faults      rate(node_vmstat_pgmajfault[5m])

Disk
  I/O time         rate(node_disk_io_time_seconds_total[5m])
  Queue depth      rate(node_disk_io_time_weighted_seconds_total[5m])
  Read/write rate  rate(node_disk_reads_completed_total[5m])
  Await            rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])

The network panels

Throughput       rate(node_network_receive_bytes_total[5m]) * 8
Errors           rate(node_network_receive_errs_total[5m])
Drops            rate(node_network_receive_drop_total[5m])
Retransmits      rate(node_netstat_Tcp_RetransSegs[5m])
Bond members     node_bonding_active / node_bonding_slaves
MTU              node_network_mtu_bytes
Link speed       node_network_speed_bytes

Panel	Ceph symptom it explains
Retransmits	slow ops, OSD flapping
Drops	incast, latency spikes
Bond members below total	reduced capacity during recovery
Throughput near link speed	recovery slower than expected
MTU mismatch across hosts	large transfers failing

The bond member panel deserves a threshold: a bond running on fewer members than configured is a silent capacity reduction.

node_bonding_active < node_bonding_slaves

Correlating with Ceph

Sharing a template variable between the Ceph and host panels is what makes correlation immediate:

Variable: host
Query:    label_values(ceph_osd_metadata, hostname)

# Ceph panel
avg by (hostname) (
  ceph_osd_apply_latency_ms
    * on(ceph_daemon) group_left(hostname) ceph_osd_metadata
){hostname=~"$host"}

# host panel
node_load1{instance=~"$host.*"}

Both on the same time axis, filtered by the same host, answers “is this OSD slow because its host is loaded” without leaving the dashboard.

The host-level causes

Host observation	Ceph consequence
CPU steal high	OSDs slow; virtualised host oversubscribed
Swap in use	severe OSD latency; OSDs should never swap
Major page faults	memory pressure; check `osd_memory_target`
Disk I/O time near 1	device saturated
Retransmits rising	network loss; slow ops follow
Load far above core count	too many OSDs per host, or another workload

Quiz

Knowledge check · 4 questions

Q1. Why is CPU steal particularly damaging for OSD hosts and hard to diagnose?

It stalls the OSD long enough for BlueStore to abort a transaction, leaving metadata that the next scrub reports as an inconsistency It delays the kernel network stack rather than the OSD process, so heartbeats are dropped on the wire while the daemon itself continues running normally It affects only the monitors, whose Paxos rounds are timing-sensitive, while cephadm pins OSD threads to cores the hypervisor cannot take The OSD is runnable but not scheduled, causing millisecond-scale delays, and steal appears as neither high utilisation nor high load inside the guest
Q2. Occasional swap usage on an OSD host is acceptable if memory pressure is only intermittent.

True False
Q3. Investigate OSDs slow on one host with normal device metrics.

All OSDs on one host show elevated latency. Device SMART is clean, device I/O time is low, and the network shows no errors. The hosts are virtual machines.
Q4. Why does the bond member panel need a threshold on `node_bonding_active < node_bonding_slaves`?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Share a host template variable between the Ceph and host panels so “is this OSD slow because its host is loaded” is answered without leaving the dashboard. Threshold-colour swap usage and CPU steal at any non-zero value — both cause severe Ceph latency and neither is visible from within Ceph.

Cross-course references

Kubernetes: node-level metrics alongside workload metrics serve the same correlation
Linux: steal time and swap are the classic invisible causes of application slowness