CephLXXXIV · Proxmox Failure ScenariosProxmox Failure Scenarios
Diagnosing VM disk latency on Ceph
What you'll learn
- Trace VM disk latency through the layers
- Measure at each layer
- Identify which layer is responsible
- Apply the appropriate remedy
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
A guest reporting slow disks has five layers between its application and the OSD’s device. Measuring at each is what localises the delay.
The layers
flowchart TD
A[Guest application] --> B[Guest filesystem and block layer]
B --> C[virtio driver]
C --> D[QEMU / librbd]
D --> E[Proxmox host: CPU, network]
E --> F[Ceph public network]
F --> G[OSD]
G --> H[Device]Measuring at each
# guest: what the application sees
# inside the VM
iostat -x 1 5 /dev/vda
fio --name=g --filename=/dev/vda --direct=1 --rw=randread --bs=4k \
--iodepth=1 --runtime=30 --time_based --readonly# host: what librbd sees for this image
rbd perf image iostat --pool pve-vms# host resources
mpstat -P ALL 1 5
vmstat 1 5# cluster: what the OSDs report
ceph osd perf | sort -k2 -rn | head -5# device: what the disks do
# on an OSD host
iostat -x 1 5Five measurements. The layer where the number jumps is the layer responsible.
Interpreting the comparison
guest 45 ms → librbd 42 ms → OSD 3 ms → device 2 ms
→ the delay is between librbd and the OSD: network or hostguest 45 ms → librbd 4 ms → OSD 3 ms → device 2 ms
→ the delay is in the guest: driver, filesystem, or CPUguest 45 ms → librbd 42 ms → OSD 40 ms → device 38 ms
→ the device is the delay| Jump between | Cause |
|---|---|
| Application and guest block layer | filesystem, guest CPU |
| Guest and librbd | virtio, guest driver, host CPU |
| librbd and OSD | network, host CPU, client throttles |
| OSD and device | OSD queueing, scheduler, contention |
| Device is the level | the device itself |
The common guest-side causes
| Cause | Check |
|---|---|
No iothread on the disk | qm config <vmid> |
| Cache mode inappropriate | same |
discard absent, image heavily fragmented | same |
| Guest CPU saturated | guest mpstat |
| Guest queue depth of 1 | guest iostat aqu-sz |
| Guest filesystem barriers or mount options | guest mount |
| CPU steal on the host | host mpstat |
qm config 100 | grep -E 'scsi|virtio|cache|iothread|discard'Remedies by layer
# guest disk configuration
qm set 100 --scsi0 pve-vms:vm-100-disk-0,cache=writeback,discard=on,iothread=1,ssd=1
# host CPU
qm set 100 --cpulimit 8
# client-side concurrency
rbd config image set pve-vms/vm-100-disk-0 rbd_op_threads 4Quiz
Knowledge check · 4 questions
Q1. The guest reports 45 ms, librbd reports 4 ms, the OSDs report 3 ms. Where is the delay?
Q2. A VM with several disks and high CPU utilisation is where the absence of `iothread=1` shows up most.
Q3. Investigate a VM disk latency complaint.
A VM owner reports very slow disk performance. The Ceph cluster is HEALTH_OK with normal OSD latency.
Q4. What does a jump between librbd-observed and OSD-observed latency indicate?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Compare guest-observed latency against rbd perf image iostat before
anything else — one command splits the problem into guest-side and
everything-below. Check iothread=1, cache, and discard in the VM
configuration; their absence is a frequent and easily corrected cause.
Cross-course references
- Kubernetes: tracing latency through pod, node, and storage layers is the same method
- Linux: measuring at each layer to localise a delay is standard practice