CephLXXIX · Slow OpsSlow Ops
What Ceph counts as a slow operation
What you'll learn
- Define a slow operation precisely
- Identify the settings that govern the threshold
- Read the operation lifecycle a slow op represents
- Interpret the counts and ages correctly
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
“Slow op” is a specific measurement with a configurable threshold, not a qualitative judgement. Knowing what it measures determines what a report of forty-seven of them means.
The definition
ceph config get osd osd_op_complaint_timeAn operation that an OSD has held for longer than osd_op_complaint_time
— thirty seconds by default — without completing is reported as slow. The
clock starts when the OSD receives the operation, not when the client
issued it.
ceph config get osd osd_op_log_threshold
ceph config get osd osd_max_scrubsWhat the report contains
[WRN] SLOW_OPS: 47 slow ops, oldest one blocked for 92 sec, daemons
[osd.12,osd.31] have slow ops.| Field | Meaning |
|---|---|
47 slow ops | operations currently held past the threshold |
oldest one blocked for 92 sec | the age of the longest-held operation |
daemons [...] | which OSDs are holding them |
The count is a current snapshot, not a total. Operations that completed after being slow are not counted, so a falling count means operations are draining.
The operation lifecycle
flowchart TD
A[Client sends op] --> B[OSD receives, clock starts]
B --> C[queued for pg]
C --> D[reached pg]
D --> E[started]
E --> F[waiting for subops from replicas]
F --> G[commit sent to client]
G --> H[apply or cleanup]Each stage is a flag_point, and the stage an operation is stuck at names
the subsystem responsible:
ceph daemon osd.12 dump_ops_in_flight | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("num_ops:", d.get("num_ops"))
for op in d.get("ops", [])[:5]:
td = op.get("type_data", {})
print(" age %6.1fs %s" % (op.get("age", 0), td.get("flag_point")))'Interpreting counts and ages
| Pattern | Reading |
|---|---|
| Count falling, age bounded | operations completing; transient |
| Count static, age growing | blocked, not slow |
| Count growing, age bounded | load exceeding capacity |
| Count growing, age growing | worsening; escalate |
| One daemon named | that OSD or something it waits on |
| Many daemons named | a shared cause — network, or one slow replica |
# sample twice to establish the trend
ceph health detail | grep -oE '[0-9]+ slow ops.*blocked for [0-9]+'
sleep 120
ceph health detail | grep -oE '[0-9]+ slow ops.*blocked for [0-9]+'Quiz
Knowledge check · 4 questions
Q1. When does the clock for `osd_op_complaint_time` start?
Q2. Lowering `osd_op_complaint_time` turns SLOW_OPS into a useful latency alert.
Q3. Interpret a slow ops report.
The first sample shows 47 slow ops with the oldest at 92 seconds. Two minutes later it shows 12 slow ops with the oldest at 45 seconds.
Q4. Why does the absence of SLOW_OPS not mean clients are not waiting?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Sample the slow ops count and the oldest age twice before acting;
both falling means the condition is resolving. Remember the measurement is
OSD-side only — clients can be waiting substantially with no SLOW_OPS
report, which is why client latency metrics are a separate signal.
Cross-course references
- Kubernetes: server-side request duration excludes client and network time identically
- Linux: a device queue time and an application wait time are different measurements