Tuning OSD latency

Advanced⏱ ~18 minceph

What you'll learn

Identify the settings that affect OSD latency
Distinguish current settings from removed ones
Change threading and queueing safely
Recognise when tuning cannot help

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

OSD tuning advice circulating online is largely obsolete, and applying it produces configuration that appears tuned while changing nothing. Knowing which settings still exist is the first requirement.

The current settings

ceph config get osd osd_op_num_shards
ceph config get osd osd_op_num_threads_per_shard
ceph config get osd osd_op_num_shards_hdd
ceph config get osd osd_op_num_shards_ssd
ceph config get osd osd_client_message_cap
ceph config get osd osd_op_queue
ceph config get osd osd_mclock_profile

Setting	Effect
`osd_op_num_shards`	independent operation queues per OSD
`osd_op_num_threads_per_shard`	worker threads serving each shard
`osd_client_message_cap`	in-flight client messages accepted
`osd_op_queue`	the scheduler: `mclock_scheduler` or `wpq`
`osd_mclock_profile`	the capacity allocation under mClock
`osd_memory_target`	BlueStore cache size per OSD

The device-class variants (_hdd, _ssd) take precedence, which is why setting the generic value often has no effect.

Removed or superseded settings

Setting	Status
`osd_op_threads`	removed; replaced by shards and threads-per-shard
`osd_disk_threads`	removed
`filestore_*`	irrelevant; FileStore is gone
`osd_client_op_priority`	WPQ only, no effect under mClock
`osd_op_thread_timeout`	still present, but a watchdog rather than a tuning knob

Any guide recommending osd_op_threads or filestore_* predates the current architecture entirely.

Changing threading

# more shards means more parallelism and more memory
ceph config set osd osd_op_num_shards_ssd 8
ceph config set osd osd_op_num_threads_per_shard_ssd 2

These require an OSD restart to take effect:

SERVICE=service
HOST=stor-04
ceph orch restart osd.${SERVICE}
# or, per host, using maintenance mode
ceph orch host maintenance enter ${HOST}

Total worker threads per OSD is shards × threads_per_shard, and the product must fit the host’s CPU budget across all its OSDs:

12 OSDs × 8 shards × 2 threads = 192 worker threads on one host

On a 32-core host that is oversubscribed, and the result is context switching rather than throughput.

Memory

ceph config get osd osd_memory_target
ceph config set osd osd_memory_target 8589934592   # 8 GiB

More cache means fewer device reads for metadata, which is often the largest single latency improvement available on HDD-backed OSDs. The constraint is host memory across all its OSDs plus headroom.

When tuning cannot help

Condition	Why tuning fails
The device is at its service-time limit	queueing changes do not make the device faster
CPU is saturated on the host	more threads make it worse
The network is the bottleneck	OSD threading is irrelevant
A single failing device	replacement, not configuration

# rule these out first
IFACE=bond0
iostat -x 1 5
vmstat 1 5
ethtool -S ${IFACE} | grep -i err

Quiz

Knowledge check · 4 questions

Q1. Why does Ceph shard the OSD operation queue rather than using one queue with many threads?

A single queue's lock becomes the bottleneck at high rates; sharding gives each shard its own queue and lock It keeps reads and writes in separate queues, so a burst of writes cannot delay reads on the same PG It lets each pool be given its own queue, which is how mClock allocates capacity between pools It reduces memory, since each shard sizes its own queue independently instead of one queue holding every operation
Q2. `osd_op_threads` is the current setting for OSD worker thread count.

True False
Q3. Evaluate a proposal to raise OSD threading.

A proposal raises osd_op_num_shards to 8 and osd_op_num_threads_per_shard to 4 to reduce latency. Hosts have 12 OSDs each and 32 CPU cores.
Q4. Name four conditions under which OSD tuning cannot improve latency.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Compute total worker threads as shards × threads-per-shard × OSDs per host before raising either; the product oversubscribes a host quickly and past saturation more threads raise latency. Treat osd_op_threads or any filestore_* setting in a guide as proof it predates the current architecture.

Cross-course references

Kubernetes: thread pool sizing per container must fit the node’s CPU allocation
Linux: oversubscribing worker threads past core count produces context switching