OSD memory and threads — where the resources go

Why this matters in production

OSD memory is the resource most often undersized, and the failure mode — the OOM killer terminating OSDs during recovery — is self-amplifying. Understanding what the settings actually bound prevents it.

osd_memory_target

ceph config get osd osd_memory_target
ceph config set osd osd_memory_target 8589934592     # 8 GiB
ceph config set osd/host:ceph-07 osd_memory_target 4294967296

It is a target for the caches BlueStore manages, not a limit on process memory. The OSD periodically measures its own footprint and adjusts cache sizes to approach the target.

What it governs:

• RocksDB block cache
• BlueStore onode cache (object metadata)
• BlueStore buffer cache (data)

What it does not govern:

• PG logs
• peering state
• in-flight client operations
• backfill and recovery reservations
• allocator metadata

ceph daemon osd.12 dump_mempools | jq '.mempool.by_pool | to_entries | map({(.key): .value.bytes}) | add'

That output is the honest picture: it shows which pools are consuming memory, including the ones outside the target.

The shard and thread model

An OSD divides its work across shards, each with a queue and worker threads:

ceph config get osd osd_op_num_shards
ceph config get osd osd_op_num_threads_per_shard

Operations are assigned to shards by PG, so a shard handles a subset of PGs and its threads process that subset’s queue. Defaults differ by device class — more shards for SSD and NVMe than for HDD, because fast devices can absorb more concurrency.

Raising shard counts increases parallelism and memory footprint, and it only helps when the daemon is the bottleneck rather than the device. This is the same measurement question as osds_per_device, and in most cases adding OSDs is a more effective answer than retuning shards.

Diagnosing memory pressure

ceph daemon osd.12 dump_mempools | jq '.mempool.by_pool.osd_pglog.bytes'
ceph health detail | grep -i memory
dmesg -T | grep -i 'out of memory'
ceph osd df tree                        # PG count per OSD

The sequence: confirm OOM kills in dmesg, check PG count per OSD, check osd_pglog in mempools, and only then consider lowering osd_memory_target — which reduces cache and therefore performance, and is a mitigation rather than a fix.

Tuning with evidence

1. Measure the peak, not the average — during peering.
2. If PG count per OSD is high, fix that first.
3. If PG logs dominate, consider lowering osd_max_pg_log_entries, accepting more backfill instead of recovery.
4. Only then adjust osd_memory_target, and prefer adding RAM.

Quiz

Production discipline

Size OSD hosts for the peering peak, not the steady state, and verify by restarting an OSD and watching the host memory peak. Check PG count per OSD before touching memory settings — far above 200 is itself the cause, and the fix is at pool level. Read dump_mempools rather than guessing which structure is consuming memory. And treat lowering osd_memory_target as a mitigation that costs cache hit rate, with adding RAM as the actual answer.

Cross-course references

• Ceph: Part XVIII (Placement Groups) for getting PG counts right.
• Ceph: Part XX (PG Peering) for what peering does with that memory.
• Ceph: Part XII (BlueStore) for the caches the target governs.