The OSD process — one daemon, many responsibilities

Foundation⏱ ~15 mincephcephadm

What you'll learn

Enumerate the concurrent responsibilities of an OSD daemon
Explain why OSD CPU and memory usage vary so widely
Locate and read OSD logs and admin socket state
Recognise which responsibility is consuming resources

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

An OSD is not a simple storage server. It is doing six things at once, and when one host’s OSDs are consuming unexpected CPU or memory, the question is always which of the six.

The six jobs

Serve client I/O for PGs where it is primary, and participate as a replica for others.
Replicate — forward writes to peers and wait for their commits.
Peer — agree with other OSDs on the authoritative state of each PG after any change.
Recover and backfill — copy objects to or from other OSDs to restore redundancy.
Scrub — compare replicas for consistency, shallow or deep.
Report — heartbeats to peers, statistics to the manager, state to the monitors.

All six happen concurrently, on shared thread pools, against the same device.

ceph daemon osd.12 status
ceph daemon osd.12 dump_ops_in_flight
ceph daemon osd.12 perf dump
cephadm logs --name osd.12 --since 1h

dump_ops_in_flight is the fastest way to see what an OSD is doing

When an OSD is slow and you want to know why:

ceph daemon osd.12 dump_ops_in_flight | jq '.ops[] | {description, age, "event": .type_data.events[-1].event}'

Each in-flight operation reports its age and its current stage. The stage names are informative: waiting for rw locks, waiting for subops from, started, commit_sent. An operation stuck at waiting for subops from [47] names the OSD it is waiting on, which turns “the cluster is slow” into “osd.47 is slow” in one command.

This must be run on the host where the OSD lives, or through cephadm shell --name osd.12.

Why resource usage varies

CPU is dominated by whichever job is active. A quiet OSD serving occasional reads uses very little. The same OSD during backfill, with deep scrub running and a peering storm from a topology change, can saturate several cores.

Memory is bounded but variable:

ceph config get osd osd_memory_target
ceph daemon osd.12 dump_mempools | jq '.mempool.by_pool | keys'

osd_memory_target (4 GiB by default) is a target that the OSD manages its caches against, not a hard limit. Actual RSS can exceed it, particularly with many PGs or long PG logs.

What consumes memory

Consumer	Grows with
BlueStore cache (RocksDB block cache, onode cache)	`osd_memory_target`
PG logs	PG count × `osd_max_pg_log_entries`
Peering state	PG count, spikes during peering
In-flight operations	client concurrency
Backfill reservations	concurrent backfills

The PG log contribution is the one that surprises people: with 200 PGs per OSD and 10,000 log entries each, that is a large structure held permanently.

Why an OSD host needs more RAM than osd_memory_target × OSD count

The common sizing error is to multiply osd_memory_target by the number of OSDs and provision exactly that.

Three things break it. osd_memory_target governs the caches the OSD manages, not total process memory — PG logs, peering structures, and in-flight operations sit outside it. Peering spikes memory substantially and transiently, and peering happens on every OSD at once after a map change. And the host needs memory for the kernel, the container runtime, and page cache.

A practical rule is to allow osd_memory_target plus roughly 1 GiB per OSD, plus several GiB for the host, and to leave headroom for the peering spike. Getting this wrong produces the worst failure mode in Ceph operations: the OOM killer terminating OSDs during a recovery, which causes more recovery, which causes more peering, which triggers more OOM kills.

Reading the logs

cephadm logs --name osd.12 --since 2h | grep -E 'slow|osd_op|heartbeat|ENOSPC'

Lines worth recognising:

Log fragment	Meaning
`slow request N seconds old`	an operation exceeded the complaint threshold
`heartbeat_check: no reply from`	cannot reach a peer
`map e12345 wrongly marked me down`	the OSD was marked down while alive — usually network
`_open_alloc opening allocation metadata`	normal startup
`bluestore(...) allocation stats`	normal periodic reporting

Quiz

Knowledge check · 4 questions

Q1. An OSD is slow and you need to know why in one command. What gives the most direct answer?

ceph osd perf, which reports the commit and apply latency of every OSD in the cluster ceph pg dump, which lists every PG the OSD holds along with its state and acting set ceph daemon osd.N dump_ops_in_flight, which names the stage each operation is stuck at ceph daemon osd.N perf dump, which reports the daemon's internal latency and throughput counters
Q2. Provisioning host RAM as osd_memory_target multiplied by the OSD count is adequate sizing.

True False
Q3. OSDs on one host are being killed by the OOM killer during recovery, causing further recovery. Break the cycle.

Host with 12 OSDs and 64 GB RAM. osd_memory_target is the default 4 GiB, so 48 GiB was considered sufficient. A disk failure elsewhere started backfill; during it, three OSDs on this host were OOM-killed, which marked them down, which increased recovery work, which triggered more kills. Pool size 3, min_size 2. Some PGs are now below min_size.
Q4. List the six concurrent jobs an OSD daemon performs and explain why they make resource usage variable.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Size OSD host memory as osd_memory_target plus roughly 1 GiB per OSD plus several GiB for the host, with headroom for the peering spike that hits every OSD at once after a map change. Reach for dump_ops_in_flight when an OSD is slow — the stage names name the thing being waited on. And treat OOM-killed OSDs during recovery as an urgent feedback loop: reduce memory demand and throttle recovery first, because each kill generates the map change that causes the next.

Cross-course references

Ceph: Part XII (BlueStore) for where the memory actually goes.
Ceph: Part LXVIII (OSD Latency) for the latency side of the same daemon.
Ceph: Part LXXIX (Slow Ops) for reading in-flight operations in anger.