Reading cluster state through the orchestrator

Intermediate⏱ ~16 mincephjq

What you'll learn

Use orch ls, ps, host ls, and device ls effectively
Distinguish declared from running state
Filter and format output for automation
Diagnose the gap between declared and actual

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The orchestrator’s view and the cluster’s view are different things, and the gap between them is where problems live. Each command answers a specific question and knowing which to reach for saves the guessing.

The four commands

ceph orch ls — what is declared, and how much of it is running

ceph orch ls

NAME          PORTS   RUNNING  REFRESHED  AGE  PLACEMENT
mgr                       2/2  4m ago     3w   label:mgr
mon                       5/5  4m ago     3w   label:mon
osd.hdd                 96/96  4m ago     3w   label:osd
rgw.default   *:8080      3/4  4m ago     2w   label:rgw;count:4

3/4 on the last row is the whole point of the command: one gateway is declared and not running.

ceph orch ps — individual daemons and their state

ceph orch ps
ceph orch ps --daemon-type osd --hostname ceph-osd-05
ceph orch ps --refresh

NAME             HOST         STATUS         REFRESHED  MEM USE  VERSION
osd.12           ceph-osd-01  running (3w)   4m ago     3821M    19.2.0
rgw.default.a    ceph-rgw-02  error          4m ago        -     19.2.0

ceph orch host ls — hosts and their labels

ceph orch host ls
ceph orch host ls --label osd

ceph orch device ls — devices and their availability

ceph orch device ls
ceph orch device ls --wide
ceph orch device ls --refresh

Declared versus running

# what should exist
ceph orch ls --format json | jq -r '.[] | "\(.service_name) \(.status.size)"'

# what does exist
ceph orch ps --format json | jq -r '.[] | "\(.daemon_name) \(.status_desc)"'

# what the cluster itself sees
ceph -s
ceph mon stat
ceph osd stat

The three views can disagree, and each disagreement means something:

Disagreement	Means
Declared > running	reconciliation blocked or in progress
Running > cluster sees	daemon running but not joined — check its log
Cluster sees > running	stale cluster state, or an unmanaged daemon

That last case matters on clusters with daemons deployed outside the orchestrator: they appear in ceph -s and not in ceph orch ps.

Refreshing

ceph orch ps --refresh
ceph orch device ls --refresh

The orchestrator caches host state and refreshes on a cycle. The REFRESHED column shows the cache age, and --refresh forces a collection — worth doing before trusting device inventory after a hardware change.

Quiz

Knowledge check · 4 questions

Q1. `ceph orch ls` shows a service as 3/4. What does that mean?

Three of the four hosts carrying that service are reachable to the orchestrator Four daemons are declared and three are running — reconciliation is blocked or in progress The service is running at 75% of the capacity its placement specification asked for Three daemons run the current image and one is still on the previous version
Q2. Orchestrator output reflects the live state of each host at the moment the command runs.

True False
Q3. Investigate a daemon count discrepancy.

`ceph -s` reports 5 monitors. `ceph orch ps --daemon-type mon` lists 4. All five are serving and the cluster is healthy.
Q4. Why does the orchestrator cache host state rather than collecting it per command?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read the REFRESHED column before acting on orchestrator inventory, and force a refresh after physical work — the cached view is the default and its staleness is invisible otherwise. Compare the orchestrator view against ceph -s periodically; daemons visible to one and not the other are a management gap that surfaces at upgrade time.

Cross-course references

Kubernetes: desired versus current replica counts express the same distinction
Linux: cached inventory in any configuration management tool has the same staleness property