The orchestrator — how ceph orch drives cephadm

Why this matters in production

ceph orch is how daemons are created, moved, upgraded, and removed on a cephadm cluster. It is declarative, which means it keeps acting after you stop typing — a property that is useful when understood and surprising when not.

The command surface

ceph orch host ls
ceph orch host add ceph-07 10.0.1.17 --labels osd
ceph orch host label add ceph-07 mon

ceph orch ls                              # services
ceph orch ps                              # daemons
ceph orch ps --daemon-type osd --host ceph-07

ceph orch apply mon --placement="3 ceph-01 ceph-02 ceph-03"
ceph orch apply -i service-spec.yaml
ceph orch daemon restart osd.12
ceph orch daemon rm mgr.ceph-02.abcdef
ceph orch upgrade start --ceph-version 20.2.1
ceph orch upgrade status

Two levels matter: services are the declared intent, daemons are the running instances. ceph orch ls shows what should exist; ceph orch ps shows what does.

Reconciliation

The orchestrator continuously compares declared services against running daemons and acts on the difference:

flowchart LR
  S[Service spec] --> R{Reconcile}
  D[Running daemons] --> R
  R -->|"missing"| C[Create daemon]
  R -->|"extra"| X[Remove daemon]
  R -->|"matches"| N[No action]

So removing a daemon that a service still declares causes it to be recreated. To remove something permanently, change or remove the service, not the daemon.

ceph orch rm mon              # removes the service declaration
ceph orch daemon rm mon.x     # removes one daemon; may be recreated

Service specifications

service_type: rgw
service_id: default
placement:
  hosts:
    - ceph-04
    - ceph-05
spec:
  rgw_frontend_port: 8080
networks:
  - 10.0.2.0/24

ceph orch apply -i rgw-spec.yaml
ceph orch ls --service-type rgw --export

--export prints the current spec, which is how you capture what a cluster is actually declaring — worth storing in version control.

Working with it

• Keep service specs in version control and apply from files rather than command-line flags, so the intent is reviewable.
• Prefer explicit host lists for monitors and managers, where placement is a design decision.
• Use labels and patterns for things where “wherever it fits” is genuinely correct, such as node-exporter.
• Check ceph orch ls before adding hosts, so you know what will be provisioned automatically.

Quiz

Production discipline

Treat service specs as policy documents kept in version control and applied from files, since they keep acting long after the command returns. Use explicit host lists for monitors and managers where placement is a failure-domain decision, and reserve labels and counts for daemons where “wherever it fits” is genuinely right. When the orchestrator appears stalled, check ceph orch status for a paused module before anything else — a cancelled upgrade is the usual cause.

Cross-course references

• Ceph: Part XLIX (cephadm) for the full deployment surface.
• Ceph: Part XCVIII (Software Upgrades) for orchestrator-driven upgrades.
• Ceph: Part XCI (Adding Storage Nodes) for host addition in practice.