Declaring services with the orchestrator

Intermediate⏱ ~17 minceph

What you'll learn

Declare services with placement specifications
Explain the reconciliation model
Apply service specifications from files
Diagnose a service that is not reconciling

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The orchestrator is declarative: you state what should exist and it works continuously to make that true. Operators used to imperative deployment are surprised when a daemon they stopped comes back, and when a change they made by hand is reverted.

Declaring

ceph orch apply mon --placement="3 label:mon"
ceph orch apply mgr --placement="2 label:mgr"
ceph orch apply mds cephfs --placement="3 label:mds"
ceph orch apply rgw default --placement="4 label:rgw" --port=8080

ceph orch ls
ceph orch ls --service-name mon --export

Placement specifications

Specification	Meaning
`"3"`	three daemons, orchestrator chooses hosts
`"host1 host2 host3"`	exactly these hosts
`"label:mon"`	every host with the label
`"3 label:mon"`	three hosts chosen from those labelled
`"*"`	every host
`"host-pattern:ceph-osd-*"`	hosts matching the pattern

Label-based placement is the most maintainable: adding a labelled host extends the service automatically, and the labels document intent.

Specification files

service_type: rgw
service_id: default
placement:
  label: rgw
  count: 4
spec:
  rgw_frontend_port: 8080
  rgw_realm: default
  rgw_zone: default
---
service_type: mds
service_id: cephfs
placement:
  label: mds
  count: 3

ceph orch apply -i services.yaml
ceph orch ls --export > cluster-services.yaml

Exporting the current state to a file and keeping it in version control is what makes a cluster’s configuration reviewable and reproducible.

The reconciliation model

The orchestrator compares the declaration with reality on a cycle and acts on the difference:

A daemon you stop is restarted
A host that fails has its daemons redeployed elsewhere, subject to placement
A labelled host added gets the labelled services
A daemon removed by hand comes back

To remove a service you change the declaration:

ceph orch rm rgw.default
ceph orch apply rgw default --placement="2 label:rgw"   # reduce the count

When reconciliation stalls

ceph orch ls
# shows RUNNING count below the declared count

ceph log last cephadm
ceph health detail
ceph orch ps --refresh

Cause	Signature
No hosts match the placement	count 0/N with no candidate hosts
Image pull failing	daemons stuck in `starting`
Host unreachable	orchestrator log shows SSH errors
Insufficient resources	daemon starts and exits
Orchestrator paused	`ceph orch status` shows paused

Export service specifications into version control

ceph orch ls --export > cluster-services.yaml

The exported file is the cluster’s declared configuration. In version control it becomes reviewable, diffable, and reproducible — and it is the document that lets you rebuild the cluster’s service layout deliberately rather than from memory.

Quiz

Knowledge check · 4 questions

Q1. You stop a daemon with systemctl and it restarts a few minutes later. Why?

The orchestrator reconciliation loop observed a gap between the declaration and reality and closed it The systemd unit cephadm wrote carries Restart=always, so systemd brings it straight back The monitors noticed the daemon missing from the map and instructed the host to restart it A health check failed on the missing daemon and its remediation action restarted the unit
Q2. `ceph orch pause` is the right tool for hand intervention on a daemon, because it halts reconciliation without touching the declarations.

True False
Q3. Perform manual intervention on a daemon without the orchestrator interfering.

An engineer needs to stop an MDS daemon temporarily to capture diagnostics from its host while it is not running. Every time they stop it, it restarts within minutes.
Q4. Why is label-based placement more maintainable than naming hosts explicitly?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Export service specifications to version control so the cluster’s declared layout is reviewable and reproducible. Use ceph orch pause for manual intervention rather than fighting the reconciliation loop — it suspends without discarding, so nothing needs restoring afterwards.

Cross-course references

Kubernetes: this is the same declarative reconciliation model as controllers and Deployments
Linux: configuration management tools that enforce desired state behave identically