CephXLIX · cephadmcephadm
Adding and managing hosts
What you'll learn
- Add a host to an orchestrated cluster
- Diagnose a host that will not be added
- Place a host into and out of maintenance
- Remove a host safely
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Adding a host is one command that depends on four things being right on that host. Knowing which four turns a failure from a mystery into a checklist.
Prerequisites
| Requirement | Check |
|---|---|
| SSH access from the cluster | ssh cephadm@newhost from a manager host |
| The cluster’s public key installed | ceph cephadm get-pub-key |
| Container runtime working | podman version on the host |
| Time synchronised | chronyc tracking |
| Hostname resolvable both ways | getent hosts newhost |
Adding
ceph cephadm get-pub-key > /tmp/ceph.pub
ssh-copy-id -f -i /tmp/ceph.pub cephadm@ceph-osd-05
ceph orch host add ceph-osd-05 10.20.0.15
ceph orch host add ceph-osd-05 10.20.0.15 --labels osd,rgw
ceph orch host ls
ceph orch ps --hostname ceph-osd-05Giving the address explicitly avoids relying on DNS resolution from every manager, which is one fewer thing to be wrong.
When a host will not add
ceph orch host add ceph-osd-05 10.20.0.15
# Error EINVAL: Failed to connect to ceph-osd-05
# work the checklist
ssh -i /etc/ceph/ceph.pub cephadm@ceph-osd-05 'hostname -f; podman version; chronyc tracking'
ceph log last cephadmceph log last cephadm is the orchestrator’s own log and usually names
the specific failure.
Maintenance mode
ceph orch host maintenance enter ceph-osd-05
# stops daemons and sets noout
# ... reboot, replace hardware, patch ...
ceph orch host maintenance exit ceph-osd-05Maintenance mode handles the flags and the daemon stops together, which is
what makes it preferable to doing each by hand — the forgotten noout is
the classic error otherwise.
ceph orch host maintenance enter ceph-osd-05 --force--force proceeds even where stopping the daemons would affect
availability. Read what it warns about before using it.
Removing a host
# 1. drain — moves daemons and OSD data off
ceph orch host drain ceph-osd-05
ceph orch osd rm status
# 2. wait for the drain to complete
ceph -s
ceph orch ps --hostname ceph-osd-05
# 3. remove
ceph orch host rm ceph-osd-05Draining removes the OSDs one at a time, letting data rebalance between each. On a host with many large OSDs that takes hours or days, and it must finish before the host is removed.
Quiz
Knowledge check · 4 questions
Q1. What does `ceph orch host maintenance enter` do beyond stopping daemons?
Q2. `ceph orch host rm` can be run immediately after `ceph orch host drain` is issued.
Q3. Retire a host from a production cluster.
A host with twelve 16 TB OSDs is being decommissioned. The cluster is at 76% capacity across 200 OSDs. The hardware must be returned within a week.
Q4. Why is `ceph log last cephadm` the right first command when a host will not be added?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Use maintenance mode rather than stopping daemons and setting flags
by hand; it combines the three steps and removes the forgotten-noout
error. Confirm remaining capacity before draining a host, and treat the
drain as an operation with a duration to be projected rather than a
command that returns.
Cross-course references
- Kubernetes: cordon and drain before node removal is the identical sequence
- Linux: any clustered service requires the same evacuate-then-remove discipline