Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskservice affecting~45 min

Add a new OSD host to a running cluster

1 · Prerequisites

Confirm every item is in place before any state change.

  • The new host meets the prerequisites (Part LI)
  • The cluster has capacity headroom for the new hosts contribution

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The new host passes cephadm check-host
  • · The cluster is HEALTH_OK
  • · The new host is reachable on the cluster network
  • · Capacity headroom is verified

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1ceph orch host add <hostname> <ip>
  2. 2ceph orch host ls verifies the host is adopted
  3. 3ceph orch daemon add osd.<host>; deploy OSDs on the host
  4. 4ceph osd tree shows the new host
  5. 5ceph progress shows recovery activity

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓ceph -s reports HEALTH_OK
  • ✓ceph osd tree shows the new OSDs on the new host
  • ✓Recovery completes; PGs return to active+clean

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the OSDs do not come up: journalctl -u ceph-osd@<id>
  • ↶If recovery overwhelms client IO: throttle osd_recovery_max_active
  • ↶If the host is added but no OSDs deploy: ceph orch daemon ls osd.<host>

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team if multiple OSDs fail to deploy
  • · Network team if the cluster network is unreachable
  • · Vendor support if hardware fails

Add a new OSD host to a running cluster

This runbook covers Add a new OSD host to a running cluster on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • The new host meets the prerequisites (Part LI)
  • The cluster has capacity headroom for the new hosts contribution

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# The new host passes cephadm check-host
# The cluster is HEALTH_OK
# The new host is reachable on the cluster network
# Capacity headroom is verified

Procedure

Step 1

ceph orch host add <hostname> <ip>

Step 2

ceph orch host ls verifies the host is adopted

Step 3

ceph orch daemon add osd.<host>; deploy OSDs on the host

Step 4

ceph osd tree shows the new host

Step 5

ceph progress shows recovery activity

Verification

After completing the procedure, verify the result:

# ceph -s reports HEALTH_OK
# ceph osd tree shows the new OSDs on the new host
# Recovery completes; PGs return to active+clean

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the OSDs do not come up: journalctl -u ceph-osd@<id>
  • If recovery overwhelms client IO: throttle osd_recovery_max_active
  • If the host is added but no OSDs deploy: ceph orch daemon ls osd.<host>

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team if multiple OSDs fail to deploy
  • Network team if the cluster network is unreachable
  • Vendor support if hardware fails

References

  1. Ceph documentation