Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskservice affecting~30 min

Add an OSD to an existing host

1 · Prerequisites

Confirm every item is in place before any state change.

  • A new disk is installed
  • The disk is at least as large as the smallest existing OSD
  • The cluster has enough headroom for backfill

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph -s is HEALTH_OK
  • · ceph osd tree shows the target host
  • · ceph orch device ls --hostname=<host> shows the new disk
  • · Time and DNS are healthy

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1ceph orch daemon add osd <host>:<device>
  2. 2OR ceph-volume lvm create --data <device> --bluestore
  3. 3Verify the new OSD comes up: ceph osd tree
  4. 4Wait for backfill: ceph progress
  5. 5Verify the PG distribution is healthy: ceph osd df tree

4 · Verification

Confirm the procedure actually fixed the problem.

  • ceph osd tree shows the new OSD up + in
  • Backfill completes
  • PGs return to active+clean
  • ceph df shows the new usable capacity

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • If the OSD fails to create: verify the device is healthy
  • If backfill is slow: tune recovery settings
  • If the OSD comes up briefly then crashes: device failure suspected

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Hardware vendor if the disk fails immediately

Add an OSD to an existing host

This runbook covers Add an OSD to an existing host on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A new disk is installed
  • The disk is at least as large as the smallest existing OSD
  • The cluster has enough headroom for backfill

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s is HEALTH_OK
# ceph osd tree shows the target host
# ceph orch device ls --hostname=<host> shows the new disk
# Time and DNS are healthy

Procedure

Step 1

ceph orch daemon add osd <host>:<device>

Step 2

OR ceph-volume lvm create —data <device> —bluestore

Step 3

Verify the new OSD comes up: ceph osd tree

Step 4

Wait for backfill: ceph progress

Step 5

Verify the PG distribution is healthy: ceph osd df tree

Verification

After completing the procedure, verify the result:

# ceph osd tree shows the new OSD up + in
# Backfill completes
# PGs return to active+clean
# ceph df shows the new usable capacity

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the OSD fails to create: verify the device is healthy
  • If backfill is slow: tune recovery settings
  • If the OSD comes up briefly then crashes: device failure suspected

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Hardware vendor if the disk fails immediately

References

  1. Ceph documentation