Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskservice affecting~45 min

Add Ceph capacity safely

1 · Prerequisites

Confirm every item is in place before any state change.

  • A pool is at or near nearfull
  • Capacity needs to be added

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph -s reports the cluster is HEALTH_OK
  • · The new disks are verified healthy
  • · The new host meets the prerequisites
  • · A maintenance window is scheduled if needed

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Decide the addition: OSDs on existing host vs new host
  2. 2For new OSDs on an existing host: ceph orch daemon add osd <host>
  3. 3For a new host: ceph orch host add <hostname>; deploy OSDs
  4. 4Verify the new OSDs come up: ceph osd tree
  5. 5Wait for backfill: ceph progress
  6. 6Verify capacity headroom is restored: ceph df

4 · Verification

Confirm the procedure actually fixed the problem.

  • ceph -s reports HEALTH_OK
  • Backfill completes
  • Pool utilisation is below nearfull
  • ceph df shows the expected usable capacity

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • If OSDs do not deploy: check the device and the host
  • If backfill is slow: tune recovery settings
  • If a new OSD produces SMART errors: replace the disk

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team

Add Ceph capacity safely

This runbook covers Add Ceph capacity safely on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A pool is at or near nearfull
  • Capacity needs to be added

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s reports the cluster is HEALTH_OK
# The new disks are verified healthy
# The new host meets the prerequisites
# A maintenance window is scheduled if needed

Procedure

Step 1

Decide the addition: OSDs on existing host vs new host

Step 2

For new OSDs on an existing host: ceph orch daemon add osd <host>

Step 3

For a new host: ceph orch host add <hostname>; deploy OSDs

Step 4

Verify the new OSDs come up: ceph osd tree

Step 5

Wait for backfill: ceph progress

Step 6

Verify capacity headroom is restored: ceph df

Verification

After completing the procedure, verify the result:

# ceph -s reports HEALTH_OK
# Backfill completes
# Pool utilisation is below nearfull
# ceph df shows the expected usable capacity

Rollback

If the procedure does not produce the expected verification, roll back:

  • If OSDs do not deploy: check the device and the host
  • If backfill is slow: tune recovery settings
  • If a new OSD produces SMART errors: replace the disk

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team

References

  1. Ceph documentation