Skip to main content
RunBook Academy

← All runbooks in Ceph

high riskdata loss risk~120 min

Recover from a complete OSD host loss

1 · Prerequisites

Confirm every item is in place before any state change.

  • A host is lost entirely
  • A maintenance window is in progress

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph osd tree shows the affected hosts OSDs are down + in
  • · ceph -s reports the cluster is HEALTH_WARN with osds down
  • · The clusters capacity headroom can absorb the recovery
  • · Replacement host (or replacement disks) is available

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1After mon_osd_down_out_interval, the OSDs are marked out automatically
  2. 2Wait for backfill: ceph progress
  3. 3Decommission the host: ceph orch host rm <hostname>
  4. 4Provision the replacement
  5. 5Add the new host: ceph orch host add <hostname>
  6. 6Deploy OSDs on the new host
  7. 7Verify the cluster: ceph -s is HEALTH_OK

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓New host is in the orchestrator
  • ✓New OSDs are up + in
  • ✓Backfill completes
  • ✓Cluster is HEALTH_OK

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the new host cannot be added: verify the network, time, DNS, SSH
  • ↶If backfill stalls: tune recovery settings
  • ↶If the cluster stays HEALTH_WARN: identify the remaining failure

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Hardware vendor if the replacement hardware is suspect

Recover from a complete OSD host loss

This runbook covers Recover from a complete OSD host loss on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (data-loss-risk)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A host is lost entirely
  • A maintenance window is in progress

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph osd tree shows the affected hosts OSDs are down + in
# ceph -s reports the cluster is HEALTH_WARN with osds down
# The clusters capacity headroom can absorb the recovery
# Replacement host (or replacement disks) is available

Procedure

Step 1

After mon_osd_down_out_interval, the OSDs are marked out automatically

Step 2

Wait for backfill: ceph progress

Step 3

Decommission the host: ceph orch host rm <hostname>

Step 4

Provision the replacement

Step 5

Add the new host: ceph orch host add <hostname>

Step 6

Deploy OSDs on the new host

Step 7

Verify the cluster: ceph -s is HEALTH_OK

Verification

After completing the procedure, verify the result:

# New host is in the orchestrator
# New OSDs are up + in
# Backfill completes
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the new host cannot be added: verify the network, time, DNS, SSH
  • If backfill stalls: tune recovery settings
  • If the cluster stays HEALTH_WARN: identify the remaining failure

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Hardware vendor if the replacement hardware is suspect

References

  1. Ceph documentation