Recover from multiple OSD failures

This runbook covers Recover from multiple OSD failures on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (data-loss-risk)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• Multiple OSDs are down (possibly on different hosts)
• A cascade event has happened

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph osd tree shows which OSDs are down
# ceph -s reports the cluster health
# The CRUSH map is readable
# Capacity headroom can absorb the recovery

Procedure

Step 1

Identify the cascade cause

Step 2

Assess which PGs are at risk

Step 3

Do NOT mark OSDs out unnecessarily

Step 4

Wait for the cascade source to heal

Step 5

Recover the OSDs (Part LVII)

Step 6

Verify the cluster: ceph -s is HEALTH_OK

Verification

After completing the procedure, verify the result:

# The cluster recovers to HEALTH_OK
# All affected PGs return to active+clean
# Capacity is recovered

Rollback

If the procedure does not produce the expected verification, roll back:

• If the cascade is persistent: identify the root cause
• If the cluster cannot recover: see the disaster-recovery runbook

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team
• Hardware vendor
• Disaster recovery team