Skip to main content
RunBook Academy

← All runbooks in Ceph

critical riskdata loss risk~240 min

Execute the Ceph disaster-recovery procedure

1 · Prerequisites

Confirm every item is in place before any state change.

  • A disaster has happened
  • A DR procedure is documented
  • A backup target is accessible

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The disaster is declared
  • · The backup target is verified
  • · The replacement hardware is ready (or cloud capacity)
  • · The DR procedure is documented

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Isolate the affected cluster (or declare the cluster total loss)
  2. 2Provision the replacement cluster (or rebuild)
  3. 3Restore the MON store from backup
  4. 4Restore the OSDs (devices, CRUSH map, services)
  5. 5Restore the auth database (or rotate the keys)
  6. 6Restore the data from backup (RBD, CephFS, RGW)
  7. 7Verify: ceph -s reports HEALTH_OK
  8. 8Validate: the application is functional

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The replacement cluster is operational
  • ✓Data is restored from backup
  • ✓The cluster is HEALTH_OK
  • ✓The application is functional

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the backup is incomplete: prioritise restore by RTO target
  • ↶If a host is unable to receive OSDs: hardware failure; engage vendor
  • ↶If the application cannot start: application-layer DR

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Disaster recovery team
  • · Application team
  • · Vendor support

Execute the Ceph disaster-recovery procedure

This runbook covers Execute the Ceph disaster-recovery procedure on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (data-loss-risk)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A disaster has happened
  • A DR procedure is documented
  • A backup target is accessible

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# The disaster is declared
# The backup target is verified
# The replacement hardware is ready (or cloud capacity)
# The DR procedure is documented

Procedure

Step 1

Isolate the affected cluster (or declare the cluster total loss)

Step 2

Provision the replacement cluster (or rebuild)

Step 3

Restore the MON store from backup

Step 4

Restore the OSDs (devices, CRUSH map, services)

Step 5

Restore the auth database (or rotate the keys)

Step 6

Restore the data from backup (RBD, CephFS, RGW)

Step 7

Verify: ceph -s reports HEALTH_OK

Step 8

Validate: the application is functional

Verification

After completing the procedure, verify the result:

# The replacement cluster is operational
# Data is restored from backup
# The cluster is HEALTH_OK
# The application is functional

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the backup is incomplete: prioritise restore by RTO target
  • If a host is unable to receive OSDs: hardware failure; engage vendor
  • If the application cannot start: application-layer DR

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Disaster recovery team
  • Application team
  • Vendor support

References

  1. Ceph documentation