Skip to main content
RunBook Academy

← All runbooks in Ceph

low riskcluster affecting~20 min

Backup Ceph configuration and keyrings

1 · Prerequisites

Confirm every item is in place before any state change.

  • A backup procedure is scheduled
  • A backup target is available

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph -s is HEALTH_OK
  • · The backup target is reachable
  • · A backup window is scheduled

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Backup the MON store: rsync -avz /var/lib/ceph/mon/ceph-<id> backup:/var/backups/ceph/mon/<id>/<date>/
  2. 2Backup the orchestrator config: /etc/ceph/, /var/lib/ceph/
  3. 3Backup the keyrings
  4. 4Backup the CRUSH map: ceph osd crush dump
  5. 5Backup the cluster map: ceph mon dump
  6. 6Verify the backups are restorable

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓Backups complete and are reachable
  • ✓Verification confirms the backup matches the live data
  • ✓Backup is in an independent failure domain

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the backup is corrupt: re-run the backup
  • ↶If the backup is not in an independent failure domain: copy to a second target

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Disaster recovery team

Backup Ceph configuration and keyrings

This runbook covers Backup Ceph configuration and keyrings on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (cluster-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A backup procedure is scheduled
  • A backup target is available

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s is HEALTH_OK
# The backup target is reachable
# A backup window is scheduled

Procedure

Step 1

Backup the MON store: rsync -avz /var/lib/ceph/mon/ceph-<id> backup:/var/backups/ceph/mon/<id>/<date>/

Step 2

Backup the orchestrator config: /etc/ceph/, /var/lib/ceph/

Step 3

Backup the keyrings

Step 4

Backup the CRUSH map: ceph osd crush dump

Step 5

Backup the cluster map: ceph mon dump

Step 6

Verify the backups are restorable

Verification

After completing the procedure, verify the result:

# Backups complete and are reachable
# Verification confirms the backup matches the live data
# Backup is in an independent failure domain

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the backup is corrupt: re-run the backup
  • If the backup is not in an independent failure domain: copy to a second target

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Disaster recovery team

References

  1. Ceph documentation