high riskcluster affecting~60 min

Restore Ceph configuration from backup

1 · Prerequisites

Confirm every item is in place before any state change.

A backup of the cluster configuration (auth, MON store, CRUSH map) is verified
The cluster is failing in a way that requires restoring from backup

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The backup files are accessible
· The replacement hosts (if needed) are ready
· Time and DNS are healthy
· Podman is installed on each host

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Restore the MON store: rsync -avz backup:/var/backups/ceph/mon/<id> /var/lib/ceph/mon/ceph-<id>
2Restore the configuration files: rsync -avz backup:/etc/ceph/ /etc/ceph/
3Restore the keyrings: ceph auth import /tmp/old-keyring.bin
4Restart the affected daemons: ceph orch daemon restart <type>.<id>
5Verify: ceph -s reports HEALTH_OK
6Verify: clients authenticate with the restored keys

4 · Verification

Confirm the procedure actually fixed the problem.

✓The MON store is restored
✓The configuration files are restored
✓The keyrings are restored
✓The cluster is HEALTH_OK
✓Clients authenticate successfully

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the restored keys are wrong: rotate them
↶If the MON store is corrupt: do not start the MON; escalate
↶If the cluster cannot be brought up: escalate to disaster recovery

6 · Escalation

When the runbook isn't enough, contact:

· Storage team
· Disaster recovery team

Restore Ceph configuration from backup

This runbook covers Restore Ceph configuration from backup on a Ceph Tentacle cluster.

When to use

Active production scenario matching the impact (cluster-affecting)

Prerequisites

A backup of the cluster configuration (auth, MON store, CRUSH map) is verified
The cluster is failing in a way that requires restoring from backup

Pre-checks

# The backup files are accessible
# The replacement hosts (if needed) are ready
# Time and DNS are healthy
# Podman is installed on each host

Procedure

Step 1

Restore the MON store: rsync -avz backup:/var/backups/ceph/mon/<id> /var/lib/ceph/mon/ceph-<id>

Step 2

Restore the configuration files: rsync -avz backup:/etc/ceph/ /etc/ceph/

Step 3

Restore the keyrings: ceph auth import /tmp/old-keyring.bin

Step 4

Restart the affected daemons: ceph orch daemon restart <type>.<id>

Step 5

Verify: ceph -s reports HEALTH_OK

Step 6

Verify: clients authenticate with the restored keys

Verification

# The MON store is restored
# The configuration files are restored
# The keyrings are restored
# The cluster is HEALTH_OK
# Clients authenticate successfully

Rollback

If the restored keys are wrong: rotate them
If the MON store is corrupt: do not start the MON; escalate
If the cluster cannot be brought up: escalate to disaster recovery

Escalation

Storage team
Disaster recovery team

References

Ceph documentation