Recover a full Ceph cluster

This runbook covers Recover a full Ceph cluster on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (cluster-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• The cluster has reached nearfull or full
• New writes are blocked
• Recovery is stalled

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s reports the nearfull / full state
# ceph health detail lists the affected pools
# ceph osd df shows the OSD utilisation
# Capacity headroom is at zero

Procedure

Step 1

Immediate: identify the affected pools; do not raise the full ratio as a sustained measure

Step 2

Mid-term: add OSDs; add hosts; rebalance data

Step 3

If the cluster is at mon_osd_full_ratio: temporarily raise it just enough to allow essential writes

Step 4

Identify the runaway workload

Step 5

Verify cluster returns to HEALTH_OK

Verification

After completing the procedure, verify the result:

# The cluster accepts writes
# No POOL_NEARFULL
# Backfill completes
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

• If the cluster cannot be reduced: escalates to the disaster-recovery runbook
• If a workload cannot be paused: identify the workload

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team
• Application team to identify the workload
• Vendor support
• Disaster recovery team