Validate the cluster after a major recovery

This runbook covers Validate the cluster after a major recovery on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (service-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• A major recovery has just completed
• The cluster is HEALTH_OK
• The cluster needs to be validated

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# Capture the cluster state: ceph -s, ceph health detail
# Check ceph osd tree: all OSDs up + in
# Check ceph pg stat: no stuck PGs
# Check ceph df: usable capacity is as expected
# Run a scrub / deep scrub if needed
# Run a benchmark to validate performance

Procedure

Step 1

Document the cluster state in the runbook

Step 2

Verify each pools policy is correct

Step 3

Run a benchmark for the applications expected load

Step 4

Update capacity forecast if needed

Step 5

Update the runbook with any new findings

Verification

After completing the procedure, verify the result:

# The cluster is HEALTH_OK
# Performance is at baseline
# Capacity forecast is current
# Runbook is up to date

Rollback

If the procedure does not produce the expected verification, roll back:

• If the cluster is not HEALTH_OK: identify the issue
• If performance is below baseline: investigate
• If capacity forecast is wrong: re-plan

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team