Investigate a degraded PG

This runbook covers Investigate a degraded PG on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (service-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• ceph -s reports HEALTH_WARN with one or more PGs degraded
• The cluster is still serving I/O

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph pg stat lists the degraded PGs and the count
# ceph pg dump_stuck lists stuck PGs
# ceph -s reports the recovery rate
# The clusters acting set is healthy

Procedure

Step 1

Read ceph health detail for the specific PG id

Step 2

ceph pg <pgid> query for the acting set

Step 3

ceph osd metadata <osd-id>

Step 4

Identify the missing replica

Step 5

Trace the missing OSD

Step 6

Decide: wait / mark out / replace

Verification

After completing the procedure, verify the result:

# PG returns to active+clean
# Recovery completes
# Cluster returns to HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

• If recovery stalls: tune recovery settings
• If the PG is stuck peering: restart the affected OSD
• If the PG is stuck in undersized: identify the missing failure domain

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team if recovery stalls