Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskservice affecting~20 min

Investigate nearfull on a pool

1 · Prerequisites

Confirm every item is in place before any state change.

  • ceph -s reports POOL_NEARFULL
  • ceph df shows the pool utilisation above 0.85

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph health detail lists the nearfull message
  • · ceph osd df tree identifies the affected OSDs
  • · ceph pool stats <pool> shows the pools objects and bytes
  • · Capacity headroom is verified

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Identify the pool
  2. 2Compute the clusters growth rate (Part LXVI)
  3. 3Decide: add capacity, rebalance, delete data
  4. 4Do not raise the threshold
  5. 5Apply the chosen remediation
  6. 6Verify the pool utilisation drops below 0.85

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The pool is no longer nearfull
  • ✓Cluster is HEALTH_OK
  • ✓Recovery / backfill completes

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the pool stays nearfull after capacity addition: check the CRUSH rule
  • ↶If the cluster headroom is exhausted: escalate

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Vendor support if a hardware failure is the cause

Investigate nearfull on a pool

This runbook covers Investigate nearfull on a pool on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • ceph -s reports POOL_NEARFULL
  • ceph df shows the pool utilisation above 0.85

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph health detail lists the nearfull message
# ceph osd df tree identifies the affected OSDs
# ceph pool stats `<pool>` shows the pools objects and bytes
# Capacity headroom is verified

Procedure

Step 1

Identify the pool

Step 2

Compute the clusters growth rate (Part LXVI)

Step 3

Decide: add capacity, rebalance, delete data

Step 4

Do not raise the threshold

Step 5

Apply the chosen remediation

Step 6

Verify the pool utilisation drops below 0.85

Verification

After completing the procedure, verify the result:

# The pool is no longer nearfull
# Cluster is HEALTH_OK
# Recovery / backfill completes

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the pool stays nearfull after capacity addition: check the CRUSH rule
  • If the cluster headroom is exhausted: escalate

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Vendor support if a hardware failure is the cause

References

  1. Ceph documentation