Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskservice affecting~120 min

Perform planned node maintenance safely

1 · Prerequisites

Confirm every item is in place before any state change.

  • A node needs maintenance
  • A maintenance window is scheduled

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The cluster is HEALTH_OK
  • · ceph osd tree shows the hosts OSDs
  • · Backfill is not currently in progress
  • · Capacity headroom can absorb the host

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Mark the OSDs out: ceph osd out osd.<id>
  2. 2Wait for backfill: ceph progress
  3. 3Perform the maintenance
  4. 4Verify the host returns: ceph orch host ls
  5. 5Re-add the OSDs: ceph orch daemon add osd <host>
  6. 6Verify recovery: ceph -s reports HEALTH_OK

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The host returns with the new kernel
  • ✓OSDs come back up
  • ✓Pool data is in active+clean
  • ✓Cluster is HEALTH_OK

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the host does not come back: hardware failure suspected
  • ↶If OSDs do not re-add: check podman, time, DNS, network
  • ↶If data is at risk: restart recovery

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Hardware vendor if the failure is hardware

Perform planned node maintenance safely

This runbook covers Perform planned node maintenance safely on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A node needs maintenance
  • A maintenance window is scheduled

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# The cluster is HEALTH_OK
# ceph osd tree shows the hosts OSDs
# Backfill is not currently in progress
# Capacity headroom can absorb the host

Procedure

Step 1

Mark the OSDs out: ceph osd out osd.<id>

Step 2

Wait for backfill: ceph progress

Step 3

Perform the maintenance

Step 4

Verify the host returns: ceph orch host ls

Step 5

Re-add the OSDs: ceph orch daemon add osd <host>

Step 6

Verify recovery: ceph -s reports HEALTH_OK

Verification

After completing the procedure, verify the result:

# The host returns with the new kernel
# OSDs come back up
# Pool data is in active+clean
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the host does not come back: hardware failure suspected
  • If OSDs do not re-add: check podman, time, DNS, network
  • If data is at risk: restart recovery

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Hardware vendor if the failure is hardware

References

  1. Ceph documentation