Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskservice affecting~60 min

Perform planned network maintenance safely

1 · Prerequisites

Confirm every item is in place before any state change.

  • A network component needs maintenance
  • A maintenance window is scheduled

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The cluster is HEALTH_OK
  • · Backfill is not currently in progress
  • · Capacity headroom is verified

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Take necessary precautions: ceph osd set noout to prevent auto-marks
  2. 2For a TOR reboot: verify LACP + MLAG are configured
  3. 3For a bond member change: verify the surviving member can carry the traffic
  4. 4Perform the maintenance
  5. 5Unset the flags: ceph osd unset noout
  6. 6Verify recovery: ceph -s is HEALTH_OK

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓Network is recovered
  • ✓No OSDs are down
  • ✓Cluster is HEALTH_OK

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If OSDs go down: verify the network; engage the network team
  • ↶If backfill is required: tune recovery settings; monitor client latency
  • ↶If a switch fails: do not proceed; engage vendor

6 · Escalation

When the runbook isn't enough, contact:

  • · Network team
  • · Storage team if OSDs need to be replaced
  • · Vendor if the switch fails

Perform planned network maintenance safely

This runbook covers Perform planned network maintenance safely on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • A network component needs maintenance
  • A maintenance window is scheduled

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# The cluster is HEALTH_OK
# Backfill is not currently in progress
# Capacity headroom is verified

Procedure

Step 1

Take necessary precautions: ceph osd set noout to prevent auto-marks

Step 2

For a TOR reboot: verify LACP + MLAG are configured

Step 3

For a bond member change: verify the surviving member can carry the traffic

Step 4

Perform the maintenance

Step 5

Unset the flags: ceph osd unset noout

Step 6

Verify recovery: ceph -s is HEALTH_OK

Verification

After completing the procedure, verify the result:

# Network is recovered
# No OSDs are down
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

  • If OSDs go down: verify the network; engage the network team
  • If backfill is required: tune recovery settings; monitor client latency
  • If a switch fails: do not proceed; engage vendor

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Network team
  • Storage team if OSDs need to be replaced
  • Vendor if the switch fails

References

  1. Ceph documentation