low riskservice affecting~20 min

Investigate an OSD down event

1 · Prerequisites

Confirm every item is in place before any state change.

The OSD is in down + in state
The cluster is HEALTH_WARN with one OSD down

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· ceph osd tree shows the OSD down
· smartctl -a /dev/<device> reports device health
· journalctl -u ceph-osd@<id> shows the daemon log
· ceph daemonperf <osd> shows the OSDs responsiveness
· Time and DNS are healthy

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read ceph health detail
2Read ceph osd tree
3Read journalctl -u ceph-osd@<id> for the last 100 lines
4Run smartctl -a /dev/<device>
5Read dmesg on the host for I/O errors
6Read the host firewall state
7Read the host time
8Conclude: device / network / daemon / kernel / time
9Decide: wait / mark out / replace

4 · Verification

Confirm the procedure actually fixed the problem.

✓Root cause identified; OSD state is correctly classified
✓The chosen response is documented and verified

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the OSD remains down: see ceph-rb-replace-osd
↶If the device is suspect: stop using the disk, replace it
↶If the host kernel is suspect: patch / reboot

6 · Escalation

When the runbook isn't enough, contact:

· Storage team if the diagnosis is unclear
· Hardware vendor if the disk is failing

Investigate an OSD down event

This runbook covers Investigate an OSD down event on a Ceph Tentacle cluster.

When to use

Active production scenario matching the impact (service-affecting)
The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

The OSD is in down + in state
The cluster is HEALTH_WARN with one OSD down

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph osd tree shows the OSD down
# smartctl -a /dev/<device> reports device health
# journalctl -u ceph-osd@`<id>` shows the daemon log
# ceph daemonperf `<osd>` shows the OSDs responsiveness
# Time and DNS are healthy

Procedure

Step 1

Read ceph health detail

Step 2

Read ceph osd tree

Step 3

Read journalctl -u ceph-osd@<id> for the last 100 lines

Step 4

Run smartctl -a /dev/<device>

Step 5

Read dmesg on the host for I/O errors

Step 6

Read the host firewall state

Step 7

Read the host time

Step 8

Conclude: device / network / daemon / kernel / time

Step 9

Decide: wait / mark out / replace

Verification

After completing the procedure, verify the result:

# Root cause identified; OSD state is correctly classified
# The chosen response is documented and verified

Rollback

If the procedure does not produce the expected verification, roll back:

If the OSD remains down: see ceph-rb-replace-osd
If the device is suspect: stop using the disk, replace it
If the host kernel is suspect: patch / reboot

Escalation

If you cannot roll back or the rollback fails, escalate:

Storage team if the diagnosis is unclear
Hardware vendor if the disk is failing

References

Ceph documentation