A daemon crash-looping after an upgrade

Why this matters in production

A crash-looping daemon consumes resources, generates noise, and can drag the cluster into repeated peering while producing no useful work.

Recognising it

ceph orch ps --refresh | grep -v ' running '
ceph crash ls | tail -20

FSID=$(ceph fsid)
systemctl status "ceph-$FSID@osd.47" | head -12

Active: activating (auto-restart) (Result: exit-code)

journalctl -u "ceph-$FSID@osd.47" --since '15 min ago' --no-pager | \
  grep -c 'Starting Ceph'

A restart count in the tens over a few minutes is a loop, not a slow
start.

Stopping the loop

ceph orch daemon stop osd.47

# or at the systemd level on the host
systemctl stop "ceph-$FSID@osd.47"
systemctl disable "ceph-$FSID@osd.47"

Stopping is the first action. A looping daemon overwrites its own
evidence, competes for CPU and memory, and repeatedly disturbs peering
for its PGs.

ceph osd set noout      # so the cluster does not start draining it
ceph -s

Capturing evidence

CRASH_ID=2026-08-18T02:11:04.128Z_9f2c
ceph crash ls-new
ceph crash info ${CRASH_ID}

journalctl -u "ceph-$FSID@osd.47" --since '1 hour ago' --no-pager \
  > /tmp/osd47-crashloop.log
grep -n -i 'assert\|abort\|signal\|traceback\|FAILED' /tmp/osd47-crashloop.log | head -20

# is this one daemon or a pattern?
ceph crash ls --format json | python3 -c '
import sys,json,collections
c = collections.Counter()
for e in json.load(sys.stdin):
    c[e.get("entity_name","?")] += 1
for k,v in c.most_common(10): print("%-16s %d" % (k,v))'

One daemon crashing is likely local — its device, its data, its host.
Several crashing identically is the release.

Deciding how to proceed

# the one-OSD case
ceph orch osd rm 47 --replace
ceph osd unset noout
ceph orch upgrade resume

# the several-OSDs case
ID=12
ceph orch upgrade pause
ceph crash info ${ID} | python3 -c '
import sys,json
d=json.load(sys.stdin)
print("version:", d.get("ceph_version"))
print("assert:", d.get("assert_msg", "")[:200])'

Quiz

Production discipline

Check the crash distribution across daemons before analysing any single crash — one OSD points at local data, several on the same assertion points at the release. Stop a looping daemon and set noout before investigating; it overwrites its own evidence while running.

Cross-course references

• Kubernetes: CrashLoopBackOff distribution across nodes separates local from systemic
• Linux: a service in a restart loop destroys the logs that would explain it