A daemon fails to start after upgrading

Advanced⏱ ~18 mincephcephadmjournalctl

What you'll learn

Identify which daemon failed and why
Read the container logs
Distinguish the failure classes
Resolve or work around it

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The upgrade halts at the failed daemon, and the cluster is mixed-version until it is resolved.

Identifying it

ceph orch upgrade status
ceph orch ps --refresh | grep -v ' running '
ceph -s
ceph health detail

NAME          HOST      STATUS         REFRESHED  VERSION
osd.47        stor-04   error          30s ago    <unknown>

ceph orch ps --daemon-id 47 --daemon-type osd --format json | python3 -c '
import sys,json
for d in json.load(sys.stdin):
    print("status:", d.get("status_desc"))
    print("version:", d.get("version"))
    print("last refresh:", d.get("refreshed"))'

Reading the logs

FSID=$(ceph fsid)
journalctl -u "ceph-$FSID@osd.47" --since '20 min ago' --no-pager | tail -80

# or via the orchestrator
ceph log last 200 cephadm

# the container's own output on the host
podman logs "ceph-$FSID-osd-47" 2>&1 | tail -80

The failure classes

Class	Signal	Resolution
Image pull failed	pull errors, no container	fix registry access, retry
Config option removed	“unrecognised option” at startup	remove the option, redeploy
On-disk format needs conversion	conversion messages, then failure	follow the release notes’ procedure
Underlying device failed	I/O errors in the log	it is a disk failure, not an upgrade failure
Insufficient memory	OOM kill in the journal	reduce cache targets or the daemon count
A genuine regression	assertion or crash on the new version	the tracker and the release notes

# the removed-option case
OPTION=osd_memory_target
ceph config dump | grep -i '<the option named in the log>'
ceph config rm osd.47 ${OPTION}
ceph orch daemon redeploy osd.47

# the memory case
grep -i 'out of memory\|oom' /var/log/kern.log | tail
free -g
ceph config get osd osd_memory_target

The distinction that matters

Was this daemon healthy before the upgrade?
  yes → the upgrade is implicated
  no  → the restart merely exposed an existing problem

# the baseline capture answers this
grep -A2 'osd.47' /backups/ceph-pre-upgrade/daemons.txt

A disk that had been failing quietly often only fails visibly when the OSD restarts, which makes the upgrade look responsible for a problem it only revealed.

Resolving

# after fixing the underlying cause
ceph orch daemon redeploy osd.47
ceph orch ps --daemon-id 47 --daemon-type osd --refresh

# if a single OSD cannot be recovered, remove it and continue
ceph orch osd rm 47 --replace
ceph orch upgrade resume

Quiz

Knowledge check · 4 questions

Q1. Why does a config option removed in a new release stop a daemon from starting?

Daemons validate configuration at startup and refuse unrecognised options rather than ignoring them, so an operator is never misled about what is in effect The upgrade rewrites the option into the monitor config store with the new schema, and the daemon aborts when the stored value no longer matches the type the release expects cephadm validates the whole config against the new image before deploying, and refuses to start any daemon on a host that still carries an option absent from that release The monitors will not hand an OSDMap to a daemon whose config epoch references an option they no longer recognise, so the daemon stalls waiting for a map that never arrives
Q2. A daemon that will not come back after an upgrade can be reporting a hardware fault that predates the upgrade entirely.

True False
Q3. Diagnose an OSD that will not start after upgrading.

The upgrade has stalled. `ceph orch ps` shows osd.47 in error state on host stor-04. The rest of the cluster is on the new version.
Q4. How do you continue an upgrade when a single OSD cannot be recovered?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check the device SMART data and the pre-upgrade baseline before treating a failed daemon as an upgrade regression — an OSD restart often exposes a disk that was already degrading. If one OSD genuinely cannot be recovered, remove it with --replace and resume rather than staying mixed-version.

Cross-course references

Kubernetes: a pod that fails to restart may reveal a pre-existing node problem
Linux: restarts expose latent hardware faults that steady-state operation hides