The container runtime under cephadm

Advanced⏱ ~17 minpodmansystemctl

What you'll learn

Verify the container runtime meets cephadm's needs
Explain how cephadm runs daemons as containers
Diagnose container-level failures
Manage images and registry access

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph daemons are containers managed by systemd units cephadm generates. When a daemon will not start, the cause is as likely to be in the runtime or the image as in Ceph — and the diagnostics are different.

Requirements

podman version
podman info | grep -E 'graphDriver|runRoot|graphRoot'
systemctl status podman.socket

Check the Ceph release’s documented minimum podman version. Storage driver matters less than having adequate space in the graph root — images are several gigabytes and every daemon on the host shares them.

df -h /var/lib/containers

How daemons run

# Cluster FSID from `ceph fsid`; substitute your own:
FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64

systemctl list-units 'ceph-*'
systemctl cat "ceph-$FSID@osd.12.service"

Each daemon is a systemd unit that runs a container with the host’s network namespace, the daemon’s data directory bind-mounted, and the appropriate device access.

# Cluster FSID from `ceph fsid`; substitute your own:
FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64

ls "/var/lib/ceph/$FSID/osd.12/"
podman ps --filter name=ceph
podman logs "ceph-$FSID-osd-12"

The daemon’s persistent state — keyring, configuration, and for OSDs the device mapping — lives on the host, not in the container. That is what makes a container replacement transparent.

Diagnosing failures

# Cluster FSID from `ceph fsid`; substitute your own:
FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64

# systemd's view
systemctl status "ceph-$FSID@osd.12.service"
journalctl -u "ceph-$FSID@osd.12.service" --since '30 min ago'

# the container's own output
podman logs --tail 100 "ceph-$FSID-osd-12"

# cephadm's view
ceph orch ps --daemon-type osd
ceph log last cephadm

Symptom	Likely cause
Unit fails immediately	image missing or pull failed
Container starts and exits	daemon-level error — read podman logs
Unit not found	orchestrator has not deployed it
Image pull fails	registry unreachable or credentials
Permission denied on devices	SELinux, or device access in the unit

Images and registries

ceph config get mgr container_image
podman images | grep ceph
ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.1

For air-gapped environments, mirror the images locally:

ceph config set mgr mgr/cephadm/container_image_base registry.internal/ceph/ceph
ceph config set global container_image registry.internal/ceph/ceph:v19.2.1

Every host must be able to pull, so a host with a registry access problem fails to deploy daemons while everything else works.

Cleaning up

podman image prune -a
podman system df

Old images accumulate after upgrades and consume the graph root. A full graph root prevents new daemons starting, which presents as a deployment failure rather than a disk-space one.

Quiz

Knowledge check · 4 questions

Q1. New daemons fail to deploy on one host while existing daemons run normally. What should you check?

The Ceph version on that host, since a daemon of a different version is refused placement The monitor quorum, since deploying a daemon needs a quorum write that running ones do not The CRUSH map, since a host absent from it cannot receive newly deployed daemons Free space in the container graph root and registry reachability from that host
Q2. A Ceph daemon's keyring and configuration live inside its container.

True False
Q3. Diagnose a daemon that will not start after an upgrade.

Following a Ceph upgrade, one OSD on one host will not start. `ceph orch ps` shows it in error state. Other OSDs on the same host upgraded successfully.
Q4. Why does cephadm bind-mount a per-daemon directory from the host?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Monitor /var/lib/containers as a distinct filesystem and prune images after upgrades; a full graph root blocks daemon deployment and is reported as a deployment failure with no mention of disk space. Read podman logs for daemon-level errors — the orchestrator and systemd both summarise them away.

Cross-course references

Kubernetes: node disk pressure from image accumulation causes the same deployment failures
Linux: container runtime storage management is ordinary host administration with real consequences here