When an image will not open

Why this matters in production

The error messages are specific and are usually skimmed. Each names a different problem with a different fix, and reading the exact text before starting the investigation saves most of the investigation.

Check the cluster first

ceph -s
ceph health detail

If PGs are inactive, the image is unavailable because its data is, and no amount of image-level investigation helps. Rule this out in one command before anything else.

The four messages

rbd: error opening image: (2) No such file or directory

The image name is not in the pool’s directory. Causes: wrong pool, wrong name, the image was renamed, or it was deleted.

POOL=rbd-vms
ID=12
rbd ls ${POOL}
rbd ls ${POOL} --long
rbd trash ls ${POOL}          # deleted but recoverable
rbd trash restore ${POOL}/${ID}

The trash is worth checking — rbd rm moves images there with a delay on many configurations, so a recent deletion may be recoverable.

rbd: image uses unsupported features: 0x38

The client cannot handle the image’s feature set. Almost always a kernel RBD client against an image with modern features.

POOL=rbd-vms
IMAGE=vm-disk-01
rbd info ${POOL}/${IMAGE} | grep features
rbd feature disable ${POOL}/${IMAGE} object-map fast-diff

rbd: error opening image: (1) Operation not permitted

A capability problem. The credential lacks access to this pool, or to the data pool of an image using --data-pool.

NAME=acme
PATH=/mnt/cephfs/projects
POOL=rbd-vms
IMAGE=vm-disk-01
ceph auth get client.${NAME}
rbd -n client.${NAME} --keyring ${PATH} info ${POOL}/${IMAGE}

rbd: error opening image: (30) Read-only file system or a hang

The pool is full, or the PGs are inactive. Back to ceph -s.

Watchers blocking exclusive access

POOL=rbd-vms
IMAGE=vm-disk-01
rbd status ${POOL}/${IMAGE}
# Watchers:
#   watcher=10.20.0.55:0/1234567 client.4152 cookie=1

Another client holds the image. If that client is genuinely gone — a crashed host — the lock can be broken:

POOL=rbd-vms
IMAGE=vm-disk-01
LOCK_ID=12
LOCKER=locker
rbd lock ls ${POOL}/${IMAGE}
rbd lock rm ${POOL}/${IMAGE} ${LOCK_ID} ${LOCKER}

Break a lock only after confirming the holder is truly dead. Two writers on one image corrupts it immediately.

Quiz

Production discipline

Read the exact error text before beginning any RBD availability investigation — the four common messages point at four unrelated causes and each has a different first command. Add rbd trash ls to the deleted-image response; the recovery window is short but the check is free.

Cross-course references

• Kubernetes: distinguishing a Pod scheduling failure from an image pull failure is the same read-the-message discipline
• Linux: mount errors are similarly specific and similarly skimmed