Recovering a container that will not start

A VM that will not boot needs a console, a rescue ISO and patience. A container that will not start needs neither, because its filesystem is a dataset or an image file on a host you already have root on. You can open it, read its logs, fix the file that broke it, and start it again - all without the container running.

The reason this is worth a lesson is that hardly anyone does it. The reflex is to restore from backup, which loses everything since the last snapshot to fix a problem that was often one line in one file.

Step one: get the real error

pct start reports a truncated failure. Its output is a summary, not the error.

# pct start 200

run_buffer: 571 Script exited with status 32
lxc_init: 845 Failed to run lxc.hook.pre-start for container "200"
__lxc_start: 2034 Failed to initialize container "200"
startup for container '200' failed

“Script exited with status 32” is the pre-start hook failing. Status 32 is mount(2) failing, which narrows it a great deal - but the useful line is one level down.

set -euo pipefail
CTID=200
LOGFILE=/tmp/lxc-$CTID-debug.log

pct start "$CTID" --debug 2>&1 | tail -20

# The persistent form, when the failure is intermittent:
lxc-start -n "$CTID" -F -l DEBUG -o "$LOGFILE" || true
grep -iE 'error|failed|denied|no such' "$LOGFILE" | tail -30

The debug log names the file, the mount, or the permission that failed. From there the diagnosis is usually immediate.

Opening the container from the host

This is the technique worth knowing. pct mount mounts a stopped container’s filesystem into the host’s namespace so you can work on it with ordinary tools.

set -euo pipefail
CTID=200

pct status "$CTID"          # must be 'stopped'
pct mount "$CTID"

ROOT="/var/lib/lxc/$CTID/rootfs"
ls "$ROOT"

# Read the container's own logs about why its init gave up.
tail -50 "$ROOT/var/log/syslog" 2>/dev/null || true
journalctl -D "$ROOT/var/log/journal" -n 50 --no-pager 2>/dev/null || true

# How full is it? This is the answer more often than anything else.
df -h "$ROOT"

pct unmount "$CTID"

Things worth doing while mounted, all of them ordinary Linux:

• Read /var/log/ and the journal to find what init tripped over.
• Fix a broken /etc/fstab, /etc/network/interfaces or unit file.
• Reset a forgotten root password by editing /etc/shadow.
• Delete the 40 GiB of logs that filled the rootfs.
• Copy data out before deciding to rebuild the container entirely.

That last one deserves emphasis. Even when a container is beyond repair, it is almost never beyond reading. Recovering the data and rebuilding around it beats restoring a week-old backup.

The full rootfs

By a wide margin the most common cause of a container that misbehaves or will not start cleanly. Containers are given small root filesystems, logs are not rotated, and nothing warns anyone.

set -euo pipefail
CTID=200
ROOT="/var/lib/lxc/$CTID/rootfs"

pct mount "$CTID"
trap 'pct unmount "$CTID"' EXIT

df -h "$ROOT"
du -xh --max-depth=2 "$ROOT" 2>/dev/null | sort -rh | head -15

# Deleted-but-held-open files do not show up in du. On a stopped
# container they are already gone, which is itself a useful fact:
# if df and du disagree on a RUNNING container, that is the reason.

Growing it:

set -euo pipefail
CTID=200

pct resize "$CTID" rootfs +8G

pct exec "$CTID" -- df -h /

pct fsck, and its real limits

set -euo pipefail
CTID=200

pct status "$CTID"          # must be 'stopped'
pct fsck "$CTID" --device rootfs

# A specific mount point instead:
# pct fsck "$CTID" --device mp0

The nuance that saves time: pct fsck is only meaningful where there is a filesystem image to check.

• On a directory or LVM-thin backend, the volume holds an ext4 image and fsck is the right tool.
• On a ZFS subvolume there is no separate filesystem to check. ZFS validates data against checksums on every read and repairs from redundancy where it exists; the equivalent operation is zpool scrub, run on the pool, and it is covered in ZFS scrub and resilver.
• On Ceph RBD the container’s filesystem lives on an image whose integrity is Ceph’s business; scrubbing is a Ceph operation.

So “run pct fsck” is not general advice. On the two most common PVE container backends - ZFS and Ceph - it is either inapplicable or the wrong layer, and reaching for it signals a mental model built on the directory backend.

Repairing a config

set -euo pipefail
CTID=200
CONF="/etc/pve/lxc/$CTID.conf"

cp "$CONF" "/root/$CTID.conf.$(date +%F-%H%M)"

pct config "$CTID"            # parsed view - what PVE believes
cat "$CONF"                   # raw view - what is actually written

# Prefer pct set over editing. It validates.
pct set "$CTID" --memory 4096

# Remove a key that is breaking the start.
pct set "$CTID" --delete mp0

pct config and cat disagreeing is the diagnosis for a malformed line: the raw file has it, the parser dropped it.

Repair or restore?

Both are legitimate. Choosing by habit is not.

The one absolute in that table is the compromise row. A container you suspect has been compromised is not repairable, because you cannot enumerate what was changed. Restore from a known-good point and treat everything after it as suspect.

Verification

A container that started is not a container that recovered.

set -euo pipefail
CTID=200

pct status "$CTID"
pct exec "$CTID" -- systemctl is-system-running || true
pct exec "$CTID" -- systemctl --failed --no-pager
pct exec "$CTID" -- df -h /
pct exec "$CTID" -- findmnt -no TARGET,SOURCE | grep -vE '^/(proc|sys|dev)'

systemctl is-system-running reporting degraded on a container you just recovered means something is still failing, and it is the check most often skipped because the container responded to pct exec and that felt like success. The findmnt line catches the other silent case: a container that started because you removed the mount point that was blocking it, and is now running without the storage its application expects.

Knowledge check