LXC updates, migration, and low-downtime maintenance

LXC containers look like lightweight VMs but they’re not. They share the host kernel, share most of the userland tooling, and depend on cgroups and namespaces for isolation. The operational practices for keeping them updated, migrated, and debugged are different from VM operations.

This lesson covers the patterns that work in production.

Container updates vs VM updates

A VM runs its own kernel and userland. You update it by running package updates inside the guest, and a kernel update requires a reboot. The reboot is the VM, isolated.

A container shares the host kernel — uname -r inside the container returns the host kernel version. Updating the kernel requires updating the host, not the container. Container updates are just userland updates (apt/dnf/apk inside the container).

This is a win: container updates are faster and don’t require rebooting the container. But it’s also a constraint: you can’t run a kernel newer than the host inside a container.

Update strategies

Three patterns for keeping containers updated:

1. Manual

pct enter 100
apt update && apt -y upgrade
exit

Simple, but easy to forget. Suitable for 1–5 containers.

2. unattended-upgrades

The standard Debian / Ubuntu unattended-upgrades package:

# Inside each container
apt install -y unattended-upgrades
dpkg-reconfigure -f noninteractive unattended-upgrades

Configure it to allow only security updates (the default). This is /etc/apt/apt.conf.d/50unattended-upgrades-local:

Unattended-Upgrade::Allowed-Origins {
    "Debian-Security:bookworm-security";
};
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";

The Automatic-Reboot "false" is critical — a container reboot is a guest reboot, but if you have services running that don’t auto-start on boot, an unexpected reboot breaks them. For containers, manual restarts after updates are safer.

For a fleet of containers, bake unattended-upgrades into the template so every clone inherits the configuration.

3. Config management

For 10+ containers, use Ansible, Salt, or Puppet to drive updates from outside:

# Ansible example
- name: Update containers
  hosts: lxc_containers
  become: true
  tasks:
    - name: Apply security updates
      apt:
        upgrade: safe
        update_cache: yes
      notify: restart container

# Handlers
- name: restart container
  command: pct reboot {\{ inventory_hostname \}}

This gives you centralised visibility, change control, and the ability to do staged rollouts (canary 10%, then 50%, then 100%).

Migration

pct migrate moves a container between nodes. It takes two mutually exclusive mode flags, documented in pct(1):

# Offline: the container must already be stopped.
pct migrate 100 pve-02

# Restart migration: PVE shuts the container down, moves it, starts it there.
# --timeout bounds the wait for a clean shutdown (default 180s).
pct migrate 100 pve-02 --restart 1 --timeout 180

# Online/live migration.
pct migrate 100 pve-02 --online 1

For a cluster with shared storage the move is quick, because the rootfs is already reachable from the target and only the configuration changes hands. For local-only storage the rootfs has to be copied, which PVE handles automatically and which turns a seconds-long operation into a minutes-long one - proportional to the container’s size and the migration link.

Containers vs VMs when they move

A migrated guest keeps its identity either way — VMID, IP, MAC, disk serial all follow it to the target. What differs is what has to be compatible on the other end.

• A VM carries its own kernel, so the constraint is the CPU: the destination must present a compatible feature set, which is what the machine and CPU type settings in machine versions exist to manage.
• A container runs on the destination’s kernel, so the constraint is the kernel: every feature, module and behaviour the container relies on must be present there.

That second point makes a mixed-kernel cluster a live hazard rather than a cosmetic inconsistency. A container that starts fine on a node running kernel 7.0 and fails, or behaves differently, on one still running an older kernel is not a migration bug — it is the container meeting a different kernel, which is the thing containers do.

Check before you rely on it:

for NODE in pve-01 pve-02 pve-03; do
  printf '%-10s ' "$NODE"
  ssh -o BatchMode=yes "root@$NODE" 'uname -r'
done

Feature flags such as nesting=1 travel with the container config, so they do not need enabling per node — but what those flags expose is the destination node’s procfs and sysfs, which is a kernel property again.

Unprivileged containers and security

LXC has two privilege modes:

• Privileged (--unprivileged 0): the container’s root is the host’s root. Any container-side rootkit is one kernel exploit away from owning the host.
• Unprivileged (--unprivileged 1, the default for new containers): the container’s root maps to an unprivileged host UID — 100000 under the default mapping. Every privilege the container holds is scoped to that mapped range.

The option is unprivileged, not privileged, and it cannot be changed after creation. The mechanics, the audit query and the migration path off privileged mode are in privileged vs unprivileged containers.

Always run containers unprivileged unless you have a specific reason not to. The few exceptions:

• Docker-in-LXC for full Docker compatibility (requires privileged or specific feature flags).
• Containers that need to mount host filesystems.
• Containers that need raw network access (unprivileged can do most things via lxc.idmap).

AppArmor profiles

PVE 9.x ships with AppArmor profiles for unprivileged containers. The default profile (lxc-container-default) is restrictive:

• No raw sockets
• No kernel module loading
• Limited /proc and /sys access
• No ptrace of processes outside the container

You can disable AppArmor for a specific container if a workload needs more access:

# /etc/pve/lxc/100.conf
lxc.apparmor.profile: unconfined

This is rarely needed. Before disabling, check the container’s logs for AppArmor denial messages:

dmesg | grep -i 'apparmor.*100'
# audit: type=1400 audit(1704067200.123:45): apparmor="DENIED" operation="ptrace" ...

If the workload legitimately needs the denied operation, add a custom AppArmor profile rather than disabling enforcement entirely.

Diagnosing container failures

Container failures fall into a few categories:

1. Won’t start

# Check the config
cat /etc/pve/lxc/100.conf

# Try to start with verbose logging
pct start 100 --debug

# Check the kernel log
dmesg --since '5 minutes ago' | grep -i 'lxc\|apparmor\|cgroup'

Common causes:

• AppArmor denial: see above. Fix with pct set 100 --features apparmor=0 (temporary) or a custom profile.
• Missing storage: the storage backend isn’t mounted on the current node. Check with pvesm status.
• Feature flag mismatch: the target node doesn’t have the features the container was created with.
• ZFS pool unavailable: the rootfs is on a ZFS pool that didn’t import. Check zpool status.

2. Runs but is sluggish

# Check cgroup limits
cat /sys/fs/cgroup/system.slice/lxc/100.slice/lxc-100.scope/memory.current
cat /sys/fs/cgroup/system.slice/lxc/100.slice/lxc-100.scope/memory.max

# Check throttling
cat /sys/fs/cgroup/system.slice/lxc/100.slice/lxc-100.scope/memory.events
# If high "low" count, the container is being OOM-killed or throttled

For CPU throttling:

cat /sys/fs/cgroup/system.slice/lxc/100.slice/lxc-100.scope/cpu.stat
# nr_periods, nr_throttled, throttled_usec

If throttled_usec is high, the container is CPU-bound and exceeding its quota. Increase the cores allocation or investigate the workload.

3. Networking broken

# Inside the container
ip addr show
ip route show
cat /etc/resolv.conf

# On the host
pct config 100
# Look at net0/net1 — is the bridge correct? IP assignment OK?

Common causes:

• Bridge doesn’t exist on the target node after live migration
• VLAN tag mismatch between container config and host bridge
• DHCP server unreachable in the container’s VLAN
• DNS misconfigured in the container (containers often need explicit /etc/resolv.conf)

4. Disk full

pct df 100
# Or inside
df -h

LXC containers share host kernel but have their own rootfs. Disk fills up the rootfs, not the host. Use pct resize 100 rootfs +5G to grow it.

Production considerations

• Updates should be staged. Update one container, verify, then roll to the rest. Config management tools make this tractable.
• Snapshots before updates. Take a ZFS or LVM-Thin snapshot before updating a container. If the update breaks the app, revert in seconds.
• Restart dependencies. Some updates require restarts (glibc, openssl, systemd). Track which containers need restarting with needs-restarting (from the yum-utils package on RHEL; Debian has no equivalent, so check each package manually).
• Container density. LXC containers share kernel memory structures. Watch the host’s slab cache, dentry cache, and inotify watches — these don’t isolate per-container.

Common mistakes

• Mounting host paths into privileged containers. This is one kernel exploit away from full host compromise. Use unprivileged containers, or pass through specific host paths with explicit read-only permissions.
• Disabling AppArmor for one workload, then forgetting. Add a custom profile or move the workload to a privileged container with the necessary restrictions.
• Updating the host kernel without restarting the host. Some kernel updates require a host reboot. The running containers keep the old kernel until the host reboots.
• Scheduling a maintenance window on the assumption that containers move without stopping. They restart. Budget a shutdown plus a start per container, measured for the slowest workload on the node.

Key takeaways

• Container updates are userland-only; kernel updates require host reboot.
• Always run unprivileged unless impossible.
• Use snapshots before updates for instant rollback.
• Measure how long a restart migration actually takes for each workload, and put that number in the maintenance plan.

Knowledge check