Attributing host symptoms to containers - triage without the runtime CLI

top shows a process burning four cores. df says the filesystem is 98% full. The conntrack table is full and new connections are being dropped. In each case the symptom is on the host and the cause is in a container, and the gap between those two facts is where container hosts eat afternoons.

The runtime CLI can close that gap when it is available. This lesson closes it without one - because the runtime daemon is exactly the thing that hangs when a host is in trouble, and because the same commands work whichever runtime is installed.

PID to container

Every container process is in a cgroup, and every OCI runtime puts the container identifier in the cgroup path. That makes /proc/PID/cgroup the single most useful file on a container host.

$ cat /proc/2192/cgroup

0::/system.slice/docker-25fb9c8b983b3ca3358b9bdc6902921ed5d6e8bc1d78db7eef0e2f5f2645f98a.scope

The 0:: prefix is the cgroup v2 unified hierarchy. The path after it names the container - here a Docker container, whose full 64-character ID is in the scope name. Other runtimes use their own shapes and all of them carry the ID:

So the first move on seeing an unexplained process is one command:

PID=2192
awk -F: '$1=="0"{print $3}' /proc/"$PID"/cgroup
cat /proc/"$PID"/comm

And the reverse direction - every process in that container:

CG=$(awk -F: '$1=="0"{print $3}' /proc/"$PID"/cgroup)
cat /sys/fs/cgroup"$CG"/cgroup.procs

cgroup.procs is the authoritative membership list. It includes processes the runtime does not know about: anything an operator started with an exec, anything the entrypoint forked.

The whole picture, per cgroup

For “which container is using the resources”, the cgroup tree is the right view of a container host, and systemd ships two tools that read it:

# The tree, with command lines
systemd-cgls --no-pager

# top, but grouped by cgroup instead of by process
systemd-cgtop

systemd-cgtop is the answer to “which container is causing this” for CPU, memory and I/O in one screen, and it needs nothing installed. When a limit is the suspect, read it and the events counter together:

sudo cat /sys/fs/cgroup"$CG"/memory.max
sudo cat /sys/fs/cgroup"$CG"/memory.events
sudo cat /sys/fs/cgroup"$CG"/cpu.stat

memory.events records oom_kill, and cpu.stat records nr_throttled and throttled_usec. A container that is slow with modest CPU utilisation and a rising nr_throttled is hitting its quota, not running out of work - and that is a conclusion no amount of looking at the application will reach.

Grouping by namespace

The cgroup answers “which container”. Namespaces answer “what does this process share with what”.

# Every namespace this process is in
sudo lsns -p "$PID"

# All network namespaces on the host, with a process in each
sudo lsns -t net

# The raw identifiers
sudo readlink /proc/"$PID"/ns/net /proc/"$PID"/ns/mnt /proc/"$PID"/ns/pid

Two processes with the same net:[...] inode are in the same network namespace - which usually means the same container, and in Kubernetes means the same pod. That is how you discover that a sidecar and an application container share a network stack but not a filesystem: same net inode, different mnt inode.

Note that lsns and the ns symlinks need root, or a process you own. As an unprivileged user you see only your own namespaces, which is a common source of “the command returned nothing” on a first attempt.

Reading a container filesystem from the host

/proc/PID/root is the process’s root directory as the kernel sees it - the merged OverlayFS view, reachable from the host without entering anything:

sudo ls /proc/"$PID"/root/etc/
sudo cat /proc/"$PID"/root/etc/os-release
sudo cp /proc/"$PID"/root/var/log/app.log /tmp/evidence-app.log

This works on distroless images with no shell, and it works when the runtime is unresponsive. It is the correct way to pull evidence out of a container you are about to kill.

The mount table tells you where the writable layer lives, which is what you need when a filesystem is filling up:

$ sudo grep -o 'upperdir=[^,]*' /proc/2192/mountinfo

upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/14621/fs

That directory is the container’s writable layer on the host filesystem. Its size is what the container has written since it started:

UPPER=$(grep -o 'upperdir=[^,]*' /proc/"$PID"/mountinfo | cut -d= -f2)
sudo du -xh --max-depth=2 "$UPPER" | sort -h | tail -20

A container that has written 40 GB into its own layer shows up here immediately, and the path tells you which snapshot to attribute it to. The mechanics of why a container fills its layer - copy-up, whiteouts that free nothing - are in linux-overlayfs-for-containers.

Sockets and traffic

Host ss shows host sockets. A container’s listening sockets live in its own network namespace and are simply absent from the host view, which is why “nothing is listening on 8080” is a misleading thing to conclude from the host.

# The container's sockets, using the host binary
sudo nsenter -t "$PID" -n ss -ltnp

# Its routes and rules
sudo nsenter -t "$PID" -n ip route
sudo nsenter -t "$PID" -n nft list ruleset

Going the other way - from an address seen on the wire to the container that owns it - runs through the veth pair:

# Host-side veth ends, their bridge, and the netns id of the peer
ip -o link show type veth

# Match a netns id to a process
sudo lsns -t net

link-netnsid on the host end and the namespace listed by lsns are the two halves of that join. Registering a name makes everything easier, and it is covered in linux-network-namespaces-by-hand:

sudo ip netns attach app "$PID"
sudo ip netns exec app tcpdump -ni any -c 20

A triage table

Knowledge check