Inspecting a container from the host — PIDs, namespaces, cgroups · Docker & Containers

A container is not an object the kernel knows about. It is a process, a set of namespace memberships, and a cgroup. Every question you can ask about a container is really a question about one of those three, and all three are readable from the host with tools that have nothing to do with Docker.

That matters twice: when the image has no shell to exec into, and when the daemon itself is the thing that is broken.

The anchor: the host PID

Read-only / Safeone process, two numbers— The same process is PID 2192 on the host and PID 1 inside its own PID namespace.

$ PID=$(docker inspect grafana --format '{{.State.Pid}}')
echo "$PID"
grep -E '^NStgid|^NSpid' /proc/"$PID"/status

2192
NStgid:	2192	1
NSpid:	2192	1

NSpid lists the process’s ID in each PID namespace it belongs to, outermost first. Two columns means one level of nesting: the host, then the container.

This is the single most useful line in this lesson, because it explains a whole class of confusion. A log message saying “killing pid 1” refers to 2192 on the host. kill 1 inside the container and kill 2192 on the host do the same thing. top on the host shows 2192 and no PID 1 for that workload anywhere.

The cgroup, in both directions

Read-only / SafePID to container— On cgroup v2 there is a single line. The scope name contains the full container ID.

$ cat /proc/2192/cgroup

0::/system.slice/docker-25fb9c8b983b3ca3358b9bdc6902921ed5d6e8bc1d78db7eef0e2f5f2645f98a.scope

That is the reverse mapping, and it is the one you need during an incident. top shows a process eating twelve cores; you have a PID and nothing else.

Read-only / Safewhich container is this PID— Extract the container ID from the process cgroup and ask Docker what it is.

PID=2192
CID=$(grep -o 'docker-[0-9a-f]\{64\}' /proc/"$PID"/cgroup | head -1 | cut -d- -f2)
docker inspect "$CID" --format '{{.Name}} image={{.Config.Image}} started={{.State.StartedAt}}'

The same trick works from the other side. Given a container, list every host process inside it:

docker top grafana

docker top runs ps on the host and filters to the container’s processes, so it works even when the image has no ps.

Namespaces are inodes

Each namespace is a file in /proc/<pid>/ns/, and the link target carries an inode number. Two processes are in the same namespace if and only if the inode matches.

Read-only / Safea host shell— The initial namespaces occupy a well-known block of inode numbers around 4026531832. A container process shows different numbers for the namespaces it does not share.

$ ls -l /proc/self/ns/

lrwxrwxrwx 1 ebrandi ebrandi 0 Aug 11 15:54 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 ebrandi ebrandi 0 Aug 11 15:54 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 ebrandi ebrandi 0 Aug 11 15:54 mnt -> mnt:[4026531832]
lrwxrwxrwx 1 ebrandi ebrandi 0 Aug 11 15:54 net -> net:[4026531833]
lrwxrwxrwx 1 ebrandi ebrandi 0 Aug 11 15:54 pid -> pid:[4026531836]
lrwxrwxrwx 1 ebrandi ebrandi 0 Aug 11 15:54 time -> time:[4026531834]
lrwxrwxrwx 1 ebrandi ebrandi 0 Aug 11 15:54 user -> user:[4026531837]
lrwxrwxrwx 1 ebrandi ebrandi 0 Aug 11 15:54 uts -> uts:[4026531838]

Comparing the two sides answers questions that are otherwise guesswork:

Read-only / Safewhat is actually isolated— Diff the container process namespaces against PID 1 on the host. Identical inodes mean that namespace is shared, not isolated.

PID=$(docker inspect grafana --format '{{.State.Pid}}')
for ns in cgroup ipc mnt net pid user uts; do
printf '%-7s host=%s container=%s\n' "$ns" \
  "$(sudo readlink /proc/1/ns/$ns)" \
  "$(sudo readlink /proc/"$PID"/ns/$ns)"
done

Which namespaces Docker actually unshares

By default a container gets its own mnt, pid, net, ipc, uts and cgroup namespaces. It does not get its own user namespace — the container’s root is the host’s root unless you enable user-namespace remapping or run rootless. That is why docker-user-namespaces-rootless exists as a separate lesson, and why the user row of the loop above normally shows identical inodes.

Options change the picture, and the inode comparison is how you confirm it rather than trusting the flag:

--network host — the net inode matches the host’s.
--pid host — the pid inode matches, and the container sees every process on the machine.
--network container:other — the net inode matches that other container, which is the sidecar pattern.
--ipc host, --uts host — the corresponding inode matches.

Each of these is a real and legitimate configuration, and each one removes an isolation boundary that a security review probably assumed was there. This reports the intent, while the inodes above report the result:

docker inspect --format '{{.HostConfig.NetworkMode}} {{.HostConfig.PidMode}} {{.HostConfig.IpcMode}}' <container>

`nsenter` — host tools inside a container

docker exec runs a binary from the image. On a distroless or scratch image there is no shell, no ss, no ip, no curl, and docker exec has nothing to run.

nsenter runs a binary from the host inside the container’s namespaces. The container needs to contain nothing at all.

Read-only / Safedebug a distroless container— Enter only the network namespace and run the host's tooling against the container's interfaces, routes and sockets.

PID=$(docker inspect api --format '{{.State.Pid}}')
sudo nsenter -t "$PID" -n ss -tlnp
sudo nsenter -t "$PID" -n ip -brief addr
sudo nsenter -t "$PID" -n ip route
sudo nsenter -t "$PID" -n cat /etc/resolv.conf

The flags select which namespaces to join:

Flag	Namespace	Typical use
`-n`	network	sockets, routes, interfaces, DNS config
`-m`	mount	see the container’s filesystem view
`-p`	pid	`ps` shows the container’s PID numbering
`-u`	UTS	hostname
`-i`	IPC	shared memory segments
`-a`	all	full `docker exec` equivalent, with host binaries

That last line of the example is worth a note: cat /etc/resolv.conf under -n alone reads the host’s file, because you joined the network namespace but not the mount namespace. Add -m to see the container’s copy. Mixing these up produces confident wrong answers during DNS debugging.

nsenter works when the daemon does not

docker exec is an API call. If dockerd is hung, deadlocked, or mid-restart, every docker command hangs with it — including the ones you need to find out why.

nsenter talks only to the kernel. As long as the container process still exists, you can enter it, read its sockets, inspect its filesystem and capture a stack trace, with the daemon completely unresponsive.

Find the PIDs without the daemon, from the cgroup tree:

for f in /sys/fs/cgroup/system.slice/docker-*.scope/cgroup.procs; do
  echo "$f"
  head -1 "$f"
done

This is the technique that turns “Docker is broken, we rebooted the host” into an actual diagnosis.

What if the container has exited by the time you look?

Every namespace it owned is destroyed as soon as the last process in it exits, and /proc/<pid>/ disappears with the process. There is nothing left to enter. The cgroup directory is removed too, taking memory.peak, memory.events and the CPU counters with it.

This is why post-mortem evidence has to be collected by something that was running before the container died: the event stream, a scraped time series, the logging driver’s output. Reaching for nsenter after the fact is too late by design.

The one thing that does survive is the container object itself, if it was not started with --rm:

docker inspect api --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.FinishedAt}}'
docker logs --tail 100 api

OOMKilled is a boolean the daemon records at exit, and it is often the entire answer.

Sanity check

Why this matters

The abstraction Docker offers is convenient right up to the moment it fails, and then it is in the way. docker exec needs an image with tools in it. docker stats needs a responsive daemon. docker ps needs the API. During the incidents that matter, one or more of those is exactly what is broken.

Underneath, none of it is special. A container is a PID you can find, a set of namespace inodes you can compare, and a cgroup directory you can read. nsenter, /proc and /sys/fs/cgroup keep working when the daemon does not, and they answer questions the Docker CLI has no command for — such as whether the isolation you configured is actually in force.

Learn the three-way mapping once and container debugging stops being a separate skill from Linux debugging, which it never really was.

The anchor: the host PID

The cgroup, in both directions

Namespaces are inodes

nsenter — host tools inside a container

Sanity check

`nsenter` — host tools inside a container