Post-maintenance verification — proving the host came back · Docker & Containers

The host rebooted, docker ps shows the containers, and the temptation is to close the window. That evidence is weaker than it looks: a container in the Up state has had its entrypoint executed and nothing more. It says nothing about whether the application inside is serving, whether the port is reachable, or whether the firewall rules that put traffic there came back.

This lesson is the verification that ends every procedure in this part. Seven layers, from the machine upward, in the order that finds problems fastest.

Layer 1: the host itself

Read-only / Safehost— Did the machine come back the way you expected?

# The kernel you intended to boot
uname -r

# Did every unit start?
systemctl is-system-running
systemctl --failed --no-pager

# Uptime confirms this is the boot you think it is
uptime -p
who -b

# Filesystems all mounted, with room
findmnt -t ext4,xfs -o TARGET,SOURCE,SIZE,USED,AVAIL,USE%
df -i /var/lib/docker

systemctl is-system-running returning degraded means at least one unit failed, and systemctl --failed names it. This is the single fastest way to find a mount that did not come up or a service that failed before Docker started — both of which explain a container that will not start, three layers further up.

Layer 2: the daemons

Read-only / Safedaemons— containerd and dockerd, and whether either is unhappy.

systemctl is-active containerd docker
docker version --format 'server={{.Server.Version}}'
docker info --format 'containers={{.Containers}} running={{.ContainersRunning}} images={{.Images}}'

# Warnings the daemon prints about the environment it found
docker info 2>&1 | grep -iE '^WARNING|no swap|no cpu' || echo 'no daemon warnings'

# Anything logged during startup
journalctl -u docker --since "$(uptime -s)" --no-pager -p warning

The daemon warnings matter after a kernel change. WARNING: No swap limit support appearing for the first time means the new kernel booted without swapaccount=1, and every memory limit on the host now behaves differently from how it did yesterday.

Layer 3: the container set

Not “are containers running” but “are the right containers running”.

Read-only / Safecontainer set— Compare against the baseline captured before maintenance.

docker ps --format '{{.Names}}' | sort > /tmp/running.after

diff /tmp/running.before /tmp/running.after && echo 'PASS: identical container set' || echo 'FAIL: see diff'

# Anything that exists but is not running
docker ps -a --filter status=exited --filter status=created --format '{{.Names}}	{{.Status}}'

Read-only / Safethe miss— One container did not come back. Its restart policy was on-failure.

$ diff /tmp/running.before /tmp/running.after

5d4
< worker

Illustrative output

Without the baseline this is invisible. Eleven containers running looks exactly like twelve containers running unless you are counting, and nobody counts at 03:40.

Layer 4: container health

Read-only / Safehealth— Health status, not run status.

docker ps --format 'table {{.Names}}	{{.Status}}'

# Anything explicitly unhealthy
docker ps --filter health=unhealthy --format '{{.Names}}'

# Anything still starting - these need another look shortly
docker ps --filter health=starting --format '{{.Names}}'

# Why a specific container is unhealthy
docker inspect --format '{{json .State.Health}}' web

Layer 5: networking on the host

The layer that fails silently after a reboot, because Docker rebuilds most of it and something else does not.

Read-only / Safenetwork state— Docker networks, published ports, and the firewall rules underneath.

# Networks exist and containers are attached
docker network ls
docker network inspect bridge --format '{{len .Containers}} container(s) attached'

# Ports are actually published and listening on the host
docker ps --format 'table {{.Names}}	{{.Ports}}'
sudo ss -tlnp | grep -E 'docker-proxy|dockerd' || echo 'no docker-proxy listeners'

# Docker's own NAT rules were re-applied
sudo iptables -t nat -L DOCKER -n --line-numbers | head -20

What Docker rebuilds on start, and what it does not

On daemon start, Docker recreates its own iptables or nftables chains (DOCKER, DOCKER-USER, DOCKER-ISOLATION-STAGE-1 and friends) and inserts the rules for every published port. This is automatic and reliable.

What it does not restore is anything you added yourself. A rule inserted by hand with iptables -I to allow a management subnet, or to block a published port from the public interface, exists only in kernel memory. It is gone after a reboot, and Docker’s own rules coming back correctly makes the host look right.

The consequence is worth stating plainly: a hand-added rule restricting access to a published port disappears on reboot, and the port becomes reachable from wherever the interface is reachable from. The maintenance window silently widened your exposure.

Persist host firewall rules through the distribution’s mechanism — netfilter-persistent, nftables.service, firewalld — and put your own rules in the DOCKER-USER chain, which Docker respects and evaluates before its own. The firewalls part of this course covers the details; the check here is simply that your rules are present after the reboot.

Read-only / Safeyour rules— Confirm the rules you added are back, not just Docker's.

sudo iptables -L DOCKER-USER -n --line-numbers
sudo systemctl is-active nftables netfilter-persistent 2>/dev/null || true

Layer 6: the application, from the host

Bypass the load balancer. You are asking whether the container works, not whether the routing works — those are two failures with two different fixes, and separating them saves a lot of time.

Read-only / Safeapplication from the host— Straight at the published port on this machine.

curl -fsS -o /dev/null -w 'local %{http_code} in %{time_total}s\n' http://127.0.0.1:8080/healthz

# And from inside the network, by service name, to test embedded DNS
docker run --rm --network app_default curlimages/curl:8.10.1 -fsS -o /dev/null -w 'internal %{http_code}\n' http://web:8080/healthz

The second command is the one that catches a broken embedded DNS resolver or a container attached to the wrong network — both of which leave the application reachable from the host and invisible to its own peers.

Layer 7: the application, as a user

Read-only / Safeend to end— Through the real name, the real TLS, the real proxy.

curl -fsS -o /dev/null -w 'end-to-end %{http_code} dns=%{time_namelookup}s tls=%{time_appconnect}s total=%{time_total}s\n' https://app.example.com/healthz

# And confirm the certificate did not expire while you were not looking
echo | openssl s_client -connect app.example.com:443 -servername app.example.com 2>/dev/null | openssl x509 -noout -dates

Only this layer proves the thing the maintenance window was supposed to preserve. Everything above it is diagnosis for when this one fails.

The whole thing as a script

Read-only / Safeverify-host— Run all seven layers, report per layer, exit non-zero on any failure.

#!/usr/bin/env bash
set -uo pipefail

fail=0
chk() { if [ "$1" -eq 0 ]; then echo "PASS  $2"; else echo "FAIL  $2"; fail=1; fi; }

test -z "$(systemctl --failed --no-pager --plain --no-legend)"
chk $? "L1 no failed systemd units"

systemctl is-active --quiet docker containerd
chk $? "L2 docker and containerd active"

docker ps --format '{{.Names}}' | sort > /tmp/running.after
diff -q /tmp/running.before /tmp/running.after > /dev/null
chk $? "L3 container set matches baseline"

test -z "$(docker ps --filter health=unhealthy -q)"
chk $? "L4 no unhealthy containers"

sudo iptables -t nat -L DOCKER -n | grep -q DNAT
chk $? "L5 docker NAT rules present"

curl -fsS -o /dev/null --max-time 5 http://127.0.0.1:8080/healthz
chk $? "L6 application responds locally"

curl -fsS -o /dev/null --max-time 10 https://app.example.com/healthz
chk $? "L7 application responds end to end"

exit "$fail"

Seven lines of output, each of which is either PASS or FAIL, and an exit code you can put in a change ticket. That is what closing a maintenance window should look like.

What if everything passes and the application is still wrong an hour later?

Two classes of problem survive this verification, and both are worth naming because the checklist cannot catch them.

Delayed work. A cron container, a nightly batch job, a certificate renewal timer, a log rotation. None of these run during the window, so none of them are exercised by it. If a mount they need did not come back, or a container they depend on has the wrong restart policy, you find out at 02:00 the following night. The mitigation is to check the next run of anything scheduled, not just the state at the end of the window — put it in the ticket as a follow-up rather than closing on the assumption.

Slow degradation. A memory limit that behaves differently under a new kernel, a connection pool that reconnected to a stale address, a cache that is cold and will be for hours. These pass every point-in-time check and show up as latency over the following day.

Neither is an argument against the checklist. It is an argument for leaving the change ticket open until the first full daily cycle has completed, and for looking at the dashboards the morning after rather than only during the window.