Running the maintenance window · Docker & Containers

The five previous lessons produced the pieces. This one is the order they go in, and the two decisions that bracket them: whether to start, and when to stop.

The window is short — a patch upgrade on a host with live restore takes about four minutes of actual work. Most of what follows is the structure that makes those four minutes safe to do at 03:00 on a host you do not know well.

Go / no-go

Every item is a command that returns an answer. None of them is a judgement call, which is the point: the gate must be applicable by whoever is holding the pager, not only by the person who planned the change.

Read-only / Safego/no-go— Run this immediately before opening the window. Any FAIL is a no-go.

#!/usr/bin/env bash
set -uo pipefail

TARGET='5:28.3.2-1~ubuntu.24.04~noble'
CAPTURE='/var/backups/docker-preupgrade/latest'
fail=0
chk() { if [ "$1" -eq 0 ]; then echo "PASS  $2"; else echo "FAIL  $2"; fail=1; fi; }

# 1. The target version exists for this distribution
apt-cache madison docker-ce | grep -qF "$TARGET"
chk $? "target version $TARGET available"

# 2. The previous version is still available for rollback
apt-cache madison docker-ce | grep -qF "$(dpkg-query -W -f='${Version}' docker-ce)"
chk $? "current version still published (rollback path exists)"

# 3. The capture exists and is recent
test -s "$CAPTURE/containers.json"
chk $? "pre-upgrade capture present"

# 4. Live restore is on, or an outage is explicitly accepted
test "$(docker info --format '{{.LiveRestoreEnabled}}')" = 'true'
chk $? "live-restore enabled"

# 5. The daemon config parses
sudo dockerd --validate --config-file /etc/docker/daemon.json > /dev/null 2>&1
chk $? "daemon.json validates"

# 6. Nothing is already unhealthy
test -z "$(docker ps --filter health=unhealthy -q)"
chk $? "no container currently unhealthy"

# 7. Enough free disk for the new packages and a layer or two
test "$(df --output=avail -BG /var/lib/docker | tail -1 | tr -dc '0-9')" -ge 5
chk $? "at least 5G free on /var/lib/docker"

exit "$fail"

The runbook

Announce. Post the start in whatever channel your team watches, with the host, the version change, the expected impact and the expected duration. One message, before anything happens.
Silence monitoring for this host. A maintenance window in the monitoring system, scoped to the host and time-boxed to the planned duration plus a margin. Time-boxed, so it expires on its own.
Take the capture. The script from the capture lesson. Copy the output directory off the host.
Record the baseline. docker ps and the StartedAt of every running container, to a file. This is what you will diff against.
Refresh package metadata. sudo apt-get update. Nothing is installed by this step.
Install the pinned versions. The daemon restarts inside this step. This is the only irreversible action in the runbook.
Verify the daemon. systemctl is-active docker and docker version. If the daemon is not active, go to the abort criteria — do not proceed.
Verify the containers. Diff StartedAt against the baseline. Identical means nothing restarted.
Verify the application. An actual request through the actual path users take. Not docker ps.
Un-silence monitoring and confirm green. Watch for one full check interval before you believe it.
Close. Post the completion with the evidence: versions before and after, container verification result, application check result.

Steps 7, 8 and 9 are three different questions and people routinely stop after the first. The daemon being up says nothing about the containers; the containers being up says nothing about whether the application works.

Read-only / Safesteps 7-9— The three verifications, in order.

# 7. Daemon
systemctl is-active docker
docker version --format 'client={{.Client.Version}} server={{.Server.Version}}'

# 8. Containers - identical StartedAt means nothing was restarted
docker inspect --format '{{.Name}} {{.State.StartedAt}}' $(docker ps -q) > /tmp/started.after
diff /tmp/started.before /tmp/started.after && echo 'PASS: no restarts'

# 8b. And nothing is unhealthy
docker ps --filter health=unhealthy --format '{{.Names}}'

# 9. Application - through the real path
curl -fsS -o /dev/null -w '%{http_code} %{time_total}s\n' https://app.example.com/healthz

Abort criteria

Write these before the window, in the ticket, as conditions rather than feelings. The operator at 03:00 should be able to read them and act without deciding anything.

Condition	Action
Daemon not active 60 seconds after install	Check `journalctl -u docker`; if it names a config key, fix it. Otherwise abort.
Daemon restarting in a loop	Abort. `Restart=always` masks the failure as flapping.
Any container failed to restore and is not in `docker ps`	Abort, then check for orphaned shims before restarting anything.
Application check fails and containers are healthy	Not an abort. Investigate — this is usually unrelated.
Any step takes longer than twice its estimate	Stop and reassess. Do not continue on momentum.
You are past the end of the announced window	Stop. Roll back or extend the window explicitly, with a message.

The window that ran four hours because nobody said stop

The characteristic failure of maintenance windows is not a bad rollback. It is the absence of a stopping rule.

Something does not work. The operator tries a fix. It half works. They try another. Each individual step is reasonable, the window quietly extends from 30 minutes to four hours, and by the end the host is in a state that matches neither the old configuration nor the new one, with no record of how it got there. The rollback that would have taken five minutes at 03:15 is now impossible because too much has changed.

The rule that prevents it is arbitrary and effective: at the announced end of the window, you either close it or you roll back. Extending is allowed, but only as an explicit decision with a new end time, announced. That single constraint is worth more than any amount of technical preparation.

Fleet order

For more than one host, the order is not “all of them”.

A staging host that matches production. Full runbook, including the application check. Wait long enough to see delayed effects — at least one business day for anything with a daily batch job.
One production host, the least critical. Full runbook. Wait again.
The rest, in batches, with the batch size chosen so that a failure at any point leaves enough capacity serving traffic.

What if the application check fails but every container is healthy?

Resist the reflex to roll back. A green daemon and green containers with a red application check is usually not the upgrade.

Work through it in this order:

Is the check itself broken? A monitoring maintenance window can suppress the thing the check depends on. Run the check by hand.
Is it upstream? A dependency outside this host — a database, an API, a DNS resolver — with coincidental timing. Check whether the failure predates step 6, using the baseline you captured.
Is it the reverse proxy? The API gap during the daemon restart can leave a container-aware proxy such as Traefik with a stale routing table. It normally reconverges within a health-check interval; if it has not, restarting the proxy is a smaller action than rolling back Docker.
Only then, the upgrade.

Rolling back a Docker upgrade to fix a database outage wastes the window and leaves you with an unexplained failure and an un-upgraded host.

Closing evidence

The completion message is short and it is not optional. Three lines:

DONE app-01: docker-ce 28.1.3 -> 28.3.2, containerd.io unchanged at 1.7.27
CONTAINERS: 9/9 StartedAt unchanged, 0 unhealthy, no restarts
APP: https://app.example.com/healthz 200 in 0.081s at 03:22Z
WINDOW: 03:04Z - 03:24Z (planned 30m). Hold NOT applied; normal patching resumes.

Anyone reading that in a week knows exactly what changed and what was proven. Compare it with the usual “upgraded docker on app-01, all good”, which proves nothing and is indistinguishable from an upgrade that quietly restarted every container.