Runbook: Drain a Docker host for maintenance

1 · Prerequisites

Confirm every item is in place before any state change.

• A change window is agreed and the remaining hosts can carry the traffic this host is currently taking
• You can reach the load balancer or service registry to remove this host, and you have tested that you can put it back
• The dependency order of the stacks on this host is known: which service writes state, which only reads, which is the front door
• The compose files or unit files for every stack on the host are in version control and their paths are known
• Out-of-band or console access, so the maintenance does not depend on an SSH path through a container
• A recent, restorable backup exists for every named volume on the host
• You know whether this host is a Swarm node, because a Swarm node is drained from a manager, not locally

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · docker ps --format "{{.Names}} {{.Image}} {{.Status}} {{.Ports}}" - the full inventory of what is running and what it publishes
• · docker ps --format "{{.Names}}" > /var/tmp/docker-predrain.txt - the list you must be able to restore exactly
• · docker compose -f "$COMPOSE_FILE" config --services - the service names and, read alongside the file, their depends_on order
• · docker volume ls --format "{{.Name}} {{.Driver}}" - every named volume that must be quiesced before the host goes down
• · docker inspect --format "{{.Name}} {{.Config.StopSignal}} {{.Config.StopTimeout}}" $(docker ps -q) - what each container expects on shutdown
• · ss -tn state established - how many client connections this host is currently holding
• · docker info - read the Swarm line; if it is active this host is drained from a manager instead
• · docker node ls (on a Swarm manager only) - confirm the cluster has capacity for this node to leave

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Record the inventory. Write docker ps --format "{{.Names}}" to /var/tmp/docker-predrain.txt. Expect a file listing every container you must bring back.
2. 2Remove the host from the load balancer or service registry FIRST, before touching any container. Expect the load balancer to report the host as down or draining.
3. 3Wait for in-flight requests to finish. Watch ss -tn state established until the established connection count stops falling. Expect it to settle near zero, not merely to decrease.
4. 4Confirm no new work is arriving: tail the front-door container log for 60 seconds with docker logs --since 60s --follow "$FRONTEND". Expect no new request lines.
5. 5On a Swarm node, drain from a manager instead of stopping containers by hand: docker node update --availability drain "$NODE". Expect docker node ls to show AVAILABILITY Drain and the tasks to be rescheduled elsewhere.
6. 6Stop the front door first, so nothing can enqueue new work: docker stop --timeout 30 "$FRONTEND". Expect the container to exit before the timeout, not to be killed at it.
7. 7Stop the stateless middle tier next, in reverse dependency order. Expect each docker stop to return within its timeout.
8. 8Stop the stateful services last, with a timeout long enough for their own shutdown: docker stop --timeout 120 "$DATABASE". Expect a clean shutdown message in docker logs, not a signal-killed message.
9. 9For a whole compose stack, prefer docker compose -f "$COMPOSE_FILE" stop --timeout 120, which stops in dependency order and does not remove anything. Expect docker compose ps --all to show every service exited.
10. 10Verify no container exited with code 137. Run docker ps -a --format "{{.Names}} {{.Status}}" and expect Exited (0) for every service that handles state.
11. 11Confirm the volumes are quiesced: with the containers stopped, no process should hold files open under the volume mountpoints. Expect sudo lsof +D on the mountpoint to return nothing.
12. 12Verify the host is serving nothing: ss -ltnp shows no published container ports still bound, and a request to the host from outside fails or is refused. Expect a refused connection, not a slow success.
13. 13Confirm the remaining hosts absorbed the traffic before starting the maintenance. Expect the service-level error rate to be unchanged on the dashboard.
14. 14Perform the maintenance. Then reverse: start stateful services first, wait for healthy, then the middle tier, then the front door.
15. 15Return the host to the load balancer, or run docker node update --availability active "$NODE" on a Swarm manager, only after every container in /var/tmp/docker-predrain.txt is running and healthy.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓ss -ltnp shows none of the published container ports from the pre-check still bound on this host
• ✓A request from another host to this host on the service port is refused or times out - it does not return a 200
• ✓The load balancer status page lists this host as out of rotation, and its request counter for this host is zero
• ✓docker ps returns no rows, or on a Swarm node docker node ls shows AVAILABILITY Drain for this node
• ✓docker ps -a --format "{{.Names}} {{.Status}}" shows Exited (0) for every stateful service - no Exited (137)
• ✓sudo lsof +D on each named volume mountpoint returns nothing, proving nothing still holds the data open
• ✓The service dashboard shows the same total request rate as before the drain, absorbed by the remaining hosts
• ✓After the maintenance, every name in /var/tmp/docker-predrain.txt appears in docker ps, and docker ps --filter health=unhealthy prints nothing

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Abandoning a drain is safe and is often the right call. Restart in reverse: stateful services first, then the middle tier, then the front door.
• ↶Bring a stack back with docker compose -f "$COMPOSE_FILE" start, which restarts the existing containers rather than recreating them and so preserves anonymous volumes.
• ↶On a Swarm node, undo the drain from a manager with docker node update --availability active "$NODE".
• ↶Wait for health before returning traffic: docker ps --filter health=unhealthy --format "{{.Names}}" must print nothing.
• ↶Re-add the host to the load balancer only after that check passes, and watch the error rate for one full health-check interval.
• ↶If a stateful container was killed at its timeout rather than exiting cleanly, do not simply restart it and return traffic. Start it in isolation, confirm it recovers its own state, and only then re-add the host.
• ↶Restoring a volume from backup is not part of this rollback. If a volume is damaged, stop and use the restore runbook with the data owner present.

6 · Escalation

When the runbook isn't enough, contact:

• · The established connection count never falls after removing the host from the load balancer: escalate to the network or load-balancer owner - something is still routing to this host.
• · A stateful container refuses to exit within its timeout and is killed with SIGKILL: escalate to the service owner before restarting it, and capture docker logs first.
• · lsof shows a process outside Docker holding files open under a volume mountpoint: escalate to the host owner; a bind mount is being used by something you are not managing.
• · On a Swarm node, tasks do not reschedule after the drain: escalate to the Swarm owner and return the node to active rather than proceeding with maintenance.
• · The remaining hosts cannot absorb the traffic and the error rate rises: abort the drain immediately, return the host to rotation, and escalate to capacity planning.

References

1. docker stop - signal and timeout behaviour
2. docker ps - status and health filters
3. Compose file reference - stop_grace_period, stop_signal, depends_on
4. Drain a node on the swarm
5. Start containers automatically (restart policies)