Draining a Docker host before maintenance · Docker & Containers

An orchestrator drains a node for you: it stops scheduling new work there, evicts what is running, waits for it to move, and tells you when the node is empty. Standalone Docker has none of that. The host is either taking traffic or it is not, and nothing but you decides which.

Draining is the procedure that gets a host from “serving production” to “safe to reboot” without dropping requests. It has four phases and each one has a check that must pass before the next begins.

The four phases

Phase	Goal	Proof it is done
1. Stop new traffic	Upstream stops sending requests here	Connection count at the proxy reaches zero for this backend
2. Let in-flight work finish	Requests and jobs already accepted complete	No active connections; queue consumers idle
3. Quiesce stateful services	Data on disk is consistent	Checkpoint or flush completed
4. Stop containers	Nothing is running	`docker ps` is empty for the stack

Phase 1 without phase 2 drops in-flight requests. Phase 4 without phase 3 is how databases end up doing crash recovery. Skipping to phase 4 — which is what “stop the containers and reboot” does — does both.

Phase 1: stop new traffic

How you do this depends on what is in front of the host, and the mechanism matters less than the verification.

Service impact possibledrain at the edge— Three common front ends. Use the one you have.

# HAProxy runtime API: set the backend server to drain
echo "set server app_backend/app-01 state drain" | sudo socat stdio /run/haproxy/admin.sock

# nginx upstream: mark the server down and reload
sudo sed -i 's|^\(\s*server 192.0.2.11:8080\)\(.*\)$|\1 down;|' /etc/nginx/conf.d/upstream.conf
sudo nginx -t && sudo systemctl reload nginx

# A health-check-driven pool: fail the health endpoint deliberately
docker exec web touch /run/drain

The third form is the most robust and the least common. If your application exposes a health endpoint that the load balancer polls, and that endpoint can be made to fail on demand, draining becomes a single flag inside the container and the load balancer removes the host on its own schedule with no configuration change anywhere else.

Proving no traffic is arriving

This is the phase that gets asserted rather than verified, and asserting it is how requests get dropped.

Read-only / Safeverify drained— Three independent signals that traffic has stopped.

# 1. Established connections to the published port
ss -tn state established '( sport = :8080 )' | tail -n +2 | wc -l

# 2. The application's own request log, over the last 60 seconds
docker logs --since 60s web 2>&1 | wc -l

# 3. Container network counters, sampled twice
docker exec web cat /proc/net/dev | awk '/eth0/ {print $2, $10}'
sleep 10
docker exec web cat /proc/net/dev | awk '/eth0/ {print $2, $10}'

All three should stop changing. The connection count is the primary signal; the log and the counters catch the cases where the load balancer stopped sending but something else — a health checker, a cron job on another host, a message queue consumer — is still driving work through the container.

Drained at the front door, still working from the queue

The classic miss. The HTTP traffic is drained, ss shows zero established connections on 8080, and everything looks finished — but the container also runs a worker that pulls from RabbitMQ or Redis. That worker has no relationship to the load balancer and keeps pulling jobs right up until SIGKILL.

The job that was in flight is lost, or worse, half-applied: the message was acknowledged, the side effect was partially written, and it will not be retried.

Draining an HTTP frontend and draining a queue consumer are different operations. The consumer needs to be told to stop accepting new messages and allowed to finish the current one — usually a signal the application handles, which means it needs a SIGTERM handler and a stop timeout long enough to use it. If you do not know whether your workers have one, that is the thing to find out before the maintenance window.

Phase 2: let in-flight work finish

There is no command for this. It is a wait, and the length of the wait is a property of your application: the longest request it serves, plus the longest job it processes.

Read-only / Safewait for quiet— Poll until the connection count has been zero for a sustained period.

#!/usr/bin/env bash
set -euo pipefail

PORT=8080
QUIET_FOR=30
DEADLINE=$(( $(date +%s) + 300 ))
quiet=0

while [ "$(date +%s)" -lt "$DEADLINE" ]; do
n=$(ss -tn state established "( sport = :$PORT )" | tail -n +2 | wc -l)
if [ "$n" -eq 0 ]; then
  quiet=$(( quiet + 5 ))
  echo "quiet for ${quiet}s"
  [ "$quiet" -ge "$QUIET_FOR" ] && { echo 'DRAINED'; exit 0; }
else
  quiet=0
  echo "$n connection(s) still established"
fi
sleep 5
done

echo 'TIMEOUT: connections still arriving after 300s' >&2
exit 1

The deadline matters as much as the wait. A drain that never completes is telling you something — usually that the load balancer change did not take effect, or that a client is connecting to the host directly, bypassing the proxy entirely. Both are worth knowing before you take the host down.

Phase 3: quiesce stateful services

For stateless containers, phase 3 is empty. For anything holding data, it is the phase that decides whether the next boot is clean.

Service impact possiblequiesce— Ask the application to flush, rather than relying on the stop signal.

# PostgreSQL: force a checkpoint so recovery on next start is short
docker exec postgres psql -U postgres -c 'CHECKPOINT;'

# Redis: write the dataset to disk synchronously
docker exec redis redis-cli SAVE

# Then stop it with a timeout it can actually use
docker stop --timeout 60 postgres

This is the same quiesce step the application-consistent backup lesson describes, applied for a different reason. The principle is identical: ask the application to reach a consistent on-disk state while it is still running, rather than hoping it manages it in the seconds between SIGTERM and SIGKILL.

Why concurrent shutdown is the wrong shape for a stack

When you reboot without draining, the daemon stops every container as part of its own shutdown. It does not order them.

For a stack of an application and a database, the two orderings have very different outcomes:

Application stops first, then the database. The application closes its connections and finishes its writes; the database sees a clean disconnect and checkpoints normally.
Database stops first, then the application. The application’s in-flight writes fail with connection errors. Depending on how it handles that, you get failed requests, a poisoned connection pool, retry storms against a database that is gone, and log noise that obscures the real event.

The daemon’s shutdown gives you a coin flip between these. docker compose stop gives you the first one deterministically, because Compose stops services in reverse dependency order — the inverse of the order it starts them, derived from depends_on.

This is the strongest practical argument for declaring depends_on even when your application retries its connections and does not strictly need it. The dependency graph is what makes shutdown ordered.

Phase 4: stop the containers

Service impact possibleordered stop— Compose stops in reverse dependency order.

docker compose -f /srv/app/compose.yaml stop --timeout 60

# Confirm nothing is left running for this project
docker compose -f /srv/app/compose.yaml ps
docker ps --format '{{.Names}}'

stop, not down. down removes containers and networks, which is a different operation with different consequences — and down -v removes the named volumes. For maintenance you want the containers to exist and be stopped, so the reboot brings them back exactly as they were.

The reverse: undraining

Undraining is the drain in reverse, and phase 1 is last rather than first.

Start the stack. docker compose up -d.
Wait for health. Every container healthy, not merely running.
Verify locally, bypassing the load balancer — curl the published port on the host itself.
Only then return the host to the load balancer pool.
Watch one full check interval before declaring the window closed.

Step 3 is the one that saves you. Returning a host to the pool and then discovering the application does not work means you have just sent production traffic to a broken backend, which is worse than the maintenance you were doing.

What if the host must be drained but the service has only one instance?

Then draining does not avoid an outage, and pretending otherwise is the mistake.

Be explicit about which case you are in:

Multiple instances behind a load balancer. Draining converts maintenance into a non-event. This is the case the procedure above is written for.
Single instance. The drain still has value — phases 2 and 3 mean in-flight requests complete and data is flushed — but phase 1 is an announced outage, not a rerouting. Say so in the change ticket, and give the window a user-visible start and end.

The failure is announcing a “zero-downtime maintenance window” for a single instance service. The drain makes the outage clean rather than absent, and the difference matters to whoever is answering support tickets during it.

If single-instance maintenance outages are a recurring problem, that is a capacity and architecture finding rather than a maintenance one — the HA part of this course is where it belongs.