A maintenance schedule you can actually leave running

The previous five lessons produced a set of commands. This one turns them into something that runs every week without you, which is the only form of maintenance that actually happens.

The design constraint is the whole lesson: an unattended prune has no operator to read the confirmation prompt, so the scoping has to be right before the -f goes on.

What is safe to automate, and what is not

The line is not “destructive versus safe”. Every command above is destructive. The line is recreatable versus not: a pruned image is a re-pull, a pruned build cache is a slow build, a pruned volume is a restore from backup if you have one and a resignation letter if you do not.

The script

Report, act, report. The two reports are what make the journal entry useful six weeks later when someone asks where an image went.

#!/usr/bin/env bash
set -euo pipefail

RETAIN_LABEL="org.example.retain=true"
CONTAINER_AGE="24h"
CACHE_AGE="168h"
CACHE_FLOOR="10GB"

echo "=== docker maintenance starting"
echo "--- before"
df -h --output=source,size,used,avail,pcent /var/lib/docker | tail -1
docker system df

echo "--- stopped containers older than $CONTAINER_AGE"
docker container prune -f --filter "until=$CONTAINER_AGE"

echo "--- dangling images"
docker image prune -f

echo "--- unreferenced images without the retain label"
docker image prune -a -f --filter "label!=$RETAIN_LABEL"

echo "--- unused networks"
docker network prune -f --filter "until=$CONTAINER_AGE"

echo "--- build cache older than $CACHE_AGE, keeping $CACHE_FLOOR"
docker builder prune -f --filter "until=$CACHE_AGE" --reserved-space "$CACHE_FLOOR"

echo "--- orphaned volumes (REPORT ONLY, never deleted here)"
docker volume ls --filter dangling=true --format '{{.Name}}' | wc -l
docker system df --format '{{.Type}} {{.Reclaimable}}' | grep -i volume || true

echo "--- after"
df -h --output=source,size,used,avail,pcent /var/lib/docker | tail -1
docker system df

echo "=== docker maintenance complete"

Three things about this script are deliberate:

1. docker image prune -a carries a label filter. Without the filter it is the command that removes your rollback. With it, the retention policy from the previous lesson is what decides, and that policy is written down.
2. Volumes are counted, never removed. The count is the alert. Somebody reads it and runs the audit.
3. df brackets the whole run. docker system df tells you what Docker thinks; df tells you what the filesystem thinks. When they disagree, the gap is logs or bind mounts, which is exactly the finding you want.

The unit and the timer

Use systemd rather than cron. The reasons are practical: output lands in the journal with the unit name attached, Persistent=true catches up a run missed while the host was down, and systemctl status answers “did it run and did it work” in one command.

[Unit]
Description=Docker housekeeping - prune recreatable objects
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/docker-maintenance
Nice=10
IOSchedulingClass=idle

[Unit]
Description=Weekly Docker housekeeping

[Timer]
OnCalendar=Sun 03:30
RandomizedDelaySec=1800
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now docker-maintenance.timer
systemctl list-timers docker-maintenance.timer

Nice=10 and IOSchedulingClass=idle matter more than they look. Pruning several gigabytes of layers is I/O-heavy, and an unniced prune on a busy host adds latency to every container on it. The maintenance job should always lose to production traffic.

Reading the outcome

# Schedule and last run
systemctl list-timers docker-maintenance.timer

# Exit status of the most recent run
systemctl status docker-maintenance.service

# Full output of the last run
journalctl -u docker-maintenance.service -n 200 --no-pager

# Just the last four weeks of before/after lines
journalctl -u docker-maintenance.service --since '4 weeks ago' --no-pager | grep -E 'before|after|reclaimed'

The last command is the one worth building a habit around. A month of before/after pairs tells you the growth rate, which is the number that decides whether the schedule is weekly or daily and whether the disk is big enough.

Thresholds and alerting

Scheduled maintenance is not a substitute for an alert. The schedule handles the predictable growth; the alert catches the day something grows a hundred times faster than usual.

#!/usr/bin/env bash
set -euo pipefail

WARN_PCT=75
CRIT_PCT=90

USED=$(df --output=pcent /var/lib/docker | tail -1 | tr -dc '0-9')

if [ "$USED" -ge "$CRIT_PCT" ]; then
echo "CRITICAL: /var/lib/docker at ${USED}%"
docker system df
exit 2
elif [ "$USED" -ge "$WARN_PCT" ]; then
echo "WARNING: /var/lib/docker at ${USED}%"
exit 1
fi

echo "OK: /var/lib/docker at ${USED}%"

Set the warning low enough that maintenance is a scheduled task rather than an incident. 75% on the partition holding /var/lib/docker gives you days of notice on most hosts; 90% gives you hours.

The monitoring part of this course covers where these checks belong and how to route them. What matters here is that the threshold is on the filesystem, not on docker system df — because the consumers Docker does not account for, container logs above all, are the ones most likely to fill a disk overnight.

Knowledge check