Idempotent shell automation — scripts that can run twice

A script that only works the first time is not automation, it is a transcript. The test is simple: run it twice in a row and once again after killing it halfway through. If the end state is the same all three times, it is idempotent. Most Docker shell scripts fail that test, and they fail it in the same three places every time.

Probing for state without matching the wrong thing

The most common existence check in Docker scripts is wrong:

The cleanest existence probe does not involve pattern matching at all. docker inspect exits 0 when the object exists and 1 when it does not:

$ docker inspect web >/dev/null 2>&1; echo $?

1

container_exists() { docker inspect --type container "$1" >/dev/null 2>&1; }
network_exists()   { docker network inspect "$1" >/dev/null 2>&1; }
volume_exists()    { docker volume inspect "$1" >/dev/null 2>&1; }

if ! network_exists app-net; then
  docker network create --driver bridge --subnet 192.0.2.0/24 app-net
fi

--type container matters: without it, docker inspect web also matches an image named web, and an image existing is not the question you asked.

set -euo pipefail and where it stops working

pipefail is the one that produces surprises in Docker scripts, because grep returns 1 when it matches nothing — which is frequently the normal case:

set -euo pipefail

# Exits the whole script when there are no stopped containers,
# because grep found nothing and pipefail propagates its status.
stopped=$(docker ps -a --format '{{.Names}} {{.State}}' | grep ' exited$')

Either handle the empty case explicitly, or do the filtering server-side where “no results” is an empty list rather than an error:

set -euo pipefail

mapfile -t stopped < <(docker ps -aq --filter status=exited)
if [ "${#stopped[@]}" -eq 0 ]; then
  echo "nothing to clean up"
  exit 0
fi
printf 'removing %d stopped container(s)\n' "${#stopped[@]}"
docker rm "${stopped[@]}"

This also fixes a second bug that set -e alone does not catch: docker rm $(docker ps -aq --filter status=exited) with no matches becomes bare docker rm, which errors with “requires at least 1 argument” — noise in the log every night on a healthy host, which trains everybody to ignore the log.

Two copies at once

#!/usr/bin/env bash
set -euo pipefail

exec 9>/var/lock/docker-backup.lock
if ! flock --nonblock 9; then
  echo "another backup run is in progress; exiting" >&2
  exit 0
fi

# ... the rest of the script; the lock is released when the shell exits

flock --nonblock exits rather than queueing, which is what you want for a periodic job — a queue of backups all starting at once when the lock finally releases is a worse outcome than a skipped run. Use exit 0 for the “already running” path so monitoring does not page on a normal condition, but log it, so a permanently stuck run is still visible.

Cleaning up after yourself

A script that creates a temporary container, a temporary network, or a temporary file has to remove them on every exit path, not just the successful one:

#!/usr/bin/env bash
set -euo pipefail

workdir=$(mktemp -d)
helper=""

cleanup() {
  [ -n "$helper" ] && docker rm -f "$helper" >/dev/null 2>&1 || true
  rm -rf "$workdir"
}
trap cleanup EXIT INT TERM

helper=$(docker run -d --rm alpine:3.20 sleep 300)
docker cp "$helper:/etc/os-release" "$workdir/os-release"
cat "$workdir/os-release"

Here || true is correct: cleanup is best-effort and must not turn a successful run into a failure. That is the distinction — suppress errors in teardown, never in the work itself.

The daemon can hang

Every docker invocation is an HTTP request to a daemon that may be unresponsive — mid-live-restore, blocked on a wedged storage driver, or waiting on an NFS mount that has gone away. The CLI has no default timeout, so a script blocks forever and the cron job never completes.

if ! timeout 30 docker version --format '{{.Server.Version}}' >/dev/null 2>&1; then
  echo "docker daemon not responding within 30s" >&2
  exit 1
fi

A liveness probe at the top of the script converts “the job hung and nobody noticed for three days” into a clear failure on the first run.

The checklist

$ shellcheck -S warning /usr/local/sbin/docker-backup.sh

In /usr/local/sbin/docker-backup.sh line 14:
for vol in $(docker volume ls -q); do
           ^-- SC2046: Quote this to prevent word splitting.

Knowledge check