Automation with shell, systemd, and Ansible · Docker & Containers

Automation is usually described as “replacing manual steps with a script”. That description is why so much of it is dangerous. The property that matters is not that a machine runs the steps instead of a person; it is that running the steps again is safe.

Almost every Docker automation defect traces back to that one property.

The idempotence problem

Here is the entire problem in two commands.

Service impact possiblerun it twice

$ docker run -d --name web -p 8080:80 nginx:1.27
docker run -d --name web -p 8080:80 nginx:1.27

a3f1c2b8e4d9...
docker: Error response from daemon: Conflict. The container name "/web" is
already in use by container "a3f1c2b8e4d9". You have to remove (or rename)
that container to be able to reuse that name.

Illustrative output

The second invocation fails. Under set -e — which every competent shell script has — the script aborts, and every step after it never runs. Your “deploy” script now half-deploys, and which half depends on whether the container happened to exist.

Now drop the --name to avoid the conflict, which is the usual instinct:

Service impact possiblethe worse fix

$ docker run -d -p 8080:80 nginx:1.27
docker run -d -p 8080:80 nginx:1.27

b71d0aa4f52c...
docker: Error response from daemon: driver failed programming external
connectivity on endpoint quirky_bose: Bind for 0.0.0.0:8080 failed: port is
already allocated.

Illustrative output

Still fails, for a different reason, and now you have an anonymous container you cannot find by name. Remove the port publication too and it finally “succeeds” — by leaving two identical containers running, both connected to the application network, both registered in Docker’s embedded DNS under the same service alias, splitting traffic between an old build and a new one.

That is the failure that costs a night: the deploy script exits zero, the dashboards look fine, and roughly half of requests hit an image from three weeks ago.

docker run is not idempotent, and it has no flag that makes it so. There is no --if-not-exists, no --replace, no declarative mode. This is not an oversight — docker run is an imperative command that creates a thing. The gap is real, and everything in the rest of this lesson exists to fill it.

Three grades of shell idempotence

Shell can approximate idempotence. It is worth knowing the grades, because the difference between them is the difference between a script that works and a script that hides failures.

Configuration changefrom worst to acceptable— Three ways to make a create operation repeatable, and what each one hides.

NET=app-net

# Grade 1: swallow every error. Never do this.
# A typo, a permission failure and a daemon outage are all "fine" now.
docker network create "$NET" 2>/dev/null || true

# Grade 2: check first. Better, but it is a check-then-act race, and
# it still cannot detect that the network exists with the WRONG subnet.
docker network inspect "$NET" >/dev/null 2>&1 || docker network create "$NET"

# Grade 3: check first, and check the properties you care about.
if docker network inspect "$NET" >/dev/null 2>&1; then
SUBNET=$(docker network inspect "$NET" \
  --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}')
if [ "$SUBNET" != "192.0.2.0/24" ]; then
  echo "FAIL: $NET exists with subnet $SUBNET, expected 192.0.2.0/24" >&2
  exit 1
fi
else
docker network create --subnet 192.0.2.0/24 "$NET"
fi

Grade 1 is the one you will find in most blog posts, and it is worse than no error handling at all: it converts every failure, including the ones you desperately want to know about, into silence.

Grade 3 is what an Ansible module does internally, and writing it out makes the cost obvious. That is roughly fifteen lines to make one resource converge. A Compose stack has containers, networks, volumes and images. Doing this properly in shell for all of them is several hundred lines of error-prone code that duplicates, badly, something that already exists.

That is the honest argument for the tools. Not “shell is unprofessional” — shell is fine — but that convergence is a solved problem and re-solving it per-script is where the bugs live.

systemd for containers

systemd’s contribution is different from Ansible’s. Ansible converges state when you run it; systemd keeps state converged, restarts things that die, orders startup, and gives you systemctl status and journalctl as the interface. For a single host, that is often exactly the missing piece.

It also has one trap that catches nearly everyone.

`docker run -d` from a unit supervises the CLI, not the container

The obvious unit is the wrong one:

[Service]
ExecStart=/usr/bin/docker run -d --name web nginx:1.27

docker run -d starts the container and exits immediately. With the default Type=simple, systemd treats the process it forked as the main process of the service — so the service’s main process is a CLI invocation that lives for about 300 milliseconds.

Everything downstream is then wrong:

systemctl status web reports the service as failed or inactive within a second of starting it, while the container runs perfectly.
Restart=on-failure fires immediately and repeatedly, because the “main process” keeps exiting. The unit hits its start rate limit and gives up.
systemctl stop web kills a CLI process that has already exited. The container keeps running. You now have a service systemd believes is stopped and a container serving traffic.
The journal contains the output of docker run, which is a container ID, and nothing from the application at all.

The last point is the one that persists longest. Teams live with the inaccurate status for months and only discover the problem during an incident, when systemctl stop does not stop the thing.

The correct shape runs the container in the foreground so that the process systemd supervises is the one whose lifetime matches the container.

Configuration change/etc/systemd/system/web.service— A container unit where every directive is load-bearing. Annotations follow.

[Unit]
Description=web (nginx container)
After=docker.service network-online.target
Requires=docker.service
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=simple
Restart=always
RestartSec=10
TimeoutStartSec=120
TimeoutStopSec=70

# Remove a container left behind by an unclean shutdown. The leading "-"
# means systemd ignores a non-zero exit, which is the normal case: on a
# clean start there is nothing to remove.
ExecStartPre=-/usr/bin/docker rm -f web

# Foreground. No -d. --rm so a crashed container does not block the next
# start. --sig-proxy is on by default, so SIGTERM reaches PID 1 in the
# container.
ExecStart=/usr/bin/docker run --rm --name web \
--publish 127.0.0.1:8080:80 \
--read-only \
--tmpfs /var/cache/nginx \
--tmpfs /var/run \
--security-opt no-new-privileges=true \
nginx:1.27

# docker stop sends SIGTERM then SIGKILL after its own grace period.
ExecStop=/usr/bin/docker stop --timeout 30 web

[Install]
WantedBy=multi-user.target

Directive by directive, because each of these is a bug you would otherwise find in production:

After=docker.service and Requires=docker.service. After= orders; Requires= creates the dependency. You need both. After= alone means “if docker is being started, start after it” — on a boot where something masked docker.service, your unit starts anyway and fails.
Type=simple with a foreground docker run. The supervised process is the CLI, which now stays alive for as long as the container does and proxies signals to it. docker run -d is what breaks this.
ExecStartPre=-/usr/bin/docker rm -f web. After a hard power loss the container record survives and the next docker run --name web fails with a name conflict — the idempotence problem, arriving via a different door. The leading - makes the ordinary “no such container” exit non-fatal.
RestartSec=10 with StartLimitBurst=5 over StartLimitIntervalSec=300. Without a rate limit, a container that fails to start restarts forever and saturates the disk with image-layer churn and journal writes. With it, five failures in five minutes puts the unit into failed and stops, which is a state a human can see. Note the two StartLimit* directives belong in [Unit], not [Service] — misplacing them is silent, and they simply do not apply.
TimeoutStopSec=70 against docker stop --timeout 30. systemd’s stop timeout must exceed Docker’s, or systemd SIGKILLs the CLI while Docker is still waiting out its own grace period, and the container is orphaned mid- shutdown. Leave headroom.

Never combine `--restart` with systemd `Restart=`

If the unit above also carried --restart unless-stopped, two supervisors would be managing the same container with no knowledge of each other, and the results are genuinely strange:

The container dies. Docker’s restart policy brings it back. systemd’s supervised docker run process has already exited, so systemd also tries to start it — and its ExecStartPre removes the container Docker just restarted.
systemctl stop web runs docker stop. Docker’s unless-stopped policy correctly does not restart it. But on the next daemon restart, always would — so a service you deliberately stopped comes back after a host reboot, ignoring systemd entirely.
Boot ordering breaks: Docker starts its restart-policy containers as soon as the daemon is up, before the units that depend on them, and before anything you sequenced with After=.

Pick one supervisor. If the container is in a systemd unit, run it with no --restart flag at all and let Restart= in the unit own it. If you are using Docker’s restart policies, do not wrap those containers in units.

What systemd is actually supervising, and why the foreground form works

There are three processes involved and they are easy to conflate.

The docker CLI you launched. It is a client. It makes an HTTP request to /var/run/docker.sock.
dockerd, the daemon, which is a long-running service systemd already manages separately. It asks containerd to create the container.
The container’s PID 1, which is a child of a containerd shim (containerd-shim-runc-v2) — not a child of your CLI, and not a child of your unit’s cgroup.

So systemd never supervises the container process directly. It cannot: the container is parented into containerd’s process tree, which is why a container survives a daemon restart with live-restore enabled.

What the foreground form buys you is a proxy with a matching lifetime. docker run without -d attaches to the container’s streams and blocks until the container exits, then exits with the container’s exit code. That gives systemd a process whose lifetime and exit status track the container’s, which is all Type=simple needs. --sig-proxy, on by default for an attached run, forwards the SIGTERM systemd sends into the container.

The consequence worth internalising: the unit’s cgroup contains the CLI, not the workload. Resource limits set in the unit (MemoryMax=, CPUQuota=) constrain a client process doing nothing, and have no effect whatsoever on the container. Container limits go on docker run, always.

Compose stacks under systemd

A Compose stack has many containers, so the foreground trick does not apply. The accepted shape is different, and its limitations should be stated plainly.

Configuration change/etc/systemd/system/myapp-stack.service— oneshot plus RemainAfterExit is the standard Compose pattern. Read the caveat below before relying on it.

[Unit]
Description=myapp Compose stack
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/srv/myapp
ExecStartPre=/usr/bin/docker compose config --quiet
ExecStart=/usr/bin/docker compose up --detach --remove-orphans --wait
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target

Type=oneshot with RemainAfterExit=yes. The command exits, and systemd keeps the unit active anyway. That is the whole point of this pattern.
ExecStartPre=docker compose config --quiet. Parses and validates the Compose file, including variable interpolation, and fails the unit before anything is touched. Without it, an unset variable produces a stack that comes up misconfigured at boot with no obvious error.
--wait. Blocks until services are running and, for services with a healthcheck, healthy. Without it docker compose up -d returns as soon as the containers are created, so systemd reports the unit active while the application is still starting — and any unit ordered After= this one starts too early.
--remove-orphans. Removes containers from services you deleted from the Compose file. Otherwise they keep running, forever, invisible to docker compose ps.

What this pattern does not give you

Be clear-eyed: after ExecStart returns, systemd has no idea what the containers are doing. The unit shows active whether the stack is healthy or every container has crashed. systemctl restart runs down then up, which is a full stack outage, not a rolling restart. And the journal for this unit contains Compose’s output, not your application logs.

Container-level supervision comes from Docker’s own restart policies here — and that is the one case where combining them is correct, because systemd is not supervising individual containers, only the stack as a whole. Put restart: unless-stopped in the Compose file.

If you need systemd to know a container is unhealthy, you need one unit per container, which means you have rebuilt Compose badly. At that point the question is whether you need an orchestrator, which the orchestration part of this course takes up.

Ansible

Ansible’s contribution is the third thing: convergence across many hosts, expressed declaratively, with a change record.

Configuration changeroles/docker_host/tasks/main.yml— Fully-qualified collection names throughout. A bare module name resolves differently depending on installed collections.

- name: Install Docker Engine
ansible.builtin.apt:
  name:
    - docker-ce
    - docker-ce-cli
    - containerd.io
    - docker-compose-plugin
  state: present
  update_cache: true

- name: Configure the Docker daemon
ansible.builtin.copy:
  content: "{{ docker_daemon_config | to_nice_json }}"
  dest: /etc/docker/daemon.json
  owner: root
  group: root
  mode: '0644'
  validate: 'python3 -c "import json,sys; json.load(open(sys.argv[1]))" %s'
notify: restart docker

- name: Ensure Docker is running and enabled
ansible.builtin.systemd_service:
  name: docker
  state: started
  enabled: true
  daemon_reload: true

- name: Run the application container
community.docker.docker_container:
  name: web
  image: "nginx:{{ nginx_version }}"
  state: started
  restart_policy: unless-stopped
  published_ports:
    - '127.0.0.1:8080:80'
  memory: 512M
  read_only: true
  security_opts:
    - 'no-new-privileges=true'

Two lines carry more weight than they look:

validate: on the daemon.json copy. Ansible runs the validation command against the staged file and refuses to install it if the command fails. Without this, a templating mistake writes malformed JSON, the handler restarts Docker, and dockerd refuses to start — taking every container on the host with it. This one line is the difference between a failed play and an outage.
community.docker.docker_container rather than docker_container. The fully-qualified name is not pedantry: short names resolve through a search path that depends on which collections are installed, and a playbook that works on the control node and fails in CI is usually this.

`docker_container` is idempotent, but not in the way people expect

The module converges: it inspects the running container, compares it to the declared state, and acts only if they differ. Run it twice and the second run reports ok.

The surprise is what counts as “differ”. The module compares a large set of container properties, and several of them change without you changing the playbook:

The image. If nginx:1.27 has been re-pulled and now resolves to a different digest, the module recreates the container. That is usually what you want, and it means a routine “check compliance” play can restart production.
Defaults you did not specify. Some properties compare against the container’s actual value rather than against “unspecified”, so a value the daemon defaulted differently after an engine upgrade reads as drift.

Recreating a container is a service interruption. So the habit that keeps this safe is: run the play in check mode first and read the change list.

ansible-playbook -i inventory site.yml --check --diff --limit docker-prod-01

If --check reports a container change you did not intend, find out why before you run it for real. The comparisons: parameter lets you pin specific properties to ignore when you have decided a difference is acceptable.

A backup script, corrected

The volume-backup script is the most-copied piece of Docker shell automation, and the common version has two real defects.

Service impact possiblebackup-volumes.sh— Named volumes only, with the archive path bug fixed and the consistency problem handled rather than ignored.

#!/bin/bash
# Back up named Docker volumes. Anonymous volumes are deliberately skipped:
# their names are 64-hex strings that mean nothing on restore.
set -euo pipefail

STAMP=$(date -u +%Y-%m-%d)
BACKUP_DIR="/var/backups/docker/$STAMP"
S3_PREFIX="s3://myorg-backups/docker/$STAMP"
mkdir -p "$BACKUP_DIR"

# Filter out anonymous volumes: 64 lowercase hex characters, nothing else.
docker volume ls --format '{{.Name}}' \
| grep -Ev '^[0-9a-f]{64}$' > "$BACKUP_DIR/volumes.txt"

while read -r VOL; do
[ -n "$VOL" ] || continue
echo "Backing up volume $VOL"
docker run --rm \
  --network none \
  --user 0:0 \
  -v "$VOL":/source:ro \
  -v "$BACKUP_DIR":/backup \
  alpine:3.20 \
  tar czf "/backup/$VOL.tar.gz" -C /source .
done < "$BACKUP_DIR/volumes.txt"

# BACKUP_DIR is absolute, so syncing it directly would produce an S3 key of
# s3://bucket/docker/2026-08-12/var/backups/docker/2026-08-12/... Sync the
# directory CONTENTS to the prefix instead.
aws s3 sync "$BACKUP_DIR/" "$S3_PREFIX/"

# Verify each archive is readable before declaring success. A tar that was
# truncated by a full disk exits zero on creation and fails here.
for ARCHIVE in "$BACKUP_DIR"/*.tar.gz; do
tar tzf "$ARCHIVE" >/dev/null || { echo "CORRUPT: $ARCHIVE" >&2; exit 1; }
done
echo "Backup verified: $(ls -1 "$BACKUP_DIR"/*.tar.gz | wc -l) archives"

A live database volume tarred while running is not a backup

tar over a running database’s data directory copies files one at a time over several minutes. The database writes throughout. The archive contains page 1 as it was at 02:00:00 and page 90,000 as it was at 02:04:30, which is a state the database was never in.

Sometimes it restores. Sometimes it restores and corrupts silently three weeks later. It is not a backup, it is a coin flip, and the coin is weighted by how busy the database was.

For any volume holding a database, use the engine’s own dump — pg_dump, mysqldump, mongodump, a Redis BGSAVE followed by copying the RDB — and back up that output. For everything else the tar is fine.

The --network none and alpine:3.20 pinning in the script above are worth keeping for a different reason: this container is handed read access to every volume on the host. It has no business reaching the network, and it should not be a floating tag.

Choosing between the three

	Shell	systemd	Ansible
Converges state	Only what you write	No — it starts and supervises	Yes, per resource
Survives reboot	No	Yes	No (it runs when you run it)
Restarts on failure	No	Yes	No
Multiple hosts	Copy it around	No	Yes, with inventory
Records what changed	No	Journal, per start	Yes, changed/ok per task
Dry run	No	No	`--check --diff`

They are complements, not alternatives, and the usual mature arrangement uses all three: Ansible converges the host and writes the unit files, systemd runs and supervises the workloads, and shell handles the genuinely one-shot things that have no state to converge.

What happens if the automation runs while a deploy is in flight?

Nothing protects you from this by default, and the collision is not rare — a nightly Ansible compliance run at 02:00 and a CI deploy triggered by a merge in another timezone will eventually overlap.

The two failure shapes:

Ansible recreates the container CI just replaced. The play declares image: myapp:1.4.0 because that is what is in the repository; CI just deployed 1.5.0. Ansible converges it back. The rollback is silent, and the version in production is now older than the version in the deploy log.
A prune runs during a build. Covered in the maintenance part, and the same class of problem: two automations, one Docker daemon, no lock.

The fix is a lock, and it is four lines:

LOCKFILE=/var/lock/docker-automation.lock
exec 9>"$LOCKFILE"
flock -n 9 || { echo 'another automation holds the lock; exiting' >&2; exit 0; }
# ... work here; the lock is released when the script exits

flock -n fails immediately rather than queueing, and exiting 0 means a skipped nightly run does not page anyone. Every automation that touches the Docker daemon on a host should take the same lock. The only thing worse than one automation with a bug is two automations racing to fix each other.

Write the operation once, by hand, and record every command and its output. You cannot automate a procedure you have not performed.
Make it idempotent, then prove it by running it twice on a scratch host and diffing the state afterwards. Running twice is the test, not reading the code.
Give it a dry run. --check in Ansible, docker compose config for a stack, or an explicit DRY_RUN guard in shell. An automation you cannot rehearse is one you will only ever run in anger.
Take a lock before touching the daemon, so two automations cannot race.
Verify convergence, not exit status. Exit zero means the commands ran. Query the resulting state and compare it to what you intended.
Fail loudly. 2>/dev/null || true is how a broken automation runs green for six months.

Why this matters

The reason docker run is not idempotent is not a defect to route around quietly. It is a boundary: docker run creates a container, and creation is inherently a once-only operation. Every tool discussed here exists because somebody hit that boundary and needed a layer that expresses desired state rather than actions.

Recognising which side of that boundary you are on is the whole skill. A script that says “create the network” is an action and will break the second time. A script that says “the network should exist with this subnet” is a state and will not. systemd and Ansible are two well-tested implementations of the second sentence, and the reason to reach for them is not that they are more professional — it is that you have already written the state version by hand, seen how long it was, and decided not to maintain it.