Shell scripts: the first automation, and where it breaks

Nearly every estate that ends up running configuration management passed through a stage of shell scripts first, and the scripts usually worked.

This lesson does not treat that stage as a mistake to be embarrassed about. A script that solves a real problem for two years is a successful piece of engineering. The question is not whether shell scripting was wrong; it is which specific properties it cannot have, so you can tell whether the script you are looking at should be improved or replaced.

The script under discussion

Here is a realistic one. Not a strawman — this is close to what a competent engineer writes when they need to update an NTP server across a fleet and have twenty minutes.

#!/usr/bin/env bash
set -euo pipefail

NEW_NTP="ntp.example.com"

while read -r host; do
  echo "=== $host"
  ssh "$host" "
    sudo sed -i 's/^pool .*/pool ${NEW_NTP} iburst/' /etc/chrony/chrony.conf
    sudo systemctl restart chrony
    chronyc sources | head -3
  "
done < hosts.txt

It is not badly written. It has set -euo pipefail, it quotes its variables, it reads the host list from a file rather than hard-coding it, and it prints a per-host header so you can follow along. Many production estates have run something very like it, successfully, for years.

Now work out what it does when host 40 of 200 is powered off.

What actually happens at host 40

$ ./update-ntp.sh

=== web38
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp.example.com               2   6    17     4   +112us
=== web39
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp.example.com               2   6    17     3    -87us
=== web40
ssh: connect to host web40 port 22: No route to host

And then it stops. Hosts 41 through 200 are never touched.

The reason is set -e combined with the exit status of ssh. When ssh cannot connect it exits 255; set -e sees a non-zero status from a command that is not part of a conditional and terminates the script. The loop does not continue.

The obvious fix, and why it makes things worse

The natural response is to stop the loop dying:

#!/usr/bin/env bash
set -uo pipefail

NEW_NTP="ntp.example.com"

while read -r host; do
  echo "=== $host"
  if ! ssh -o ConnectTimeout=5 "$host" "..."; then
    echo "FAILED: $host" >&2
  fi
done < hosts.txt

This is better and it is what most people write next. It now completes all 200 hosts and prints a list of failures to stderr.

It has also just introduced the problem the previous lesson ended on. The run now produces two hundred lines of output plus some failures interleaved on stderr, and the operator’s actual question — which hosts are now correct? — is answerable only by reading all of it. In practice nobody does. The run “completed”, the terminal scrolled, and the nine hosts that failed are drift as of that moment.

The four things it cannot give you

Everything above is fixable with more shell. You can capture per-host exit codes into an array, distinguish 255 from other codes with a wrapper, write structured output as JSON with jq, and re-run only failures. People do all of this, and the result is a fifteen-hundred-line shell program that is a configuration management tool with fewer users and no tests.

The point is not that shell cannot do it. The point is which properties you have to build, because they are not there by default.

1. Per-host structured success and failure

The loop produces a text stream. Correlating outcome with host means parsing prose you wrote yourself, and the parsing breaks the first time a remote command prints something unexpected.

What you actually want is a record per host with a machine-readable outcome: succeeded, failed, unreachable, or made no change because it was already correct. Building that in shell means designing a protocol between the remote command and the loop, and then maintaining it.

2. Safety on a second run

This is the biggest one. Look at the sed again:

sudo sed -i 's/^pool .*/pool ntp.example.com iburst/' /etc/chrony/chrony.conf

Run it twice and it is fine, because the second pass matches the line it already wrote and replaces it with itself. That is lucky, not designed.

Now consider the version somebody writes a month later when they need to add a line rather than replace one:

echo "pool ntp.example.com iburst" | sudo tee -a /etc/chrony/chrony.conf

Run that twice and the file has the line twice. Run it eleven times and chrony is configured with the same pool eleven times. Nothing errors. Nobody notices until somebody reads the file.

This is the property called idempotency, and it is the difference between a script you can safely re-run and one you cannot. A script you cannot re-run cannot be used for convergence, cannot be used to verify, and cannot be resumed after it stops at host 40 — because you do not know which of the first 39 already have the change.

Shell can be idempotent. It requires you to write every operation as “check, then act if needed”, every time, by hand, correctly:

if ! grep -qx "pool ntp.example.com iburst" /etc/chrony/chrony.conf; then
  echo "pool ntp.example.com iburst" >> /etc/chrony/chrony.conf
  systemctl restart chrony
fi

That is the correct version. It is four lines instead of one, it has to be done for every single operation, and it is the thing people skip when they are in a hurry — which is when they are writing the script.

3. A description of intended state rather than steps

The script says do this. It does not say this should be true.

The difference sounds academic until you need to answer a question the steps cannot answer. “Are all forty hosts using the right NTP server?” requires a description of what right means, held separately from the procedure that achieves it. The script contains the answer only in the sense that a recipe contains a cake.

This is also why a script cannot detect drift. Running it does not tell you what was wrong; it just makes everything match, silently, including the hosts that were already fine. The information about what differed — which is the information you actually wanted — is destroyed by the act of fixing it.

4. A review artefact

A change to the fleet arrives as a modification to a script that also contains connection handling, error handling, logging and iteration logic. A reviewer looking at a diff has to separate the intent of the change from the machinery around it.

That is a solvable problem with discipline. It is a much smaller problem when the intent is a declarative statement and the machinery lives somewhere else entirely.

When a script is still the right answer

The four limits above are real, and none of them means “never write a script”. Scripts remain correct for:

The last row is worth dwelling on. A read-only loop that gathers information from a fleet has none of the four problems, because it makes no changes, so idempotency is irrelevant and there is no state to declare.

$ while read -r host; do
printf '%-10s %s\n' "$host" "$(ssh -o ConnectTimeout=5 "$host" 'uname -r' 2>&1 | head -1)"
done < hosts.txt

web01      6.8.0-45-generic
web02      6.8.0-45-generic
web03      6.5.0-41-generic
web04      ssh: connect to host web04 port 22: No route to host

Note that even here the failure mode leaks into the data: web04 has a connection error where the other rows have a kernel version, in the same column, as a string. Anything downstream that treats this as a table now has an error message in a version field. That is the first limit showing up even in the read-only case — but at a severity you can live with.

Improve or replace?

A practical test, applied to a script you have inherited:

1. Does it change state? If not, leave it alone. Read-only loops are fine.
2. Is it safe to run twice? If you cannot answer without reading every line, that is the answer.
3. How often does it run? Once a year: leave it. Weekly: replace it.
4. Does anyone re-run it to check? If the answer is no because it is not safe to, that is the limit biting.
5. Has it grown error handling, retry, host lists and logging? If so it is becoming a configuration management tool. Replacing it with one that already exists is cheaper than finishing it.

Knowledge check