Error handling - set -euo pipefail and trap

Shell scripts fail in unexpected ways. Error handling makes failures predictable and recoverable. This lesson covers the production patterns.

set -euo pipefail

At the top of every production script:

set -euo pipefail

• -e: exit on any command failure.
• -u: error on undefined variables.
• -o pipefail: pipeline fails if any command in the pipe fails.

Without these, scripts continue past errors and produce unpredictable results.

Caveats

set -e is not a guarantee. It is a set of rules with real exemptions, and the exemptions are where production scripts silently carry on after a failure. Know all of them:

• Conditionals. A command whose status is tested is exempt: the condition of if and while, anything to the left of && or ||, and anything under !. This is by design — the script is already handling that status.

• Any command followed by ||. cmd || true is the idiom for “failure is expected here”, and it works because of the rule above.

• Arithmetic that evaluates to zero. ((count++)) on a count of 0 returns the old value, 0, which the shell reads as a failure status of 1 — and set -e kills the script. A counter increment is the usual victim:

  set -e
  count=0
  ((count++))          # exits the script: the expression evaluated to 0
  count=$((count + 1)) # safe: an assignment, not an arithmetic command
  ((count++)) || true  # also safe, if you want the ++ form

• Assignment from a command substitution. When the substitution is part of a local, declare, export or readonly, the status you see is the declaration’s, not the command’s — the failure is discarded and set -e never fires:

  set -e
  bad()  { local out=$(false); echo "still running, \$? is $?"; }   # prints 0
  good() { local out; out=$(false); echo "never reached"; }         # exits

  ShellCheck flags this as SC2155. Declare on one line, assign on the next.

• Subshells whose status is consumed. ( cmd_a; cmd_b ) || handler puts the whole subshell to the left of ||, so set -e is suspended for everything inside it.

• set -u and the - expansions. ${VAR-default} and ${VAR:-default} are exactly the forms that are safe under set -u — supplying a default is their purpose, and neither triggers an unbound-variable error. It is the bare $VAR that aborts. Use ${VAR:?message} where the variable is mandatory and you want a named error rather than a generic one.

For commands where failure is expected:

# Allow failure
cmd || true

# Or: check explicitly
if ! cmd; then
    handle_error
fi

Which stage of the pipeline failed?

set -o pipefail tells you that a pipeline failed. It does not tell you where, and the exit status you get is the last non-zero one, not the first. ${PIPESTATUS[@]} holds the status of every stage:

set -o pipefail
restic backup /data | tee /var/log/backup.log | gzip > /var/log/backup.log.gz
status=("${PIPESTATUS[@]}")
for i in "${!status[@]}"; do
    (( status[i] == 0 )) || echo "pipeline stage $i exited ${status[i]}" >&2
done

PIPESTATUS is rebuilt by every command, including the echo you were about to use to debug it. Copy it into an array on the line immediately after the pipeline, before anything else runs:

false | true | false
echo "checking..."               # this overwrites PIPESTATUS
echo "${PIPESTATUS[@]}"          # 0 - the status of the echo above

Without pipefail, that pipeline exits 0 because only the last stage counts. With pipefail it exits 1, and PIPESTATUS shows 1 0 1 — which is what tells you the backup itself failed, not just the compression.

Exit codes

Use specific exit codes in production scripts:

E_OK=0
E_USAGE=1
E_RUNTIME=2
E_NOT_FOUND=3

[[ $# -eq 1 ]] || { err "Usage: $0 <file>"; exit $E_USAGE; }

trap

Catch signals for cleanup:

TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT

# Now use $TMPDIR; cleanup runs on script exit

trap '...' EXIT runs on any exit (success, error, signal).

Multiple traps:

cleanup() {
    rm -rf "$TMPDIR"
    log "Cleaned up"
}
trap cleanup EXIT INT TERM

Error reporting

Log errors with context:

log() {
    echo "[$(date -Is)] $*" >&2
}

err() {
    local rc=$?
    log "ERROR: $*" "rc=$rc" "line=${BASH_LINENO[0]}/>"
    exit 1
}

${BASH_LINENO[0]}/> is the current line number. Useful in debugging.

Common patterns

Cleanup on failure

cleanup() {
    rm -rf "$TMPDIR"
}
trap cleanup EXIT INT TERM

Log to syslog

log() {
    logger -t my-script "$*"
}

Retry on transient failures

retry() {
    local max_attempts=3
    local delay=5
    local attempt=1
    while ((attempt <= max_attempts)); do
        if "$@"; then
            return 0
        fi
        log "Attempt $attempt failed; retrying in $delay seconds"
        sleep "$delay"
        ((attempt++))
    done
    err "Failed after $max_attempts attempts: $*"
}

retry curl -fsSL https://example.com/install.sh

Checkpoint

checkpoint() {
    local name="$1"
    echo "$name" >> /var/lib/my-script/checkpoint
}

if grep -q "step-3" /var/lib/my-script/checkpoint; then
    log "Resuming from step 3"
else
    log "Starting from beginning"
fi

Knowledge check