Runbook: Investigate a shell pipeline that produced wrong output

Symptoms

• A scheduled shell script (cron job, systemd timer, CI step) finishes with exit 0 but produces wrong numbers, missing entries, or empty output.
• A monitoring check that depends on the script reports healthy when the underlying service is not.
• A diff between “yesterday’s output” and “today’s output” shows an unexpected change that does not match reality.

Diagnosis

The fastest path is to break the pipeline apart and inspect each stage in isolation; do not change the script until you have evidence of which stage misbehaved.

1. Reproduce manually. Run the script directly from a shell, exactly as the scheduler would invoke it; capture stdout, stderr, and the exit code. bash -x ./script.sh traces every command if you need to see expansion
2. Identify the failing stage. Without changing the script, inject tee between stages and re-run. The first stage whose output diverges from expected is the offender
3. Check the data, not the code. The most common cause of a "broken pipeline" is the input changed. head -1 the input; if a header line, a comment, or a field separator moved, the rest of the pipeline misbehaves
4. Check locale-sensitive tools. sort, grep, and awk all default to the C locale. If the input contains accented characters or non-ASCII punctuation, the order, match, or count may be wrong. Set LC_ALL=C explicitly when the data is ASCII
5. Check exit codes per stage. set -o pipefail is often not set. A failed left-hand command silently reports the right-hand success. Run each stage manually with set -e and confirm
6. Quote and glob check. Look for unquoted variables that should be quoted ($file vs "$file"), unquoted globs that should be quoted, and missing null termination between find and xargs

Resolution

Common fixes, with the diagnostic that points to each:

1. Regex does not match the new log format. Symptom: awk/grep output is empty or partial. Fix: update the regex; add a fallback pattern; consider JSON parsing with jq
2. Field separator changed. Symptom: awk prints the wrong field. Fix: set FS explicitly with -F; avoid relying on default whitespace splitting when the input may have leading whitespace
3. Locale issue with sort. Symptom: ranked output has upper-case before lower-case, or accented characters are out of place. Fix: LC_ALL=C sort or sort -f (case-insensitive)
4. pipefail was not set. Symptom: a failed left-hand stage is masked by a successful right-hand stage. Fix: set -o pipefail at the top of the script
5. Word splitting broke on a filename with spaces. Symptom: downstream command reports "file not found" or produces output for files that do not exist. Fix: null-terminate the pipeline with find -print0 | xargs -0 or quote every variable
6. The input file was empty or rotated. Symptom: pipeline silently produces empty output. Fix: check for input existence and non-empty size before processing; alert on "empty pipeline output" as a separate signal
7. Cron ran with a different PATH. Symptom: command not found in cron, works in shell. Fix: set PATH explicitly at the top of the cron entry

Verification

1. The corrected pipeline produces expected output on the original input. Run with the same data the scheduler would have seen; compare to a known-good baseline
2. The script reproduces successfully three times in a row. Random data, ordering, and timestamps should not change the answer
3. The original scheduled invocation (cron, timer) succeeds after redeployment. Confirm by waiting for the next scheduled run, or trigger it manually if the scheduler supports it
4. No new errors in the journal during the next scheduled run. journalctl -u <service-or-timer> --since "1 day ago" should show clean execution

Escalation

If the source data is the problem — a new field, a new log format, a different delimiter — escalate to the team that owns the producer of that data. The pipeline is correctly written for the contract it was given; the contract changed.

If the failure cannot be reproduced — happens once, then works — escalate to the application owner. The intermittent class of bug usually lives in a race or a backpressure signal in the producer.

If the script is part of a monitoring path, the on-call alerting chain may also be affected. Notify the monitoring owner; consider manually emitting a synthetic alert to confirm the alerting path itself is still functional.