Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

intermediateConfiguration~20 min

Break/Fix: A cron job returns exit 0 but produces no check results

Reported symptoms

  • ●A scheduled monitoring check reports success every run
  • ●The output file has a header and a trailing "done" line but no check results
  • ●The on-call page never fires even though the service is unhealthy
  • ●A diff against yesterday shows "no change" — which is wrong because the service is brand new

Evidence

  • · `crontab -l` shows the entry, redirecting both streams into /var/log/healthcheck.log
  • · `ls -l /var/log/healthcheck.log` shows a small file, owned by root, modified 5 minutes ago
  • · `cat /var/log/healthcheck.log` shows a header, three "monitoring-tool: command not found" lines, and a trailing "done"
  • · `run-parts --test /etc/cron.hourly` does not list the backup copy healthcheck.sh at all
  • · `grep CRON /var/log/syslog` shows the script ran every 5 minutes; cron logs the CMD but never an exit status
  • · `PATH` in the cron environment is /usr/bin:/bin only
  • · `monitoring-tool` lives in /usr/local/bin, which is not in that PATH
  • · The script has no `set -e`, so it continues past a failed command
Diagnosis and resolutionclick to reveal

Root cause

Three independent faults stack. (1) The cron job's PATH is the minimal default (/usr/bin:/bin) but the script calls a tool installed under /usr/local/bin, so every invocation fails with "command not found". (2) The script has no `set -e`, so bash does not stop at the failed command — it writes the error to stderr and carries on. The script's exit status is the status of its LAST command, a trailing `echo` that succeeds, so the script exits 0. The redirection has nothing to do with it: `cmd > file 2>&1` always reports cmd's own status. (3) Nobody was alerted because cron does not alert on exit status at all. cron mails a job's OUTPUT to MAILTO; this crontab redirects both streams into a file, so cron produces no mail and no signal whatever the exit code. The exit status is simply discarded. The /etc/cron.hourly backup copy never ran either, because run-parts ignores any filename containing a dot.

Remediation

Add `PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` at the top of the crontab. Add `set -euo pipefail` to the script so a failed command aborts it instead of being ignored. Restore a failure signal: set `MAILTO=` and stop redirecting stderr into the file, or move the job to a systemd timer with `OnFailure=` — with the current crontab, fixing PATH alone would still have surfaced nothing. Add a sentinel heartbeat as the last line and alert on its absence. Rename the /etc/cron.hourly copy to `healthcheck` with no extension (or invoke run-parts with --regex) so run-parts will execute it.

Verification

Run the script under a faithfully minimal PATH and confirm it now exits non-zero. Confirm the log file contains the diagnostic lines rather than "command not found". Confirm `run-parts --test /etc/cron.hourly` now lists the script. Trigger a synthetic failure and confirm the page fires within one cron cycle.

Prevention

Every crontab entry should set PATH explicitly. Every scheduled script should start with `set -euo pipefail`. Every scheduled job needs a failure channel that does not depend on its own output — MAILTO, an OnFailure= unit, or a heartbeat that is alerted on by absence. Never name a file in /etc/cron.d or /etc/cron.hourly with an extension. Lint scheduled scripts with ShellCheck in CI.

Reported symptoms

  • A scheduled monitoring check reports success every run.
  • The output file has a header and a trailing “done” line but no check results.
  • The on-call page never fires even though the service is unhealthy.
  • A diff against yesterday shows “no change” — which is wrong because the service is brand new.

Evidence provided

$ crontab -l | grep healthcheck
*/5 * * * * /usr/local/bin/healthcheck.sh > /var/log/healthcheck.log 2>&1

$ ls -l /var/log/healthcheck.log
-rw-r--r-- 1 root root 268 Aug 11 04:05 /var/log/healthcheck.log

$ cat /var/log/healthcheck.log
== healthcheck 2026-08-11T04:05:01+00:00
/usr/local/bin/healthcheck.sh: line 4: monitoring-tool: command not found
/usr/local/bin/healthcheck.sh: line 5: monitoring-tool: command not found
/usr/local/bin/healthcheck.sh: line 6: monitoring-tool: command not found
done

$ head -3 /usr/local/bin/healthcheck.sh
#!/bin/bash
LOG=/var/log/healthcheck.log
echo "== healthcheck $(date -Is)"

$ grep CRON /var/log/syslog | tail -2
Aug 11 04:05:01 host CRON[12345]: (root) CMD (/usr/local/bin/healthcheck.sh > /var/log/healthcheck.log 2>&1)
Aug 11 04:10:01 host CRON[12346]: (root) CMD (/usr/local/bin/healthcheck.sh > /var/log/healthcheck.log 2>&1)

$ ls /etc/cron.hourly
0anacron  healthcheck.sh  logrotate

$ run-parts --test /etc/cron.hourly
/etc/cron.hourly/0anacron
/etc/cron.hourly/logrotate

$ which monitoring-tool
/usr/local/bin/monitoring-tool

$ env -i bash -c 'PATH=/usr/bin:/bin; /usr/local/bin/healthcheck.sh; echo exit=$?'
== healthcheck 2026-08-11T04:32:10+00:00
/usr/local/bin/healthcheck.sh: line 4: monitoring-tool: command not found
/usr/local/bin/healthcheck.sh: line 5: monitoring-tool: command not found
/usr/local/bin/healthcheck.sh: line 6: monitoring-tool: command not found
done
exit=0

$ env -i bash -c 'PATH=/usr/local/bin:/usr/bin:/bin; /usr/local/bin/healthcheck.sh; echo exit=$?'
== healthcheck 2026-08-11T04:32:44+00:00
api: OK
database: DEGRADED - replica lag 412s
queue: OK
done
exit=0

Work the evidence before reading on

Four facts have to be reconciled, and no single cause explains all of them:

  1. The tool exists on the host, yet the script cannot find it.
  2. Three commands failed, yet the script exited 0 — and it still exits 0 once PATH is repaired and the tool reports DEGRADED.
  3. The failure was written to a file the whole time, yet nobody was paged.
  4. There is a second copy in /etc/cron.hourly that never appears in any log at all.

Write down a hypothesis for each before continuing. Fixing only the first one leaves the incident able to recur silently — fact 2 shows the check would go on reporting success even with a correct PATH.

Root cause

1. cron’s PATH is not your PATH

The cron job runs with the minimal default PATH=/usr/bin:/bin. The script calls monitoring-tool, which is installed in /usr/local/bin. Without an explicit PATH line in the crontab that directory is never searched, so every one of the three calls fails with “command not found”.

2. Bash did not stop, and the exit status came from the last line

The script has no set -e. Bash writes command not found to stderr, returns 127 for that one command, and carries on to the next line. The script therefore runs to completion, and its exit status is the status of its last command — the trailing echo "done", which succeeds. So the script exits 0.

This is not only a PATH problem. The last evidence block shows the same script exiting 0 with a correct PATH while monitoring-tool reports DEGRADED. Without set -e the script ignores the tool’s exit status too, so a genuinely failing check reports success in exactly the same way.

3. cron never alerted, and never would have

This is the fault that matters most, because fixing PATH alone does not fix it.

cron does not alert on a non-zero exit status. It does not look at exit status at all. cron mails a job’s output to the address in MAILTO. This crontab sends both stdout and stderr into a file, so there is no output to mail, no mail is sent, and no signal of any kind reaches a human — whatever the exit code happens to be.

Had the PATH been correct from day one and the tool later started exiting non-zero, this crontab would still have paged nobody.

4. run-parts ignores filenames containing a dot

The backup copy at /etc/cron.hourly/healthcheck.sh has never run. run-parts — which is what executes /etc/cron.hourly — accepts only names made of ASCII letters, digits, underscores and hyphens unless --lsbsysinit or --regex is given. A .sh extension makes the file invisible to it. Note that run-parts --test lists exactly the files it would run, so the script’s absence from that listing is the evidence, not its presence.

Resolution

  1. Set PATH in the crontab. Edit with crontab -e and prepend a PATH line covering every directory the script needs:
  2. `` PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ``
  3. Make the script fail loudly. At the top of /usr/local/bin/healthcheck.sh, add set -euo pipefail so a failed command aborts the run instead of being stepped over. Quote every variable reference. Add a sentinel heartbeat as the last line: echo "$(date -Is) heartbeat $(hostname)" so the absence of the heartbeat is itself an alert
  4. Restore a failure channel. Fixing PATH alone changes nothing about alerting. Either set MAILTO=oncall@example.com in the crontab and stop redirecting stderr into the file (>> /var/log/healthcheck.log only, leaving stderr to be mailed), or move the job to a systemd timer whose service has OnFailure=alert@%n.service — systemd, unlike cron, does act on exit status
  5. Fix the cron.hourly copy. Rename /etc/cron.hourly/healthcheck.sh to /etc/cron.hourly/healthcheck — no extension — or keep the name and invoke run-parts with --regex. Confirm with run-parts --test /etc/cron.hourly that it is now listed
  6. Re-test under a faithful cron environment. env -i on its own is not enough — set the PATH explicitly:
  7. `` env -i bash -c 'PATH=/usr/bin:/bin; /usr/local/bin/healthcheck.sh; echo exit=$?' ``
  8. With set -euo pipefail in place this must now exit non-zero while PATH is still wrong, and exit 0 with a non-empty log of real diagnostics once PATH is fixed
  9. Add a CI check. Lint the script with ShellCheck in CI. Reject merges that remove set -e, drop the PATH line, or reintroduce a full 2>&1 redirect on a job with no other failure channel.

Verification

  1. Confirm the script now fails when it should. Temporarily rename the tool and run the script; with set -euo pipefail it must exit non-zero and stop at the first failure
  2. Confirm the log contains real diagnostics. cat /var/log/healthcheck.log should show check results and a heartbeat line, not "command not found"
  3. Confirm the hourly copy is reachable. run-parts --test /etc/cron.hourly should now list healthcheck
  4. Confirm the failure channel works. Force a failure and confirm mail arrives at MAILTO, or that the OnFailure= unit fired: journalctl -u alert@healthcheck.service --since "10 minutes ago"
  5. Wait for the next scheduled run. Confirm the log is rewritten with fresh output and the heartbeat timestamp advances
  6. Trigger the alert path with a synthetic failure. Stop the monitored service and confirm the page fires within one cron cycle

Prevention

  • Every crontab entry should set PATH explicitly at the top.
  • Every scheduled script should start with set -euo pipefail.
  • Every scheduled job needs a failure channel that does not depend on its own output: MAILTO, a systemd OnFailure= unit, or a heartbeat alerted on by absence. A job whose stdout and stderr both go to a file has no failure channel at all.
  • Never give a file in /etc/cron.hourly, /etc/cron.daily or /etc/cron.d an extension. run-parts silently skips it.
  • Lint scheduled scripts with ShellCheck in CI.
  • For systemd timers, set Environment="PATH=…" in the unit file — systemd drops the cron-style sparse PATH and runs with the system manager environment by default, but Environment= is still the right place to be explicit.