Job logging and failure detection - knowing what happened

A scheduled job that runs silently is a job whose failures are invisible. Logging and alerting make failures visible.

Capture output

cron

cron emails the output if MAILTO is set:

MAILTO=ops@example.com
0 2 * * * /usr/local/bin/backup.sh

Without MAILTO, output is lost. Alternative: log to a file:

0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

systemd

systemd captures stdout and stderr to journald:

[Service]
ExecStart=/usr/local/bin/backup.sh
StandardOutput=journal
StandardError=journal

View:

sudo journalctl -u backup.service
sudo journalctl -u backup.service --since today

Exit codes

• 0: success.
• non-zero: failure.

cron: cron does not interpret exit codes but does email the output if there is any.

systemd: Type=oneshot with Restart=no (default). If the service exits non-zero, it is recorded as failed in journald.

is-active is the wrong predicate for a oneshot job

A Type=oneshot unit runs, finishes, and goes back to inactive (dead). That is what success looks like. So this pattern alerts on every successful run:

# WRONG - fires on success as well as failure
sudo systemctl is-active backup.service || \
    logger -t backup "backup failed"

Check it against a real oneshot unit that completed successfully:

$ systemctl is-active systemd-tmpfiles-clean.service; echo exit=$?; systemctl show -p Type -p Result -p ActiveState systemd-tmpfiles-clean.service

inactive
exit=3
Type=oneshot
Result=success
ActiveState=inactive

is-active returned “inactive” and exit 3 on a run that succeeded. Result=success is the field that carries the truth.

Use is-failed instead. It is true only when the unit actually entered the failed state:

# CORRECT
systemctl is-failed --quiet backup.service && \
    logger -t backup "backup FAILED"

# Or read the result directly
systemctl show -p Result --value backup.service   # success | exit-code | timeout | signal

is-active remains the right check for a long-running daemon and for a .timer unit, both of which are supposed to stay active. It is only wrong for the oneshot job that is meant to end.

Better: let systemd tell you

Polling asks a question at a moment you chose. OnFailure= fires the instant the unit fails, with no scheduler of its own:

# /etc/systemd/system/backup.service
[Unit]
OnFailure=job-failure-notify@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

# /etc/systemd/system/job-failure-notify@.service
[Unit]
Description=Notify that %i failed

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure %i

%n expands to the failing unit’s full name, %i to the instance string, so one template unit serves every job on the host. Verify a unit file before installing it with systemd-analyze verify /path/to/unit.

Detect missed runs

cron: a missed run is silent. No output, no email.

systemd: OnCalendar= with Persistent=true runs missed runs on next boot. Missed runs without Persistent=true are dropped.

Detect with monitoring:

# Check when the timer last fired
sudo systemctl list-timers --all | grep backup

# Alert if the last run was more than 25 hours ago
LAST_RUN=$(stat -c %Y /var/lib/my-job)
NOW=$(date +%s)
if (( (NOW - LAST_RUN) > 25 * 3600 )); then
    logger -t my-job "Last run > 25h ago"
fi

Alerting patterns

Job failed

# Systemd - is-failed, not is-active (see above)
systemctl is-failed --quiet backup.service && {
    curl -X POST https://alerts.example.com/webhook \
        -d '{"text":"backup service failed"}'
}

Wiring a webhook to is-active is how a team ends up muting the backup alert in week one: it pages on every successful run, so the genuine failure - which looks identical - arrives into a filter.

Job didn’t run

For scheduled jobs, monitor that they actually ran:

# Prometheus
- alert: BackupMissedRun
  expr: time() - backup_last_success_timestamp > 90000
  for: 1h

Job too slow

A job that takes longer than usual is a sign of trouble:

- alert: BackupSlow
  expr: backup_last_run_duration > 3600

Distinguish causes

A job failure can be:

• The job script crashed: application bug, missing input.
• The infrastructure failed: disk full, network down, service unavailable.
• The job did not run: timer not enabled, machine off.

For each cause, a different response. Monitoring that distinguishes these:

# Is the timer armed? is-active IS correct here - a timer stays active
systemctl is-active backup.timer

# Did the last run fail? is-failed, because a healthy oneshot is inactive
systemctl is-failed backup.service

# How did it end, and when?
systemctl show -p Result -p ExecMainStatus -p ExecMainExitTimestamp backup.service

# Check last journal entries
sudo journalctl -u backup.service -n 20

# Check resources
df -h /var/lib/backup

The two systemctl predicates answer different questions. is-active backup.timer tells you whether the job is still scheduled. is-failed backup.service tells you whether the last execution went wrong. A timer that is inactive and a service that never failed is the “job did not run” case - the one cron cannot report at all.

Knowledge check