Timeouts, retries and jitter - scheduled jobs that fail safely

A scheduled job that works is easy. A scheduled job that fails safely is the engineering. The three questions this lesson answers are: how long is this job allowed to take, what happens when it fails, and what happens when 400 hosts run it at the same instant.

Locking, covered in the previous lesson, stops a job overlapping itself. It does nothing about a job that hangs forever, a job that retries into a duplicate charge, or a fleet that stampedes a package mirror at 03:00:00 exactly.

Measure the runtime before you bound it

You cannot pick a timeout without knowing the distribution. The data is already recorded for every systemd-triggered job:

systemctl show backup.service -p ExecMainStartTimestamp -p ExecMainExitTimestamp

ExecMainStartTimestamp=Tue 2026-08-11 03:00:14 UTC
ExecMainExitTimestamp=Tue 2026-08-11 03:07:42 UTC

For the history rather than the last run, ask the journal for the invocation records:

journalctl -u backup.service --since '30 days ago' -o short-iso | grep -E 'Started|Succeeded|Deactivated'

2026-07-13T03:00:11+0000 host1 systemd[1]: Started backup.service - Nightly backup.
2026-07-13T03:06:58+0000 host1 systemd[1]: backup.service: Deactivated successfully.
2026-07-14T03:00:09+0000 host1 systemd[1]: Started backup.service - Nightly backup.
2026-07-14T03:08:31+0000 host1 systemd[1]: backup.service: Deactivated successfully.

Seven to nine minutes across a month. The number you want is not the mean; it is the worst observed run plus headroom for the growth you expect before anyone looks at this again. A bound of 30 minutes on a nine-minute job is not sloppy - it is the difference between “this job is stuck” and “this job is slow tonight”.

Bound the runtime

Under systemd

Which directive bounds the job depends on Type=, and picking the wrong one silently bounds nothing.

A timer-triggered batch job is almost always Type=oneshot, so it is almost always TimeoutStartSec=:

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup.sh
TimeoutStartSec=1800
TimeoutStopSec=120

When it fires, the unit enters failed with a distinctive result, which is what makes it alertable:

systemctl show backup.service -p Result -p ActiveState

Result=timeout
ActiveState=failed

Result=timeout is different from Result=exit-code. An alert that distinguishes them tells you whether the job broke or the job hung, which are different pages to different people.

For a long-running service that is not oneshot - a daemon you want recycled if it runs away - RuntimeMaxSec= is the right directive, and RuntimeRandomizedExtraSec= adds a random extra window on top so a fleet of them does not all recycle together.

Under cron

cron has no equivalent, so wrap the command:

# /etc/cron.d/backup
# m h dom mon dow user command
0 3 * * * root /usr/bin/timeout --signal=TERM --kill-after=120 1800 /usr/local/sbin/backup.sh

timeout sends SIGTERM at 1800 seconds and, if the process is still alive 120 seconds later, SIGKILL. The gap matters: a backup that gets SIGKILL with no grace period leaves its temporary state behind. Check the exit status to distinguish the cases:

timeout 1 sleep 5
echo "exit=$?"

exit=124

Exit 124 is the one to key an alert on: it means the timeout fired rather than the command failing on its own. GNU coreutils also reserves 125 for timeout itself failing, 126 for a command found but not executable, and 127 for not found; anything else is the command’s own status. --preserve-status suppresses this and returns the command’s status even on timeout, which is occasionally what a wrapper script wants and is almost never what you want in cron.

Retry, but only when the job is idempotent

A retry is safe when running the job twice produces the same result as running it once. That is a property of the job, not of the scheduler, and no systemd directive can give it to you.

For the “no” rows, make the job idempotent before you make it retry. The usual mechanism is a marker the job checks and writes in the same step - a row in a table keyed by the run date, or a state file:

#!/bin/bash
set -euo pipefail

STATE_DIR=/var/lib/daily-summary
TODAY=$(date +%F)
mkdir -p "$STATE_DIR"

if [ -e "$STATE_DIR/$TODAY.done" ]; then
  echo "already sent for $TODAY, nothing to do"
  exit 0
fi

/usr/local/bin/send-summary --date "$TODAY"
touch "$STATE_DIR/$TODAY.done"

Now the job is safe to retry, and only now should you configure a retry.

Retry in systemd

Restart=on-failure works for Type=oneshot, but it needs a burst limit or a permanently broken job will spin:

# /etc/systemd/system/sync-index.service
[Unit]
Description=Rebuild search index
StartLimitIntervalSec=3600
StartLimitBurst=3

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/sync-index.sh
TimeoutStartSec=1800
Restart=on-failure
RestartSec=120

Three attempts two minutes apart, then the unit stays failed for the rest of the hour. The rate-limit directives are documented in man 5 systemd.unit, so they belong in [Unit] - systemd tolerates them in [Service] for backward compatibility, but putting them where they are documented means the next person finds them.

Verify before enabling:

systemd-analyze verify /etc/systemd/system/sync-index.service

Output limited to unrelated warnings about units already installed on the system means your unit parsed cleanly.

Jitter: the fleet-wide problem

OnCalendar=daily fires at 00:00:00. On one host that is fine. On 400 hosts pointed at one package mirror, one backup target or one monitoring endpoint, it is a synchronised thundering herd: the mirror sees its entire day’s connection load inside two seconds, sheds most of it, and every host records a failure.

RandomizedDelaySec= fixes it by delaying each firing by a random amount in [0, value]:

# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly backup

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=1800
Persistent=true

[Install]
WantedBy=timers.target

400 hosts now spread across a 30-minute window. The delay is re-rolled on each firing unless you set FixedRandomDelay=true, which derives it from the machine ID instead - so each host keeps its own stable slot, run after run. Use the fixed form when you want a predictable per-host schedule that is still spread across the fleet, and the random form when you want to avoid a host permanently drawing the worst slot.

Confirm the delay was applied by comparing the timer’s two timestamps:

systemctl list-timers backup.timer

NEXT                        LEFT     LAST                        PASSED  UNIT          ACTIVATES
Wed 2026-08-12 03:11:47 UTC 11h left Tue 2026-08-11 03:19:02 UTC 12h ago backup.timer  backup.service

03:11:47 rather than 03:00:00 is the randomised delay showing up in the schedule.

For cron, there is no jitter directive. Sleep a computed amount inside the job, derived from something stable per host:

#!/bin/bash
set -euo pipefail

# Spread across a 1800-second window, deterministically per host.
WINDOW=1800
OFFSET=$(( $(hostname | cksum | cut -d' ' -f1) % WINDOW ))
sleep "$OFFSET"

/usr/local/sbin/backup.sh

Deriving the offset from the hostname rather than from $RANDOM means a host keeps its slot, so a failure at 03:14 is reproducible at 03:14 tomorrow.

Verify the calendar expression before you ship it

OnCalendar= syntax is easy to get subtly wrong, and the cost of a mistake is a job that silently never runs. systemd-analyze calendar resolves an expression without deploying anything:

systemd-analyze calendar 'Mon *-*-* 03:00:00'

Normalized form: Mon *-*-* 03:00:00
    Next elapse: Mon 2026-08-17 03:00:00 UTC
       From now: 5 days left

Ask for several iterations when the pattern is meant to repeat:

systemd-analyze calendar --iterations=3 'Mon..Fri *-*-* 09,17:00:00'

Normalized form: Mon..Fri *-*-* 09,17:00:00
    Next elapse: Tue 2026-08-11 17:00:00 UTC
       From now: 59min left
   Iteration #2: Wed 2026-08-12 09:00:00 UTC
       From now: 16h left
   Iteration #3: Wed 2026-08-12 17:00:00 UTC
       From now: 24h left

A few iterations is enough to catch the classic errors: a *-*-01 you meant to be monthly but which lands on the first of the month at midnight, or a day-of-week filter that quietly excludes every occurrence.

Note what the normalised form does for you. Mon *-*-* 3:00 normalises to Mon *-*-* 03:00:00 - it is valid, and the seconds field you omitted defaults to zero. Reading the normalised line back is how you confirm systemd understood the expression the way you meant it, not merely that it parsed.

Putting the three together

A production timer-triggered job usually carries all of it:

# /etc/systemd/system/reindex.service
[Unit]
Description=Rebuild search index

StartLimitIntervalSec=7200
StartLimitBurst=3

[Service]
Type=oneshot
ExecStart=/usr/bin/flock -n /var/lock/reindex.lock /usr/local/sbin/reindex.sh
TimeoutStartSec=3600
Restart=on-failure
RestartSec=300

# /etc/systemd/system/reindex.timer
[Unit]
Description=Rebuild search index hourly

[Timer]
OnCalendar=hourly
RandomizedDelaySec=600
Persistent=true

[Install]
WantedBy=timers.target

The lock prevents overlap, TimeoutStartSec= bounds the hang, the burst limit stops a broken job spinning, and the randomised delay keeps the fleet from arriving together. Each of the four exists because of a different failure, and removing any one of them brings that failure back.

One last check before enabling:

sudo systemd-analyze verify /etc/systemd/system/reindex.service /etc/systemd/system/reindex.timer
sudo systemctl daemon-reload
sudo systemctl enable --now reindex.timer
systemctl list-timers reindex.timer

Knowledge check