Locking and concurrency - preventing duplicate jobs

A scheduled job that takes longer than its interval can have two instances running. Locking prevents this. This lesson covers flock and lock-file patterns.

Why locking matters

A backup job scheduled hourly takes 30 minutes. If the host is slow, the next hour’s backup starts before the previous one finishes. Two backups write to the same target: data corruption, doubled disk usage, race conditions.

The fix: serialise with a lock.

flock

flock is the standard tool. It uses file locking via fcntl or flock:

flock /var/lock/my-job.lock -c /usr/local/bin/my-job.sh

The lock is held while my-job.sh runs. A second invocation waits for the lock (or fails, with -n).

For non-blocking:

flock -n /var/lock/my-job.lock -c /usr/local/bin/my-job.sh || {
    echo "Already running"
    exit 0
}

For a timeout:

flock -w 60 /var/lock/my-job.lock -c /usr/local/bin/my-job.sh || {
    echo "Could not acquire lock within 60s"
    exit 1
}

flock in a shell script

LOCKFILE=/var/lock/my-job.lock
exec 200>"$LOCKFILE"
flock -n 200 || { echo "Already running"; exit 0; }

# Critical section - protected by the lock
/usr/local/bin/my-job.sh

exec 200> opens the file as fd 200. flock 200 locks fd 200. The lock is released when the script exits (and fd 200 is closed).

systemd timer with flock

flock is not built into systemd. But systemd already gives you most of what a lock is for: a unit that is still active when its timer elapses is not started again. systemd logs the skip and waits for the next elapse.

my-job.service: Deactivated successfully.
my-job.timer: Not restarting, unit my-job.service is already active.

So for a job that only ever runs from its own timer, you do not need flock at all. You still need it when the same script can also be launched by hand, by cron, or by a second unit — or when the lock must span more than one unit.

When you do wrap the job, wrap it once:

[Unit]
Description=My job

[Service]
Type=oneshot
ExecStart=/usr/bin/flock -n /run/lock/my-job.lock /usr/local/bin/my-job.sh

Two rules make or break this unit:

• Exactly one ExecStart. For Type=simple (the default) a second ExecStart= is a hard error and the unit refuses to load, so the timer fires and nothing runs. For Type=oneshot systemd accepts multiple lines and runs them in sequence — which is worse, because the job executes twice per trigger and the first copy holds no lock at all. That is exactly the doubled-backup corruption this lesson opened with.
• Give the first token as an absolute path. systemd resolves a bare flock against a small compiled-in search path (/usr/local/bin, /usr/bin, and the sbin counterparts) — it does not use your PATH. Anything outside those directories fails to load with “Neither a valid executable name nor an absolute path”, and older systemd (before v239) refused bare names outright. Write /usr/bin/flock and the unit behaves the same everywhere. Confirm the search path with systemd-path search-binaries-default.

Validate before enabling. This is read-only:

systemd-analyze verify /etc/systemd/system/my-job.service

A silent pass means the unit will load. The two-ExecStart version reports:

my-job.service: Service has more than one ExecStart= setting,
  which is only allowed for Type=oneshot services. Refusing.

To force every invocation through the timer, so the hand-run path cannot exist in the first place, add:

[Unit]
RefuseManualStart=yes

The job then starts only when the timer elapses, and systemd’s own “already active” guard is the whole lock.

Lock files without flock

For scripts that cannot use flock (e.g. non-bash scripts):

LOCKFILE=/var/lock/my-job.lock

# Acquire
if [[ -e "$LOCKFILE" ]]; then
    echo "Already running"
    exit 1
fi
trap "rm -f $LOCKFILE" EXIT
echo $$ > "$LOCKFILE"

This is racy — two processes can both see no file and both create it. Use flock from inside the script instead, which needs no wrapper and no external command:

# Preferred. The kernel holds the lock against the open descriptor and
# releases it when the process dies, however it dies.
exec 9>/run/lock/my-job.lock
flock -n 9 || { echo "Already running"; exit 0; }

# ... work ...
# No cleanup needed. Descriptor 9 closes at exit and the lock goes with it.

A mkdir lock is the fallback where flock is genuinely unavailable, and it is not equivalent:

LOCKDIR=/run/lock/my-job

mkdir "$LOCKDIR" 2>/dev/null || { echo "Already running (or a stale lock)"; exit 1; }
trap 'rmdir "$LOCKDIR"' EXIT     # single quotes: expand at trap time, not now

mkdir is atomic on a single filesystem, so two processes cannot both succeed — which is the property the racy version above lacks. Read the callout before choosing it anyway.

Common pitfalls

• Stale locks: only an issue for on-disk locks. A flock is released by the kernel when the holder dies; a lock file or lock directory left by a SIGKILL, an OOM kill or a power loss blocks every subsequent run until someone removes it by hand. Prefer flock; if you cannot, detect staleness with the boot ID as above. A timeout does not help — it bounds how long you wait, not whether the lock has an owner.
• Lock on a network filesystem: NFS locking depends on lockd or NFSv4 state and fails in exactly the partition scenarios you wanted the lock for. Keep the lock on local storage (/run/lock), even when the data it protects is remote.
• Lock cleanup on signal: flock needs none. Manual lock files need an explicit trap, and that trap covers only the signals that can be caught.
• /var/lock vs /run/lock: on modern distributions /var/lock is a symlink to /run/lock, which is a tmpfs — so the lock is cleared at boot, which is what you want. Do not place locks under a persistent path such as /tmp or a home directory, where a stale one survives the reboot that would otherwise have cleaned it up.

Knowledge check