Break/Fix: the patch run fails on a third of the fleet with a lock error and succeeds when rerun

Reported symptoms

The monthly patch run has failed on part of the fleet three months running. Each time between 60 and 80 hosts out of 200 report a package manager lock error and the play aborts on those hosts.

Each time, the on-call engineer reran the play against the failed hosts about twenty minutes later and it succeeded completely.

The ticket has been closed twice with the word “transient”.

What was checked and ruled out:

• Not distribution-specific: the Debian and RHEL hosts fail at similar rates.
• Not a bad mirror: the successful hosts in the same run used the same mirrors.
• Not human interference: nobody is logged in at 03:00.
• Not inventory-correlated: the failing hosts share no group, rack, region, role or build date.

The only thing anyone noticed was that the failing set was different each month, and that observation was treated as further evidence that the problem was random.

Evidence provided

$ grep -m2 -A3 'FAILED' logs/patch-2026-08-11.log

fatal: [app041]: FAILED! => {"changed": false, "msg": "Failed to lock apt for exclusive operation: Failed to lock directory /var/lib/apt/lists/: E:Could not get lock /var/lib/apt/lists/lock. It is held by process 2841 (apt-get)"}
fatal: [db017]: FAILED! => {"changed": false, "msg": "Failed to obtain dnf lock, held by process 3120 (dnf)"}

$ ansible app041 -i inventory -b -m ansible.builtin.command -a 'ps -o pid,etimes,cmd -p 2841'

  PID ETIMES CMD
2841    418 /usr/bin/apt-get --quiet --force-confold ... dist-upgrade

$ ansible app041 -i inventory -b -m ansible.builtin.command -a 'systemctl list-timers --all --no-pager'

NEXT                        LEFT     LAST                        UNIT
Wed 2026-08-12 03:22:41 UTC  11h      Tue 2026-08-11 02:57:03 UTC apt-daily.timer
Wed 2026-08-12 06:14:02 UTC  14h      Tue 2026-08-11 06:11:55 UTC apt-daily-upgrade.timer

$ ansible app041 -i inventory -b -m ansible.builtin.command -a 'systemctl show apt-daily.timer -p OnCalendar -p RandomizedDelaySec'

OnCalendar={ OnCalendar=*-*-* 06,18:00:00 }
RandomizedDelaySec=43200

$ grep -n -B2 -A6 'Apply security updates' patch.yml

22:- name: Apply security updates
23:  ansible.builtin.package:
24:    name: '*'
25:    state: latest
26:  register: patch_result

Work the evidence before reading on

The failure names the process holding the lock. That process is not Ansible.

1. Look at the timer’s randomised delay and the size of the window it spreads work across. Roughly what fraction of 200 hosts would you expect to be inside a 15-minute patch run at any given moment?
2. The failing set changes every month and correlates with nothing. What kind of cause produces a victim set that is different every time and correlated with nothing?
3. The rerun succeeds twenty minutes later. What has changed in those twenty minutes, on the host?

Before continuing: the run “succeeded” on the other 130 hosts. Given that a second system is also installing packages on those hosts, what exactly does that success tell you about their package state?

Root cause

1. Two systems are patching the same machines

The distribution ships timers that refresh package metadata and, on the Debian hosts, apply security upgrades without being asked. They are enabled by default on a fresh install, they run as root, and nothing about them appears in the Ansible repository.

The Ansible patch play does the same job on its own schedule. Neither system knows the other exists.

2. The lock decides who loses, and randomisation decides who collides

The package database lock is exclusive by design - two package transactions on one host would corrupt the database, so the lock is correct and necessary.

Ansible waits for it, but not indefinitely. ansible.builtin.apt defaults lock_timeout to 60 seconds and ansible.builtin.dnf defaults to 30. Neither is long enough for a dist-upgrade or a full metadata refresh, which routinely take several minutes.

The distribution timers apply a large randomised delay specifically so that a fleet does not hit its mirrors simultaneously. That randomisation is precisely what produces a different set of collisions every month. The victim set is not noise around a hidden pattern - the randomisation is the pattern, and it was the strongest available clue.

3. The rerun was not a fix, it was a second draw

Twenty minutes later the other transaction has finished, so the lock is free and the run succeeds. Nothing was repaired; the collision simply did not happen the second time.

That is why the incident was closed as transient twice, and it is the most expensive mistake in the whole scenario, because it left the underlying condition in place for three months.

Resolution

1. Repair the hosts that were left partway through a patch cycle before anything else. A failed patch run can leave packages downloaded but not configured; on Debian hosts check dpkg --audit and complete any interrupted configuration deliberately.
2. Decide who owns patching, as a policy decision rather than a technical one. Two systems patching the same host cannot both be right, and tuning the timeout only changes which one usually wins.
3. If Ansible owns patching, mask the distribution timers through the same configuration management, so the decision is written in the repository and survives a rebuild. Masking rather than disabling prevents a dependency from starting them again.
4. If unattended upgrades own security patching by policy, remove package installation from the Ansible play entirely and keep it as a verification and reporting play. A play that only asserts cannot collide.
5. Raise lock_timeout to reflect how long legitimate other work takes - several minutes, not the interactive default - so a brief overlap waits rather than fails.
6. Add a pre-flight check that refuses to start while a package transaction is in progress. Failing before the run is far cheaper than failing partway through it, and it produces a clear message instead of a lock error.
7. Move the Ansible patch window away from the distribution timer window as defence in depth, accepting that a twelve-hour randomised delay means separation alone is not a fix.
8. Reopen and correct the two tickets closed as transient, with the timer evidence attached, so the next intermittent failure is not closed the same way.

Verification

1. The guard fires under a deliberately arranged collision. On one test host, start a long metadata refresh, then run the patch play: the pre-flight check must refuse to start and say why. This is the check that can fail, and running the play on a quiet host proves nothing.
2. The distribution timers are gone where they should be. systemctl list-timers --all on a sample shows no package-related timers, and systemctl is-enabled apt-daily.timer reports masked.
3. The masking survives a converge. Re-run the configuration management and confirm the timers are still masked; a host-level mask that the next converge undoes is not a fix.
4. No host is mid-transaction. dpkg --audit returns nothing on Debian hosts and the equivalent RPM verification is clean, across the hosts that failed in the last three runs.
5. Three consecutive scheduled runs report zero lock failures. One clean run is not evidence about a collision that happens by chance.
6. The patch report accounts for every package change. Compare installed versions before and after a run against what the run reported; unexplained differences mean a second system is still active somewhere.
7. A rebuilt host comes up with the timers masked. Build one from the pipeline and check, because this is the path by which a second patching system silently returns.

Prevention

• One system owns package state on a host, and the choice is written down. This is a correctness requirement, not a preference: two patching systems make the estate undescribable long before they make it fail.
• Mask the distribution timers as managed configuration, not as a one-time host change. Rebuilt hosts reacquire defaults.
• Set lock_timeout deliberately. The module defaults are sensible for a human at a terminal and short for a fleet run competing with a background transaction.
• Guard before you work. A pre-flight task that refuses to start while another transaction is running turns a mid-run failure into a clean refusal with a readable message.
• Never close an intermittent failure as transient without characterising the victim set. Same hosts every time, different hosts correlated with load, and different hosts correlated with nothing are three different diagnoses, and the third one means look at the schedule.
• Make systemctl list-timers --all a standard early step when investigating anything periodic. It is cheap and it enumerates every other system that thinks it owns the machine.