Fighting the automatic updater for the package lock · Ansible

Of the thirty hosts that failed in the signature scenario, the single most likely cause is not a bug in the play, a drifted config or a missing package. It is that another automated system was patching the same host at the same time.

The error is unmistakable once you have seen it once:

Read-only / Safewhat lock contention looks like in a run log— Two hosts, two package managers, same underlying cause. Extracted from the log of a completed run; nothing is executed here.

$ grep -A 3 'fatal:' /var/log/ansible/patch-2026-08-11.log | head -12

fatal: [app-047]: FAILED! => {"changed": false, "msg": "Failed to lock apt for
exclusive operation: Failed to lock directory /var/lib/apt/lists/: E: Could not
get lock /var/lib/apt/lists/lock. It is held by process 2841 (apt-get)"}

fatal: [app-112]: FAILED! => {"changed": false, "msg": "Failed to obtain the DNF
lock. Another app is currently holding the yum lock; waiting for it to exit...
The other application is: PackageKit"}

Illustrative output

The signature: it clusters, it does not scatter

This is the diagnostic that separates lock contention from everything else, and it is the reason the lesson exists.

Random failures scatter. If thirty hosts fail because of a bad package in a repository, they will be spread across your serial batches in proportion to how many of them run that distribution. If thirty hosts fail because of hardware or a flaky network, they scatter too.

Lock contention clusters by time. The automatic updater is driven by a timer — apt-daily-upgrade.timer on Debian family, a dnf-automatic timer or PackageKit refresh on RHEL family. The timer fires within a randomised window, and the hosts that collide with your run are the hosts whose window overlapped the batch you were on at that moment.

Batching makes this worse in a way that is worth stating plainly. A serial: 20 run over 300 hosts takes as long as it takes, and each batch occupies a different slice of the clock. The batch that happens to run at 02:31, when the timer window opens, fails. The batches on either side do not. From the recap it looks like a mysterious cluster of identical hosts behaving differently; from the timer it is obvious.

The `lock_timeout` options, and what each is actually worth

Both package modules can wait for the lock. Verified with ansible-doc against ansible-core 2.21.3:

Read-only / Safethe documented lock_timeout defaults — read from ansible-doc on 2.21.3— ansible-doc reads the module documentation shipped with the installed ansible-core. It executes nothing against any host.

$ ansible-doc ansible.builtin.apt | grep -A 6 lock_timeout ; ansible-doc ansible.builtin.dnf5 | grep -A 4 lock_timeout

   lock_timeout  How many seconds will this action wait to acquire a
               lock on the apt db.
               Sometimes there is a transitory lock and this will
               retry at least until timeout is hit.
      default: 60
      type: int

 lock_timeout  This is currently a no-op as dnf5 does not provide an
               option to configure it.
               Amount of time to wait for the dnf lockfile to be
               freed.
      default: 30
      type: int

Module	Default	Reality
`ansible.builtin.apt`	`60` seconds	Waits, and retries transitory locks
`ansible.builtin.dnf`	`30` seconds	Waits for the dnf lockfile
`ansible.builtin.dnf5`	`30` seconds	Documented as a no-op — dnf5 provides no way to configure it

Raising the timeout is not the fix

The obvious move is lock_timeout: 600. It will reduce your failure count, and it is still the wrong answer, for four reasons.

It makes the run slower and less predictable. Each contended host now blocks for up to ten minutes. With serial: 20, one contended host delays its entire batch, and a maintenance window with a hard end time becomes a window you cannot size.

It does not remove the second writer. Waiting means you patch after the automatic updater. The updater has now installed a set of packages you did not choose, at a version you did not pin, possibly restarting services, and your play then runs on top of that. The run reports success. What was installed on that host is not what your play specified.

It converts a loud failure into a quiet inconsistency. A locked host that fails is on your failed list. A locked host that waited and then succeeded looks identical to a host that had no contention at all — and the two ended up with different package sets.

It does not help at all on dnf5. See above.

Two automated systems patching one host is a design error

The real problem is not the lock. The lock is doing its job: it is preventing two package transactions from corrupting each other, and without it you would be reading a much worse lesson about a broken RPM database.

The problem is that two independent systems both believe they own package state on that host. One of them was configured when the host was built and nobody has thought about it since; the other is your change management process. They will disagree about versions, about timing, and about whether a service may be restarted.

Pick one. If Ansible owns patching, the automatic updater must be disabled and that must be verifiable. If the automatic updater owns security patching, your play must not upgrade packages, and your change process must account for a fleet whose package versions move without your involvement.

There is no third option where both run and everything is fine. There is only a third option where both run and you have not yet noticed.

The precondition, written as a gate

If Ansible owns patching, the automatic updater being off is a documented precondition of the change, in the same category as “the host is in the load balancer” or “there is a current backup”. Preconditions are asserted, not hoped for.

Read-only / Safeassert the precondition before touching any package— Asks systemd whether any automatic-update unit is enabled, and refuses to proceed where one is. systemctl is-enabled reads unit state and writes nothing, so changed_when is false; failed_when is false because is-enabled exits non-zero for a masked or absent unit, which is the outcome we want. Run this as a pre-flight play before the patch play.

- name: Patch preconditions
hosts: patch_targets
gather_facts: true
tasks:
  - name: Ask systemd about every automatic-update unit
    ansible.builtin.command:
      argv:
        - systemctl
        - is-enabled
        - '{{ item }}'
    register: updater_state
    changed_when: false
    failed_when: false
    loop:
      - apt-daily.timer
      - apt-daily-upgrade.timer
      - dnf-automatic.timer
      - dnf-automatic-install.timer
      - packagekit.service

  - name: Refuse to patch a host that has a second package-state owner
    ansible.builtin.assert:
      that:
        - updater_state.results
          | selectattr('stdout', 'eq', 'enabled')
          | list | length == 0
      fail_msg: >-
        An automatic updater is enabled here. Patching would race a
        second package transaction. Resolve before the window opens.
      success_msg: 'This host has a single owner for package state.'

The permanent fix belongs in the host build, not in the patch play: the automatic updater is masked at provisioning time, in the same role that configures everything else, and the assertion above exists to catch the hosts where that did not happen.

Configuration changemasking the updater in the base role, where it belongs— Masking rather than merely disabling, because a package upgrade can re-enable a disabled unit. This is a configuration change on the managed node and is deliberately part of the build role, not the patch play.

- name: Ansible owns package state on this fleet
ansible.builtin.systemd_service:
  name: '{{ item }}'
  enabled: false
  state: stopped
  masked: true
loop:
  - apt-daily.timer
  - apt-daily-upgrade.timer
when: ansible_facts['os_family'] == 'Debian'

Masking rather than disabling is deliberate: a disabled unit can be re-enabled by a package upgrade shipping a new preset, which is exactly the kind of quiet reversal that produces a cluster of failures six months later with no change to blame.

Under the hood

Why is this a lock at all, and why is waiting for it not free?

dpkg takes an exclusive flock on /var/lib/dpkg/lock-frontend for the duration of a transaction, and apt takes further locks on the lists and archives directories. dnf uses a lockfile under /var/lib/dnf/ or /var/cache/dnf/. In every case the lock protects a database that describes what is installed; two concurrent writers would produce a package database that does not match the filesystem.

The Ansible modules do not hold the lock themselves. They shell out to the package manager, which takes the lock, and lock_timeout controls how long the module keeps retrying while the manager reports the lock busy. That indirection is why the option cannot do anything on dnf5: the retry needs a supported way to ask the underlying tool to wait, and dnf5 does not expose one.

It also explains the second-order failure. If the automatic updater is killed mid-transaction — by a timeout, a reboot, or an operator resolving the incident with pkill — the lock file may be released while the database is inconsistent. The next run then fails with something that reads like a completely unrelated problem (dpkg was interrupted, you must manually run dpkg --configure -a), on a host that appears healthy. Reaching for the timeout knob makes this outcome more likely, not less, because a longer wait means your run is more likely to be the thing that is still running when the window closes.

What happens if the fleet is mixed and I cannot disable the updater everywhere?

Then say so explicitly, and design the run around it rather than pretending it is not true.

Three things make a mixed estate survivable:

Split the inventory by ownership. A group where Ansible owns package state and a group where the updater does. They get different plays. The group where the updater owns patching does not get an apt: upgrade task, because running one there is the collision, not the fix.

Schedule outside the window. The timers are visible. Read systemctl list-timers across the fleet, find the union of windows, and run outside it. This is fragile — a distribution upgrade will reset the timer to its default — so it is a mitigation, not a solution, and it needs to be re-verified rather than assumed.

Make the collision visible when it happens. Set lock_timeout to something short rather than long, and treat the resulting failures as a report about which hosts still have two owners. A short timeout that fails loudly gives you a list. A long timeout that succeeds quietly gives you drift.

Knowledge check

Why this matters

This failure is worth a lesson of its own because it is simultaneously the most common cause of a mixed-result patch run and the one most often fixed at the wrong layer.

The instinct is to treat it as a timing problem: the lock was busy, wait longer. That reading produces a change that appears to work, removes the error message, and leaves the actual condition in place — two systems writing to the same package database, disagreeing about versions, restarting services on their own schedules. The fleet drifts, quietly, and the next investigation into why two supposedly identical hosts run different package versions starts from scratch because the evidence was suppressed six months earlier.

The question the failure is really asking is a governance question: who owns package state on this host? It has one correct answer per host, it should be written down, and the play should refuse to run where the answer is unclear.

The signature: it clusters, it does not scatter

The lock_timeout options, and what each is actually worth

Raising the timeout is not the fix

The precondition, written as a gate

Knowledge check

The `lock_timeout` options, and what each is actually worth