Patching Debian and Ubuntu · Ansible

ansible.builtin.apt is a wrapper around a package manager that was designed for an interactive administrator, and the module’s job is to answer, in advance and without asking, every question apt would have put on the terminal.

Most of those answers are defaults you did not choose. One of them decides what happens to a configuration file you edited by hand. This lesson is about which defaults are load-bearing.

Everything below was read from ansible-doc ansible.builtin.apt in a virtualenv pinned to ansible-core 2.21.3, not from memory.

Read-only / Safereading the contract rather than assuming it— Read-only. ansible-doc reports the parameters and defaults of the module version you actually have installed, which is the only version whose behaviour matters.

$ ansible-doc ansible.builtin.apt

   dpkg_options  Add `dpkg' options to `apt' command. Defaults to `-o
               "Dpkg::Options::=--force-confdef" -o
               "Dpkg::Options::=--force-confold"'.
      default: force-confdef,force-confold

 lock_timeout  How many seconds will this action wait to acquire a
               lock on the apt db.
      default: 60

 upgrade  If yes or safe, performs an aptitude safe-upgrade.
          If full, performs an aptitude full-upgrade.
          If dist, performs an apt-get dist-upgrade.
      choices: [dist, full, 'no', safe, 'yes']
      default: 'no'

 state   choices: [absent, build-dep, latest, present, fixed]
      default: present

The cache, and why refreshing it on every task is wrong

apt will not install a version it does not know exists, so the package lists have to be current. update_cache: true runs the equivalent of apt-get update.

The default is not to update the cache at all — the module leaves update_cache unset rather than defaulting it to false, which amounts to the same thing for a play but is worth knowing when you read the documentation.

The naive fix is to set update_cache: true on every apt task. On a 300-host fleet with six package tasks, that is 1800 metadata refreshes against your mirror inside one run, and a mirror that starts returning 503 halfway through.

Configuration changerefresh once, then reuse— Updates the package lists on the managed node. Downloads metadata and rewrites /var/lib/apt/lists; it installs nothing. cache_valid_time makes the refresh a no-op if the lists are already fresh.

- name: Refresh the package lists if they are older than an hour
ansible.builtin.apt:
  update_cache: true
  cache_valid_time: 3600

cache_valid_time is in seconds and defaults to 0, which means “always refresh”. Setting it makes the task idempotent in a useful sense: a rerun within the hour does nothing and reports ok.

The documentation also records a convenience that is easy to trip over: since Ansible 2.4, setting cache_valid_time explicitly implies update_cache: yes. So this is a cache refresh even though it does not say so:

- name: This refreshes the cache, despite appearances
  ansible.builtin.apt:
    name: nginx
    state: present
    cache_valid_time: 3600

`state: latest` versus `upgrade`

These are different operations and the difference is what gets touched.

state: latest with a name upgrades the named packages and their dependencies. Nothing else on the host moves.

upgrade ignores name entirely and upgrades everything the system has an update for. The documentation is explicit: “This does not upgrade a specific package, use state=latest for that.”

For a patch run you almost always want upgrade, because the point of patching is the set of packages you did not enumerate.

What each upgrade mode is permitted to remove

This is the distinction that matters, and the parameter names do not convey it.

Value	Underlying operation	May remove installed packages?
`safe` (and its alias `yes`)	`aptitude safe-upgrade`	No
`full`	`aptitude full-upgrade`	Yes
`dist`	`apt-get dist-upgrade`	Yes
`no` (default)	nothing	—

safe-upgrade upgrades packages only where doing so requires no removals. If a package can only be upgraded by removing something else, safe leaves it at the old version — which is conservative, and which also means a safe upgrade can silently leave a security update unapplied.

full-upgrade and dist-upgrade will resolve those cases by removing whatever is in the way. On a well-maintained host that is usually an obsolete transitional package. On a host that someone has been hand- editing for four years it can be your monitoring agent.

Service impact possiblea defensible fleet upgrade task— Upgrades every package with an available update and restarts whatever the maintainer scripts restart. SERVICE-IMPACT is the honest badge: this can restart sshd, the database and the application in one task.

- name: Apply all available updates
ansible.builtin.apt:
  upgrade: safe
  autoremove: false
  fail_on_autoremove: true
  lock_timeout: 300
register: apt_upgrade

- name: Record what changed, for the audit trail
ansible.builtin.debug:
  var: apt_upgrade.stdout_lines
when: apt_upgrade.changed

autoremove: false is explicit rather than redundant. The default is already false, and writing it down states that removing orphaned dependencies is a separate, deliberate maintenance action rather than something a patch run does on the way past.

`only_upgrade`: patch what is there, install nothing new

only_upgrade: true upgrades a named package only if it is already installed, and does nothing if it is not.

Configuration changethe same task against a heterogeneous fleet— Upgrades the listed packages where they exist. On a host that does not run nginx, this installs nothing rather than adding a web server to a database host.

- name: Upgrade the packages we own, where they are installed
ansible.builtin.apt:
  name:
    - nginx
    - openssl
    - openssh-server
  state: latest
  only_upgrade: true

Without only_upgrade, state: latest on a package that is absent installs it. Run that task against a group that turned out to include the database tier and you have just installed nginx on your databases — a change that will pass every health check and confuse somebody in four months.

The parameter that decides what happens to your edited config files

dpkg_options defaults to force-confdef,force-confold, and that default answers the most consequential question a Debian upgrade asks.

When a package upgrade ships a new version of a configuration file that you have modified locally, dpkg normally stops and asks. In a non-interactive run it cannot ask, so the answer is pre-selected:

force-confold — keep the version currently on disk. Your modified file survives; the maintainer’s new file is written alongside it as .dpkg-dist and ignored.
force-confdef — where the package provides a default action, take it, and only fall back on confold when there is no default.

Together they mean: your local edits win, silently, every time.

The default is the safe one for uptime and the dangerous one for security

Keeping the local file is right for uptime — an upgrade that overwrote your tuned postgresql.conf at 02:00 would be far worse.

It is wrong when the reason for the update is the configuration file. A security update that changes the default cipher list in /etc/ssh/sshd_config, or that disables a protocol version, does nothing on any host where somebody once edited that file. The upgrade reports success. The vulnerability remains. The patch report says the package version is current, and it is — the configuration is not.

Finding these is a separate job from patching:

find /etc -name '*.dpkg-dist' -o -name '*.dpkg-new' -o -name '*.ucf-dist'

Every hit is a file where the maintainer wanted to change something and your policy said no. Review them on a schedule. Empty output is the answer you want.

The deeper fix is the one the configuration-management parts of this course argue for throughout: if a file matters, Ansible should own it as a template, and the .dpkg-dist question stops being interesting because the desired content is asserted on every run rather than inherited from whatever the package left behind.

The lock, and the neighbour that holds it

On Ubuntu and Debian, unattended-upgrades is enabled by default on many installations, and it does exactly what you are trying to do, on its own timer, holding the same dpkg lock.

Read-only / Safewhat the failure looks like— Read-only reproduction of the error text. The diagnosis people reach for — the network, the mirror, the module — is wrong; the lock is held by a process on the same host.

$ ansible-playbook -i inventories/production patch.yml --limit webservers

fatal: [web07.example.com]: FAILED! => {
  "changed": false,
  "msg": "Failed to lock apt for exclusive operation: Failed to lock directory /var/lib/dpkg/: E:Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2841 (unattended-upgr)"
}

The obvious diagnosis is that something is broken. Nothing is broken. Another patching mechanism got there first, and on a fleet it will hit a different subset of hosts every month, which is why the failure looks random.

Three responses, in increasing order of correctness:

Wait longer. lock_timeout defaults to 60 seconds. An unattended-upgrades run that is downloading a 200 MB kernel will hold the lock for considerably longer than that. Raising it to 300 turns most of these into a slow success.

Wait for the lock explicitly, so the run log says what it waited for:

Read-only / Safechecking for the neighbour before starting— Read-only. Fails the host early with a message that names the actual cause, rather than failing later inside a package task with an error people misdiagnose.

- name: Check whether unattended-upgrades holds the apt lock
ansible.builtin.command:
  argv:
    - fuser
    - /var/lib/dpkg/lock-frontend
register: dpkg_lock
changed_when: false
failed_when: false
retries: 30
delay: 10
until: dpkg_lock.rc != 0

- name: Refuse to patch a host whose lock is still held
ansible.builtin.fail:
  msg: >-
    {{ inventory_hostname }} still has the dpkg lock held after five
    minutes. Another package operation is in progress; this host is
    excluded from the run rather than fought with.
when: dpkg_lock.rc == 0

Stop competing. The correct long-term answer is that one thing patches a host. If Ansible does it, unattended-upgrades should be configured for security-only or disabled, and that decision should be recorded in the same role that does the patching.

Under the hood

apt is one of the modules that can respawn itself under a different Python interpreter, and knowing this saves an evening of confused debugging.

The module needs python3-apt — the Python bindings to libapt — because it talks to the APT cache through the library rather than parsing command output. If the interpreter Ansible selected cannot import python3-apt, the module checks the system-owned interpreters for it as well. If it is found there, the module re-executes itself under that interpreter.

If it is not found anywhere, the behaviour depends on auto_install_module_deps, which defaults to true: the module will attempt to install the dependency itself.

Two consequences worth carrying:

A patch task can install a package before it patches anything. On a minimal host with no python3-apt, the first apt task performs an install you did not write. That is usually welcome and occasionally a surprise in a change record that is supposed to enumerate every package touched. Setting auto_install_module_deps: false makes the host fail loudly instead.

Interpreter discovery and module dependencies are separate problems. A host where ansible_python_interpreter points at a virtualenv without python3-apt will respawn under /usr/bin/python3 for apt tasks and stay in the virtualenv for everything else. When one module behaves as though it is running under a different Python from the rest of the play, this is why.

From ansible-core 2.21 there is one more autonomous behaviour: if the apt lists directory is absent entirely, the module recreates it by running apt-get update, whether or not you asked for a cache refresh.

What happens if you pin an exact version with apt and the repository no longer has it?

The task fails, which is correct, and the reason it fails is worth understanding because it explains a class of confusing pinning bugs.

Writing name: nginx=1.24.0-2ubuntu1 asks for an exact version. The documentation records that an exact version request carries an implicit APT pin priority of 1001, which is high enough to permit a downgrade — that is deliberate, and it is what makes exact-version pinning usable as a rollback tool.

But repositories garbage-collect. Distribution security repositories in particular keep only the current version of each package, so the exact version you pinned last month may simply not exist today. The failure is No package matching 'nginx=1.24.0-2ubuntu1' is available, and it will appear on every host at once, months after the pin was written and long after anyone remembers writing it.

This is the case for a local mirror or an artifact repository that you control the retention policy of. Pinning to a version somebody else can delete is a dependency on their retention policy, and it will be discovered during an incident rather than during a review.

For comparison, default_release — the -t option — carries an implicit priority of 990, which prefers the named release without the downgrade permission an exact version grants.

Knowledge check

Why this matters

The apt module makes patching look like a one-line task, and for most runs it is. The lesson is not that the module is hard to use — it is that a one-line task carries a set of pre-selected answers to questions that would have been asked out loud on a terminal.

Three of those answers change what your fleet looks like afterwards. dpkg_options decides whether a security update’s configuration change lands or is quietly discarded. The upgrade mode decides whether the run is permitted to uninstall things. lock_timeout decides whether a neighbour holding the lock is a slow success or a red host at 02:00.

None of them are wrong defaults. They are defaults chosen for the average case, and the value of reading ansible-doc before writing the task is that you find out which ones your case is not average in.