Break/Fix: privilege escalation stopped working overnight and sudo -l says the account is fine · Break/Fix · Ansible

Reported symptoms

The 06:00 patch run fails on 61 of 200 hosts. Every failure is on a task with become: true; every task without it succeeds on the same hosts.

The message is the same everywhere:

fatal: [app017]: FAILED! => {"msg": "Missing sudo password"}

Nobody has ever configured a sudo password for this account, because it has always had NOPASSWD. So the investigation started, reasonably, on the controller:

Is the vault password file readable? Yes.
Did become_password get set somewhere by accident? No.
Did the service account’s SSH key change? No, and ssh as that account works by hand.
Is the account still in the right group? Yes.

Then somebody ran sudo -l -U svc_ansible on a failing host and it printed a NOPASSWD rule. At that point the team concluded the sudoers policy was fine and went back to looking at the controller, where they stayed for three hours.

Evidence provided

Read-only / Safethe connection is healthy

$ ansible app017 -i inventory -m ansible.builtin.ping

app017 | SUCCESS => {
"changed": false,
"ping": "pong"
}

Read-only / Safethe same command with escalation

$ ansible app017 -i inventory -m ansible.builtin.ping -b

app017 | FAILED! => {
"msg": "Missing sudo password"
}

Read-only / Safea real NOPASSWD rule - which is why sudoers was ruled out

$ ansible app017 -i inventory -m ansible.builtin.command -a 'sudo -l -U svc_ansible'

User svc_ansible may run the following commands on app017:
(root) NOPASSWD: /usr/bin/dnf, /usr/bin/systemctl

Read-only / Safea host that still works

$ ansible app093 -i inventory -m ansible.builtin.command -a 'sudo -l -U svc_ansible'

User svc_ansible may run the following commands on app093:
(root) NOPASSWD: ALL

Read-only / Safewhat sudo is actually being asked to authorise

$ ansible-playbook -i inventory patch.yml --limit app017 -vvv 2>&1 | grep -m1 'sudo -H -S -n'

sudo -H -S -n  -u root /bin/sh -c 'echo BECOME-SUCCESS-xkjqwe ; /usr/bin/python3 /var/tmp/ansible-tmp-1786550412.7-1841-193/AnsiballZ_dnf.py'

Read-only / Safethe target has been describing the fault precisely since 06:00

$ ansible app017 -i inventory -m ansible.builtin.command -a 'journalctl -t sudo --since 06:00 -n 3 --no-pager' -b --become-method su

Aug 11 06:04:11 app017 sudo[3312]: svc_ansible : command not allowed ; TTY=unknown ; PWD=/home/svc_ansible ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-xkjqwe ; /usr/bin/python3 /var/tmp/ansible-tmp-.../AnsiballZ_dnf.py

Work the evidence before reading on

The controller is fine. The connection is fine. The account is fine. The sudoers rule is present, syntactically valid and grants NOPASSWD.

Read the -vvv line and the sudo -l output side by side. Write down the command sudo is being asked to run, and then write down the list of commands it is allowed to run. Do they intersect?
AnsiballZ_dnf.py is a file in a temporary directory. Its name changes every run. Could any allow-list entry match it?
The failure began at 06:00 and spread alphabetically. What runs alphabetically across a fleet?

Before continuing: when a playbook uses ansible.builtin.dnf, what binary does the target actually execute?

Root cause

1. Ansible never runs the tool the module is named after

This is the fact the whole incident turns on, and it is not obvious from the outside.

When a task uses ansible.builtin.dnf, Ansible does not construct a dnf command line and run it. It assembles the module’s Python source and its dependencies into a single self-contained payload, transfers it to the target as AnsiballZ_dnf.py in a temporary directory, and executes it with the target’s Python interpreter. The module code then does whatever it does - which for dnf involves the package manager’s Python bindings, not a shell invocation of /usr/bin/dnf at all.

So the command presented to sudo is always of the form:

/bin/sh -c 'echo BECOME-SUCCESS-... ; /usr/bin/python3 /var/tmp/ansible-tmp-<timestamp>-<pid>-<n>/AnsiballZ_<module>.py'

The module name appears only in a filename, inside a directory whose name is different on every run.

2. An allow-list of tools can never match

The hardening baseline replaced NOPASSWD: ALL with NOPASSWD: /usr/bin/dnf, /usr/bin/systemctl. Those are the two tools the playbook appears to use, and if a human were running the patch by hand they would be exactly right.

They cannot match the interpreter invocation. Not with more entries, not with wildcards on the temporary directory - and an allow-list that permits /usr/bin/python3 with an arbitrary script argument grants unrestricted root anyway, which is the opposite of what the baseline was trying to achieve.

When no rule matches, sudo does not fail silently. It falls through to its default behaviour, which is to ask for a password. Ansible has none, so the connection plugin raises Missing sudo password.

3. The message describes the request, not the reason

Missing sudo password is literally true: sudo prompted, and no password was supplied. It says nothing about why sudo prompted, and the natural reading - a credential is missing - is wrong. Every hour spent on the vault, the become password file and the service account was spent chasing that reading.

The target knew. journalctl -t sudo on any failing host had been logging command not allowed since 06:00.

Under the hood

Two Ansible mechanisms are visible in the -vvv line and both are worth recognising on sight.

The echo BECOME-SUCCESS-<random> prefix is how Ansible decides whether escalation worked. It escalates, immediately echoes a per-connection random token, and watches for that token in the output stream. Seeing it means the escalated shell is running; a password prompt or an error instead means it is not. This is why escalation failures are detected before any module output exists, and why they surface as connection errors rather than as module failures.

/var/tmp/ansible-tmp-<epoch>-<pid>-<n>/ is the remote temporary directory. It is created per task, the payload is written into it, and it is removed afterwards. Its uniqueness is deliberate - concurrent runs must not collide - and it is precisely what makes path-based authorisation of the payload impossible.

There is one configuration that removes the temporary file entirely: pipelining = True streams the payload to the interpreter over the existing SSH session. It does not change the escalation shape, though. The command sudo sees is still the interpreter. Pipelining is also disabled by default because it requires requiretty to be off in the targets’ sudoers, which is its own become-related failure.

sudo -l renders the policy; it does not tell you whether your command matches

sudo -l -U svc_ansible answers “what is this user allowed to run”. It printed a NOPASSWD rule, and the team read that as “escalation works”.

Those are different claims. The rendered policy is correct and the command being run is not in it. Any check of a command allow-list has to name the command:

sudo -l -U svc_ansible /usr/bin/python3

That form asks whether a specific command is permitted and answers with sudo: a password is required when it is not. It would have ended this incident in ninety seconds.

The general shape recurs everywhere in authorisation debugging: listing a policy is not the same as evaluating it against the request you actually make.

Resolution

Establish the split precisely. Run ansible all -i inventory -m ansible.builtin.ping -b across the fleet and record which hosts fail. That list is the blast radius of the baseline rollout and you will need it to confirm the repair.
Get root on one failing host by a path that does not depend on the broken rule - an existing session, the console, out-of-band management, or the configuration-management agent, which runs as root and does not consult sudoers.
Confirm the diagnosis there with sudo -l -U svc_ansible /usr/bin/python3, which must report that a password is required. This is the check that distinguishes this fault from a genuine credential problem.
Deploy the corrected rule through the configuration-management system that deployed the broken one. A hand-edit on 61 hosts is undone by the next converge and hides the fact that the source is still wrong.
Restore escalation for the automation account. If the security requirement behind the baseline is real, change the shape of the control rather than its contents: unrestricted sudo for the account, combined with SSH access restricted by source address and key, and auditing from the run log and the target audit daemon.
Canary to one host and verify with ansible <host> -m ansible.builtin.ping -b before touching the other 60.
Roll out, then re-verify a random sample across roles and regions rather than assuming convergence implies correctness.
Take the finding to the security team with the -vvv line as evidence. The baseline is wrong about how Ansible works, and it will be applied to the next fleet unless the source is corrected.

Verification

Escalation works on a repaired host. ansible <host> -i inventory -m ansible.builtin.ping -b returns SUCCESS. This is the smallest command that exercises the whole become path.
The check can fail. Run the same command against a host that has not yet been repaired and confirm it still reports Missing sudo password. A verification that passes on every host, including the broken ones, is not a verification.
The specific command is authorised. sudo -l -U svc_ansible /usr/bin/python3 on a repaired host reports the command as permitted rather than demanding a password.
The target agrees. journalctl -t sudo --since -10min on a repaired host shows the python3 invocation being authorised, with no command not allowed entries.
A real run completes. ansible-playbook -i inventory patch.yml --check --limit <host> finishes with no become failures; a ping -b exercises escalation once, a playbook exercises it per task.
Convergence does not undo it. Re-run the configuration-management agent and repeat the first two checks. This is where a hand-fix that skipped the source is exposed.
The full fleet is clean. ansible all -i inventory -m ansible.builtin.ping -b reports zero failures, and the count of hosts matches the count from the original survey.

Prevention

Record in the repository that Ansible escalates the Python interpreter running a generated payload, never the tool a module is named after. This single sentence prevents the entire class of command-allow-list proposals.
Test authorisation changes by exercising them. sudo -l renders a policy; ansible ... -m ping -b evaluates it. Only the second one can catch a rule that does not match.
Canary sudoers changes to one host, with an automated escalation check, before any fleet rollout. Authorisation changes deserve the same care as firewall changes because both can remove the means of repair.
Keep the automation account’s escalation broad and its reachability narrow. Constrain who may become the account and from where; audit what it did from the controller run log and the target audit daemon, which record actions rather than permissions.
Alert on become failures as their own category. A fleet-wide escalation failure is simultaneously an outage and a signal that authorisation changed without coordination.
When a fault spreads alphabetically or by batch, look for a configuration-management rollout before looking anywhere else. That ordering is not a property of any fault; it is a property of a converge.

What if the sudoers change had been a syntax error instead of a narrowed rule?

Considerably louder, considerably faster to diagnose, and total rather than partial.

A parse error anywhere in /etc/sudoers or an included drop-in makes sudo refuse the entire policy on that host. Every user loses sudo, including any human trying to fix it, and the error names the file and line. Ansible would report a become failure too, but a human logging in would hit the same wall immediately and the cause would be on screen.

The narrowed rule is worse precisely because it is valid. Everything parses, sudo -l prints something reassuring, humans running commands by hand are unaffected, and only the automation - which runs a command nobody thought about - is refused.

There is a third variant that is quieter still and is a security finding rather than an outage. If the baseline had granted NOPASSWD: /usr/bin/python3 in an attempt to make Ansible work, every task would have succeeded and the account would have had unrestricted root by way of an arbitrary script argument, while the policy file appeared to be an allow-list. A rule that makes the automation work is not evidence that the rule is a restriction.

Why this matters

The security baseline was written by people who understood sudo and did not know how Ansible executes modules. The automation team understood Ansible and did not read the target’s sudo log. Between the two, three hours went into a controller that was never broken.

The specific fact - that escalation applies to the interpreter and a generated payload, not to dnf or systemctl - is worth carrying because it is invisible from the playbook. Nothing in ansible.builtin.dnf suggests that the command sudo evaluates is /usr/bin/python3. Once you know it, the whole family of command-allow-list proposals resolves in one sentence, and a class of authorisation incidents stops happening.

The transferable habit is smaller and more general: when a permission check fails, find out what was actually requested before arguing about what is allowed.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Ansible never runs the tool the module is named after

2. An allow-list of tools can never match

3. The message describes the request, not the reason

Resolution

Verification

Prevention