Reading a become failure

Escalation failures are among the most common things anyone asks about Ansible, and they are unusually diagnosable: the error messages are specific, and each one is produced by a different condition on the managed node.

Part XLV covers the general troubleshooting method — verbosity levels, variable inspection, connection debugging. This lesson covers the escalation-specific error shapes, so that when you reach Part XLV you already know which half of the run to investigate.

First: which half failed?

Before reading any message, read the recap.

$ ansible-playbook -i inventories/prod site.yml

PLAY RECAP
app01.example.com : ok=0  changed=0 unreachable=1 failed=0
app02.example.com : ok=3  changed=0 unreachable=0 failed=1

app01 never connected: wrong key, wrong port, host down, host key mismatch. Escalation is not involved and nothing about sudo will help.

app02 connected, ran three tasks, and failed on the fourth. If the error mentions sudo, the credentials worked and the authorisation did not. That is a change on the managed node, not on the controller.

Lesson 1 made this distinction; it is worth repeating because the usual instinct on any become error is to check the SSH key, and the recap has already ruled that out.

The five shapes

Missing sudo password

The rule matched and it is not NOPASSWD. Ansible has no password to supply, and the sudo plugin’s default -n flag makes sudo fail immediately rather than prompt.

The fix is one of: supply the password (-K for an interactive run, a password file or vaulted variable for an unattended one — lesson 3), or change the grant.

What makes this message confusing is when it appears. A fleet that has worked for two years starts producing it on a subset of hosts, and the cause is almost never the controller. Someone changed sudoers on those hosts: a new central policy, a rebuilt host that missed the bootstrap play, a package upgrade that shipped a new /etc/sudoers, or an image that never had the rule.

ansible -i inventories/prod all --become -m ansible.builtin.command -a "id -un" \
--one-line | grep -v 'rc=0.*root'

Incorrect sudo password

A password was supplied and sudo rejected it. Genuinely a wrong secret — or a stale one, which is the same thing arriving from a different direction.

The shape worth recognising: a rotation play changed the automation account’s password on some hosts and failed partway. Subsequent runs now succeed on the rotated hosts and fail on the rest, or the reverse, depending on which password the controller holds. The recap shows a mixture and the fleet is in two states.

Timeout waiting for privilege escalation prompt

The prompt never arrived in a form Ansible recognised. Four causes, in roughly descending order of likelihood:

A shell profile writes to stdout on login. An MOTD, a banner, a Welcome to line, a corporate legal notice, or a .bashrc that echoes something. Anything printed before the prompt can push it out of the window the connection plugin is examining. This is the most common cause and the least suspected, because the banner has been there for years and nobody associates it with Ansible.

A localised sudo. The prompt or the failure message is in a language the plugin’s string list does not contain.

requiretty in sudoers. sudo refuses to run without a terminal, so nothing appears at all. On an older estate this is also what defeats pipelining.

A become_user that does not exist. sudo emits unknown user and exits; depending on the sudo version and locale, that message may not match anything in the failure list either.

ansible -i inventories/prod all -m ansible.builtin.raw -a "echo MARKER" \
| grep -B2 -A2 MARKER

user is not allowed to execute

sudoers has no rule matching this combination. sudo tells you which combination it tried, and every field in the message is diagnostic:

$ ansible -i inventories/prod app02.example.com --become -m ansible.builtin.command -a 'id -un'

app02.example.com | FAILED | rc=1 >>
Sorry, user ansible-dbops is not allowed to execute '/bin/sh -c echo BECOME-SUCCESS-...' as root on app02.example.com.

Read it as four questions:

• ansible-dbops — is this the user you expected? A mismatch here means ansible_user differs from what you assumed, which ansible-inventory --host will confirm.
• /bin/sh -c ... — this is the command sudo evaluated, and it is why command scoping does not work (lesson 4).
• as root — is this the become_user you intended? On a host whose grant permits (appsvc) and not (root), a task that omitted become_user defaults to root and fails exactly here. That is the grant working as designed.
• on app02.example.com — the host field. In a centrally distributed sudoers, a rule can match on one host and not another.

Assessing a partial fleet after an escalation failure

An escalation failure mid-play leaves the fleet in two configurations, and the assessment is separate from the fix.

ansible-playbook -i inventories/prod site.yml --check --diff \
--limit "$(cat failed-hosts.txt | paste -sd, -)"

The order that keeps this honest:

1. Stop. Do not re-run the play to see if it works now. If the sudo rule is still wrong, you will fail on a different subset and widen the split.
2. Enumerate. The recap from the failed run names every host and its counters. Save it; it is the only record of the split.
3. Assess with --check. Read-only, and it tells you what each host still needs.
4. Fix the grant, on a canary first.
5. Converge, with serial so a second failure is contained.

Part XLIV covers judging whether a partial success is a failed change. The escalation-specific point is that the split is by host, not by task, so the fleet is not “half configured” — it is two populations, one of which is complete.

Collecting evidence without leaking the password

Escalation problems get escalated to a vendor or another team, and that is where secrets end up in tickets.

Two facts make this manageable. The become password is written to the escalated process’s standard input, not passed as a command-line argument, so it does not appear in the command Ansible logs at any verbosity. And the connection plugin strips the recognised become prompt from captured output before it is displayed.

What does leak is everything else:

• -vvv prints module arguments. A task handling a credential shows it unless the task carries no_log: true.
• Registered variables appear in debug output and in some callback plugins.
• ANSIBLE_DEBUG=1 produces very large output including internals, and is almost never what you want in a ticket.

ansible -i inventories/prod app02.example.com --become \
-m ansible.builtin.command -a "id -un" -vvv 2>&1 | tail -40

Reproducing with id -un rather than with the failing production task is the whole technique: the escalation either works or it does not, and a command with no arguments cannot leak anything through them.

Knowledge check