Diagnosing privilege escalation failures · Ansible

Escalation failures are the second half of the transport story, and they respond to the same discipline: reproduce it outside Ansible first.

The connection succeeded — you know that, because a task that did not need become would have worked. What failed is the step after it. That narrows the problem to one command, run over a connection you have already proved.

The one command that splits the problem

Read-only / Safeask sudo directly, over the connection Ansible uses— The -n flag means non-interactive: sudo refuses to prompt and fails immediately if a password would be required, which is exactly the condition Ansible is in. The remote command is true, so nothing changes on the host.

ssh ops@app-047.example.com 'sudo -n true' ; echo "exit=$?"

# and to see what the sudoers policy actually grants this user
ssh ops@app-047.example.com 'sudo -l -n'

Read the result before touching anything

Result of `sudo -n true`	What it means	Where the fault is
Exits `0`, no output	Escalation works non-interactively	Not sudo. Look at `become_user`, the module, or the temp file case below
`sudo: a password is required`	The rule exists but is not `NOPASSWD`	Supply `become_password`, or change the rule
`user is not allowed to execute`	No matching sudoers rule at all	The grant is missing, or is scoped to commands that do not include this one
`sudo: no tty present`	`requiretty` is set in sudoers	The sudoers policy, not Ansible
Hangs	A prompt is waiting on a terminal that does not exist	Same as above; `-n` should prevent it, so this points at a wrapper

Each row sends you somewhere different, and each is one command away. Reaching for become_flags, become_exe or pipelining before running this is guesswork on a system where you can simply ask.

The defaults are worth knowing while reading that table. Verified from ansible-config list on 2.21.3: become_method defaults to sudo, become_user to root, and become_ask_pass to false — so by default Ansible escalates to root via sudo and will not prompt, which is why a rule without NOPASSWD fails rather than asking.

The four signatures

`Missing sudo password`

The remote user has a sudoers rule that requires a password, and no password was supplied. Ansible cannot prompt in an automated run.

Two fixes, and the choice is a policy decision rather than a technical one. Either the automation account gets NOPASSWD for the commands it needs — narrow, auditable, documented in the sudoers file — or the password is supplied from a vault at run time, which means the password exists in the automation path and must be protected everywhere it goes.

Most estates choose NOPASSWD scoped tightly, because a password that automation knows is not much of a second factor.

`Incorrect sudo password`

A password was supplied and rejected. Worth distinguishing from the previous case because the diagnosis is different: this one means your vault content or your lookup is wrong, not that your sudoers policy is.

Check it without printing it. sudo -n true returning “a password is required” proves the policy; whether the password you hold is correct is a separate question and one you resolve by rotating rather than by echoing.

`user is not allowed to execute`

No rule matched. The most common cause on a mixed fleet is that the rule exists in /etc/sudoers.d/ on the hosts built by one process and not on the hosts built by another — which makes this a fleet-consistency finding as much as a permissions one.

The second most common: the rule is scoped to specific commands, and Ansible does not run the command you think. It runs a shell invoking a Python interpreter on a temp file, so a sudoers rule granting /usr/bin/systemctl does not grant what a systemd_service task actually executes.

Command-scoped sudo rules and Ansible do not fit well

A sudoers rule listing individual binaries is good security practice and a poor match for Ansible’s execution model, because every module runs as python3 /path/to/AnsiballZ_<module>.py under a shell.

Estates resolve this in one of three ways: grant the automation account NOPASSWD: ALL and control blast radius through who can run the controller (common, and honest about what is being granted); grant NOPASSWD on the interpreter path (narrower on paper, equivalent in practice, since running arbitrary Python as root is running arbitrary code as root); or avoid become for the tasks that do not need it and scope the grant to a genuinely small set of command tasks.

The least-privilege part of this course develops the trade-off. The point here is diagnostic: if a rule looks correct and escalation still fails, check what the rule is scoped to before checking anything else.

`no tty present` / `requiretty`

sudo is configured to require a terminal. Ansible does not allocate one by default, and pipelining makes it less likely to have one.

This is a sudoers setting on the managed node, and the fix belongs there — Defaults:<user> !requiretty scoped to the automation account rather than removed globally. Working around it from the Ansible side by forcing a pty is possible and hides a policy the host’s owner set deliberately.

Becoming an unprivileged user

The awkward case, and the one where Ansible’s own mechanics are the problem rather than the sudoers policy.

When you connect as ops, become: true and become_user: appuser, the module payload is written to a temp directory owned by ops, and then a process running as appuser has to read it. Neither user is root, so no ordinary permission arrangement makes that work without either widening the file or using POSIX ACLs.

The failure message names the situation directly, and the documented options are:

Approach	What it costs
POSIX ACLs on the temp directory (`setfacl` available on the target)	Requires `acl` installed and a filesystem mounted with ACL support
`world_readable_temp`	The module payload is world-readable while it exists
`pipelining`	Avoids writing the payload at all, but requires `requiretty` off and does not work for every module
Connect as the target user directly	Cleanest where the credential model allows it

Read-only / Safewhat world_readable_temp actually says — read from ansible-doc on 2.21.3— The shell plugin option, read from the installed ansible-core documentation. Nothing is executed against a host.

$ ansible-doc -t shell sh | grep -A 14 world_readable_temp

   world_readable_temp  This makes the temporary files created on the
                      machine world-readable and will issue a
                      warning instead of failing the task.
                      It is useful when becoming an unprivileged
                      user.
      set_via:
        env:
        - name: ANSIBLE_SHELL_ALLOW_WORLD_READABLE_TEMP
        ini:
        - key: allow_world_readable_tmpfiles
          section: defaults
        vars:
        - name: ansible_shell_allow_world_readable_temp
      default: false
      type: boolean

World-readable temp files and no_log do not compose

The option makes the module payload world-readable on the managed node. The payload contains the module’s arguments.

So a task that passes a credential to a module — even one carrying no_log: true, which controls what is printed and not what is written to the target — puts that credential in a world-readable file on the managed node for the duration of the task. Any local user can read it during that window.

That is an acceptable trade only when the tasks running under this become configuration handle no secrets at all, and it is a decision to make per group rather than globally. Setting allow_world_readable_tmpfiles in ansible.cfg applies it to every host in the estate, including the ones where the tasks do carry credentials.

Prefer ACLs where they are available, or connect as the target user.

`ANSIBLE_KEEP_REMOTE_FILES`

The debugging tool of last resort: keep the module payload on the managed node so you can run it by hand.

Read-only / Safewhat the setting is and what it disables — read from ansible-config on 2.21.3— ansible-config list prints the shipped default and documentation. The second description line is the part most people do not expect.

$ ansible-config list | grep -A 6 '^DEFAULT_KEEP_REMOTE_FILES'

DEFAULT_KEEP_REMOTE_FILES:
default: false
description:
- Enables/disables the cleaning up of the temporary files Ansible used to execute
  the tasks on the remote.
- If this option is enabled it will disable ``ANSIBLE_PIPELINING``.

Two things follow. First, it changes execution behaviour: with pipelining disabled, the payload is written to disk on every task, which is a different code path from the one you were debugging. Second, and more seriously:

Read-only / Safeusing it, and cleaning up after it— Scoped to one host by --limit and to one invocation by the environment variable prefix. The final step removes the preserved payloads from the target, which is a change on the managed node and is deliberately explicit.

# one host, one run, one task
ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -i inventory/prod site.yml \
--limit app-047.example.com --tags failing_task -vvv

# then, on the host, run the preserved payload by hand to see the raw error
ssh ops@app-047.example.com \
'ls -d ~/.ansible/tmp/ansible-tmp-*/ | tail -1'

# and clean up when finished — this deletes files, so it is deliberate
ssh ops@app-047.example.com 'rm -rf ~/.ansible/tmp/ansible-tmp-*'

Under the hood

Why does escalation happen inside the SSH session rather than as part of it?

Ansible connects as the remote user and then wraps the remote command in the become plugin’s invocation — for sudo, something structurally like sudo -H -S -n -u root /bin/sh -c 'python3 .../AnsiballZ_x.py'. The connection and the escalation are two distinct steps over one channel.

That separation is why the diagnostic split in this lesson works at all: you can prove the connection independently, and then prove the escalation independently, using the same two commands a shell user would.

It also explains two behaviours that look strange otherwise. The password, when one is needed, is written to sudo’s stdin — the -S flag — which is why it must be available to the controller and why it appears in the process’s input rather than on a command line. And the temp file problem exists because the payload is written by the connecting user before the escalated command runs: the file has to exist before sudo is invoked, so it is created with the connecting user’s ownership and then read by a process that may be neither that user nor root.

Pipelining removes the intermediate file by feeding the module to the remote interpreter’s stdin, which is why it sidesteps the unprivileged become problem entirely — and why keep_remote_files disabling pipelining is not an arbitrary interaction but a necessity: there is no file to keep unless one is written.

What happens if become works for most hosts and fails on a few?

Treat it as a fleet-consistency finding rather than a permissions problem, because that is almost always what it is.

The sudoers grant for an automation account is created by something — a base role, a provisioning template, an image, a manual step during a migration. Hosts that missed that step have no rule, and they cluster by whatever distinguishes them: build date, image version, the datacentre that was onboarded by a different team.

The diagnostic that pays: collect sudo -l -n output across the whole group, hash or normalise it, and count the distinct answers. One distinct answer is a consistent fleet. Three means three populations, and the failing hosts are one of them — which tells you far more than fixing the three hosts would.

Then fix it in the role that should have applied it, and let the fleet converge, rather than adding a sudoers line by hand on three machines that will drift again.

Knowledge check

Why this matters

Escalation failures feel like Ansible failures because Ansible is what printed the message, and that impression sends people to the wrong configuration file.

The reality is that become is a thin wrapper: Ansible runs sudo and reports what sudo said. Almost every message in this lesson is a message you can obtain in one command over a connection you have already proved works, and the diagnosis is a sudoers question answered by the person who owns the sudoers policy on that host.

The habit has a second benefit worth stating. Reproducing the failure with sudo -n true produces evidence you can put in a ticket without redaction — no module arguments, no verbose output, no credential anywhere near it. Compare that with the alternative approach, which is to run the play at -vvvv and paste the result, and you have another reason the boring command is the right first move.

The one command that splits the problem

The four signatures

Missing sudo password

Incorrect sudo password

user is not allowed to execute

no tty present / requiretty