Runbook: Troubleshoot an authentication failure

Permission denied (publickey) is one of the least informative messages in system administration, and it is deliberately so - sshd will not tell a client why it was rejected. That means the client message tells you almost nothing, and every minute spent theorising from it is wasted.

The method here is simple: find out which key was offered, find out which keys were expected, and compare them. Everything else is a variation on that.

When to use this runbook

• A run fails with Permission denied (publickey).
• Authentication works by hand and fails from Ansible, or vice versa.
• A host stopped authenticating after a change nobody connects to SSH.
• A newly onboarded host never authenticated.

Blast radius

None while diagnosing. The fixes touch one host or one group, and the one to be careful with is editing sshd_config - see the rollback section, because that is the change that turns an authentication problem into a lockout.

Step 1: Confirm it is authentication

ansible web02.example.com -m ping -o

Permission denied is good news in one narrow sense: the network is fine, sshd is running, and the problem is in a much smaller space.

Step 2: Establish the scope

ansible all -m ping -o | tee auth-scope.txt
grep -c SUCCESS auth-scope.txt
grep -c 'Permission denied' auth-scope.txt

Hosts do not independently stop trusting a key at the same moment. If everything failed at once, look at the controller and you will usually be finished before you would otherwise have finished reading one host’s sshd log.

Step 3: Find out what Ansible is actually offering

Not what ansible.cfg says. What the process does.

ansible web02.example.com -m ping -vvvv 2>&1 \
| grep -E 'ESTABLISH|SSH: EXEC|Offering|Authentications that can continue'

That prints the literal ssh command line Ansible builds, with every option: the user, the port, the identity file, the proxy command. Read it rather than reconstructing it - the difference between what you believe is configured and what is on that line is where a good proportion of these incidents live.

ansible-inventory -i inventories/production --host web02.example.com \
| grep -iE 'ansible_(user|host|port|ssh)'

Step 4: Reproduce by hand, correctly

sudo -iu ansible ssh -vv \
-o BatchMode=yes \
-o IdentitiesOnly=yes \
-i /home/ansible/.ssh/id_ed25519 \
-o ConnectTimeout=10 \
ansible@192.0.2.12 true

Three flags, and all three matter:

• sudo -iu ansible - testing as yourself proves nothing about the automation account, which has a different key and a different home directory.
• BatchMode=yes - without it SSH may fall back to a password prompt, and you conclude authentication works when what worked was you typing a password automation does not have.
• IdentitiesOnly=yes - without it SSH offers every key in the agent and every default identity file. A host that does not trust the key you are testing may still authenticate with a different one, and you will conclude the key is fine.

Read the -vv output for the Offering public key: lines. That is the list of what the client tried, in order, with fingerprints.

Step 5: Read the server side

The client cannot tell you why. The server can.

# From the console, or over a working access path
sudo journalctl -t sshd -t sshd-session -n 100 --no-pager

What to look for:

Accepted publickey for ansible from 192.0.2.1 port 51234 ssh2: ED25519 SHA256:REPLACE_ME
Failed publickey for ansible from 192.0.2.1 port 51234 ssh2: ED25519 SHA256:REPLACE_ME
Authentication refused: bad ownership or modes for directory /home/ansible/.ssh
User ansible from 192.0.2.1 not allowed because none of user's groups are listed in AllowGroups
Invalid user ansible from 192.0.2.1

Each of those points somewhere completely different. The fingerprint in a Failed publickey line is the key the client offered - compare it against what the host expects, which is the next step.

Step 6: Ask sshd where it reads keys from

Never assume ~/.ssh/authorized_keys.

sudo sshd -T -C user=ansible,host=web02.example.com,addr=192.0.2.1 \
| grep -iE 'authorizedkeysfile|authorizedkeyscommand|allowgroups|allowusers|denyusers|pubkeyauthentication|permitrootlogin'

The -C matters: sshd -T without a connection specification does not evaluate Match blocks, and a Match block can override AuthorizedKeysFile for exactly the user you are debugging.

Two answers change everything:

• authorizedkeysfile points somewhere central - for example /etc/ssh/authorized_keys/%u. Your edits to the home directory are irrelevant.
• authorizedkeyscommand is not none - keys come from a helper (LDAP, a key server, a CA). Nothing on the host’s filesystem grants or revokes access. Escalate to whoever owns that source.

Step 7: Compare offered against expected

# What the controller holds
sudo -iu ansible ssh-keygen -lf /home/ansible/.ssh/id_ed25519.pub

# What the host trusts - reads a whole authorized_keys file
sudo ssh-keygen -lf /home/ansible/.ssh/authorized_keys

Step 8: Check the boring causes

More authentication incidents are caused by these than by keys.

# Permissions - sshd refuses group- or world-writable paths
stat -c '%a %U:%G %n' /home/ansible /home/ansible/.ssh /home/ansible/.ssh/authorized_keys
# expect: 700 or 755 on the home, 700 on .ssh, 600 on authorized_keys

# Full disk - breaks auth and reports it as a key problem
df -h /home /var /tmp

# Account state
passwd -S ansible
chage -l ansible | grep -i expire

# Is the account excluded by policy?
sudo sshd -T -C user=ansible,host=web02.example.com,addr=192.0.2.1 \
| grep -iE 'allowgroups|allowusers|denygroups|denyusers'
id ansible

The permissions one is worth memorising: sshd refuses to use authorized_keys if the file, the .ssh directory or the home directory is writable by anyone other than the owner. A role that set mode: '0777' on a home directory produces Permission denied (publickey) on every host it touched, and nothing about the message suggests permissions.

A full /home is the second: sshd cannot do what it needs, and reports a public key failure.

Step 9: Fix, then verify

Apply the narrowest fix that addresses what you found. Then verify properly:

# Default path, no special flags - this is what automation will do
ansible web02.example.com -m ping -o
echo "exit=$?"

# And prove it is the intended key being accepted, not a fallback
sudo -iu ansible ssh -o BatchMode=yes -o IdentitiesOnly=yes \
-i /home/ansible/.ssh/id_ed25519 ansible@192.0.2.12 true && echo 'intended key OK'

# Whole group
ansible web -m ping -o | grep -c SUCCESS

Then confirm the fix exists in the repository. A key added by hand, a permission corrected on one host, an sshd_config edited live - all of these disappear at the next converge or the next rebuild.

Rollback

# 1. Validate before reloading - a bad config stops sshd starting
sudo sshd -t && echo 'config OK'

# 2. Reload from your CURRENT session, and keep it open
sudo systemctl reload ssh 2>/dev/null || sudo systemctl reload sshd

# 3. From a SECOND terminal, prove a new login works
ssh -o BatchMode=yes ansible@192.0.2.12 true && echo 'new session OK'

# 4. Only now close the first session

Step 3 is the whole point. An existing session survives a reload of a broken configuration; the next login does not. Closing the session before testing a new one is how an authentication problem becomes a lockout.

Common patterns

Escalation

Escalate when:

• A key you do not recognise is in authorized_keys, or a host key changed without explanation. Security, immediately.
• AuthorizedKeysCommand is in use. The host is not authoritative.
• The account is locked or expired, or the disk is full. Host owner.
• Every host failed simultaneously and the controller checks out.