PAM and NSS troubleshooting - fixing auth that breaks

Authentication failures are stressful because they block access. A systematic approach - read the stack, check the logs, test with getent, validate - turns them from panic into routine fixes.

Debug a PAM failure

A user reports “login is failing”. The user provides:

• Username.
• Service (login, sshd, sudo).
• Error message (if any).

Step 1: Check the auth log

sudo journalctl -t sshd -t sshd-session -n 50
sudo journalctl | grep -i 'pam\|auth' | tail -50

Match on the syslog identifier (-t), not the unit (-u). The SSH unit is sshd.service on RHEL-family systems but ssh.service on Debian-family systems, where sshd.service is only an alias. journalctl -u sshd therefore prints -- No entries -- on Ubuntu, which reads as “no failures” when the failures are right there.

Common lines:

pam_unix(sshd:auth): authentication failure; logname=... user=alice ...
pam_faillock(sshd:auth): ... (account is locked)

The format is pam_<module>(<service>:<type>): <message>. Read it carefully.

Step 2: Test the service directly

sudo sshd -t        # config syntax
sudo ssh -vvv user@localhost    # verbose client

The -vvv shows every step of the SSH handshake.

Step 3: Enable PAM debug logging

Edit the service config (e.g. /etc/pam.d/sshd):

auth required pam_unix.so debug

The debug flag adds verbose logging to syslog.

sudo systemctl reload sshd
sudo journalctl -f
# Try the failing login

The syslog now shows every PAM module’s verdict.

Step 4: Test each module individually

# Test pam_unix - does the shadow entry exist, and is the account usable?
# Never print field 2: that is the password hash, and putting it on a
# terminal writes it into scrollback, tmux buffers and shell history.
sudo getent shadow alice | cut -d: -f1,3-8
sudo passwd -S alice        # P = usable password, L = locked, NP = no password
sudo chage -l alice         # expiry, inactivity, last change

# Test pam_faillock
faillock --user alice

# Test pam_ldap / pam_sss
id alice
getent passwd alice

Debug an NSS failure

A command reports “no such user” but the user exists.

Step 1: Test getent

getent passwd alice

If empty:

• alice does not exist in any NSS source.
• An NSS module is failing.

If populated: the command should work. Check the application uses NSS (it usually does).

Step 2: Check NSS configuration

cat /etc/nsswitch.conf

Look for typos in module names or missing modules.

Step 3: Check each module

# files
grep alice /etc/passwd

# sss
sssctl user-show alice

# ldap (direct)
ldapsearch -H ldap://ldap.example.com -b "ou=People,dc=example,dc=com" "(uid=alice)"

Step 4: Check NSS module loading

ldd /lib/x86_64-linux-gnu/libnss_sss.so
ldd /lib/x86_64-linux-gnu/libnss_files.so

A missing library means the module will silently fail to load.

Recover from a bad PAM config

If a PAM change locks out all users:

Step 1: Get a console

Cloud console, IPMI, hypervisor console. Boot into recovery mode if needed.

Step 2: Edit /etc/pam.d/<service>

# Restore from backup
sudo cp /etc/pam.d/sshd.backup /etc/pam.d/sshd

Or comment out the offending line.

Step 3: Restart the service

sudo systemctl restart sshd

Step 4: Test

# From another terminal or window
ssh user@host

Common error messages

Knowledge check