Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

advancedIdentity / auth~30 min

Break/Fix: Nobody can log in and sudo hangs after the LDAP servers go away

Reported symptoms

  • ●SSH logins to every host hang for about a minute and then fail
  • ●sudo hangs on the one host where an engineer still has a live session
  • ●Running services are unaffected and keep serving traffic normally
  • ●ls -l shows numeric UIDs instead of usernames
  • ●A local console login as root works immediately

Evidence

  • · `ssh user@host` hangs, then returns "Permission denied (publickey,password)"
  • · `getent passwd alice` returns nothing and takes about 60 seconds
  • · `getent passwd root` returns immediately from /etc/passwd
  • · `systemctl status sssd` shows active, with "Backend is offline" in the journal
  • · `ldapsearch -x -H ldaps://ldap1.internal -b dc=example,dc=com` times out
  • · `ss -tn state established dport = :636` shows no connections from this host
  • · `grep cache_credentials /etc/sssd/sssd.conf` returns nothing
  • · `sudo -l` hangs; /etc/nsswitch.conf lists "sudoers: files sss"
Diagnosis and resolutionclick to reveal

Root cause

Both directory servers are unreachable, so every name-service lookup that is routed to sss has to wait for its timeout before failing. SSH cannot resolve the account, so authentication fails. sudo is worse: nsswitch routes sudoers through sss as well, so even an already-authenticated session blocks on the same timeout before falling back. Nothing had cached credentials, because cache_credentials was never enabled, so the fleet had no offline mode at all. Running services are unaffected because they resolved their identity at start-up and never look it up again - which is exactly why the outage looks smaller than it is until the first service restart.

Remediation

Restore the directory first; everything else is mitigation. While it is down, use the break-glass local account and console access. Then remove the single point of failure: enable cache_credentials and a long offline credential expiry in sssd, shorten the LDAP network and search timeouts so failures are fast rather than hanging, list more than one directory server, keep a local break-glass account with a distinct password in a vault, and make sure sudo has a files-based rule that resolves without the directory.

Verification

With the directory blocked at the firewall on one test host, confirm a previously-seen user can still log in from the sssd cache, confirm sudo answers within seconds rather than hanging, and confirm the break-glass account works from the console. Confirm getent fails fast rather than blocking for a minute.

Prevention

Central identity is a tier-0 dependency and needs the same redundancy and failure testing as storage. Cache credentials, bound timeouts, at least two directory servers in different failure domains, a tested break-glass account per host, and a documented console access path. Test it by blocking the directory in a game day, not by waiting for the real outage.

Reported symptoms

  • SSH logins to every host hang for about a minute and then fail.
  • sudo hangs on the one host where an engineer still has a live session.
  • Running services are unaffected and keep serving traffic normally.
  • ls -l shows numeric UIDs instead of usernames.
  • A local console login as root works immediately.

Evidence provided

$ time getent passwd alice
real    0m58.104s

$ time getent passwd root
root:x:0:0:root:/root:/bin/bash
real    0m0.004s

$ ls -l /srv/app
drwxr-xr-x 2 10412 10500 4096 Aug  9 11:02 releases

$ journalctl -u sssd -n 5
Aug 11 08:31:02 host sssd_be[1420]: Backend is offline
Aug 11 08:31:02 host sssd_be[1420]: Failed to connect to ldaps://ldap1.internal
Aug 11 08:31:33 host sssd_be[1420]: Failed to connect to ldaps://ldap2.internal
Aug 11 08:31:33 host sssd_be[1420]: Domain example.internal is Offline

$ ldapsearch -x -H ldaps://ldap1.internal -b dc=example,dc=com -s base
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

$ grep -E 'cache_credentials|offline_credentials|ldap_network_timeout|ldap_uri' /etc/sssd/sssd.conf
ldap_uri = ldaps://ldap1.internal, ldaps://ldap2.internal

$ grep -E '^(passwd|group|sudoers):' /etc/nsswitch.conf
passwd:   files sss
group:    files sss
sudoers:  files sss

$ systemctl status nginx --no-pager | head -3
● nginx.service - nginx
     Active: active (running) since Fri 2026-08-07 14:20:11 UTC

Work the evidence before reading on

Five observations, and they do not all have the same cause:

  1. Every lookup for a directory user takes about a minute and then fails, while root answers instantly.
  2. Authentication fails on hosts that had these users logged in yesterday.
  3. sudo hangs even for a session that is already authenticated.
  4. Running services are completely unaffected.
  5. Both directory servers are unreachable at once.

Candidate causes worth ruling in or out: DNS failure, a firewall change, expired LDAP TLS certificates, both directory servers down, an nsswitch misconfiguration, and a missing offline cache. Decide which explain which symptom before continuing — several are true at the same time, and the fix for each is different.

Root cause

1. The directory is a hard dependency on every lookup

/etc/nsswitch.conf routes passwd, group and sudoers to sss after files. Local accounts in /etc/passwd resolve instantly and never touch the network — which is why root works and alice does not, and why the numeric UIDs in ls -l are the same fault wearing a different hat.

For anything not in files, every lookup goes to sssd, sssd is offline, and the caller waits for the timeout.

2. There was no offline mode, because caching was never enabled

cache_credentials is absent from sssd.conf. sssd caches identity data by default, but without cache_credentials = true it stores no credential material, so it cannot authenticate anyone while the backend is offline. The fleet therefore had no degraded mode at all: the directory going away meant nobody could log in anywhere, at the same moment.

3. sudo hangs because sudoers is a directory lookup too

This is the part that surprises people. An engineer with a live session is already authenticated, so sudo “should” work. But sudoers: files sss means sudo asks the directory which rules apply before it does anything, and that lookup blocks for the full timeout. The one person still able to touch the host loses their ability to fix it.

4. Running services are fine, which makes the outage look smaller than it is

nginx resolved its User= at start-up and has not looked it up since. Every already-running service behaves the same way, so traffic keeps flowing and dashboards stay green.

The trap is that the next restart, deploy, or automatic Restart=on-failure will fail to resolve its user and the service will not come back. The outage is latent, not absent. Freeze deployments for the duration.

Resolution

  1. Get in through a path that does not use the directory. Out-of-band console (BMC/serial), then the local break-glass account. If you cannot answer "how do I log in with LDAP down?" from memory, that is the first prevention item, not a step here.
  2. Freeze deployments and restarts fleet-wide. Running services are fine; restarted ones will not come back until identity resolves. Announce this before someone discovers it during a routine deploy.
  3. Confirm the failure domain before touching client config. Is it the directory itself, the network path, DNS, or an expired LDAP server certificate? Each looks identical from the client:
  4. ``bash getent hosts ldap1.internal # DNS resolving? nc -zv ldap1.internal 636 # port reachable? openssl s_client -connect ldap1.internal:636 </dev/null | openssl x509 -noout -dates ``
  5. Restore the directory. Everything below is mitigation, not a fix.
  6. Give the fleet an offline mode. In /etc/sssd/sssd.conf, per domain:
  7. ``ini cache_credentials = true offline_credentials_expiration = 0 # 0 = never expire cached credentials ldap_uri = ldaps://ldap1.internal, ldaps://ldap2.internal, ldaps://ldap3.dr ``
  8. Bound the timeouts so failures are fast, not hanging. A minute of blocking per lookup is what turned this into an unfixable outage:
  9. ``ini ldap_network_timeout = 5 ldap_opt_timeout = 5 ldap_search_timeout = 10 dns_resolver_timeout = 3 ``
  10. Take sudo off the critical path. Keep an explicit files-based rule for the break-glass account and the on-call group so sudo resolves without the directory:
  11. ``text # /etc/sudoers.d/10-breakglass (validate with: visudo -c -f /etc/sudoers.d/10-breakglass) breakglass ALL=(ALL:ALL) ALL ``
  12. Apply and restart sssd, clearing the stale cache: sudo sss_cache -E && sudo systemctl restart sssd

Verification

  1. Prove the offline mode on one test host. Block the directory on that host only, then log in as a user who has logged in before. It must succeed from the sssd cache
  2. Prove lookups now fail fast. time getent passwd nosuchuser should return in single-digit seconds, not a minute
  3. Prove sudo still answers. time sudo -l with the directory blocked should return promptly using the files-based rule
  4. Prove the break-glass path. Log in with the break-glass account over the console, not over SSH, and confirm it can escalate
  5. Prove a service can restart without the directory. systemctl restart nginx on the test host must succeed, confirming its User= resolves
  6. Unblock and re-verify that the cached and live paths agree: getent passwd alice should return the same entry with the directory reachable again