Central identity failure modes - what breaks when LDAP or AD is down

Central identity is the substrate on which modern Linux fleets run. A central identity outage cascades: logins fail, sudo fails, automation that uses central credentials fails. This lesson is about designing for resilience.

How central identity breaks

1. AD or LDAP server unreachable
2. SSSD cannot reach the central server
3. SSSD enters offline mode (if cache_credentials is true)
4. Cached credentials work for users who logged in before
5. New users cannot log in
6. Service accounts that rely on central credentials fail
7. Automation breaks

Symptoms:

• New logins fail.
• sudo fails (“user is not in the sudoers file”).
• Backup jobs that use AD credentials fail.
• Monitoring agents that use AD credentials fail.

Design for resilience

Cache aggressively

SSSD caches identity information by default. Tune the cache for longer offline windows:

[domain/example.com]
cache_credentials = True
entry_cache_timeout = 14400    # 4 hours
pwd_expiration_warning = 7

[sssd]
services = nss, pam

A longer cache reduces the blast radius of an outage but increases the staleness of identity changes (e.g. a disabled user remains “enabled” until the cache expires).

Always have a local fallback

Every host should have at least one local user that can log in without central identity:

sudo useradd -m -G wheel,sudo breakglass
sudo passwd breakglass

Test the local login regularly:

ssh breakglass@host

breakglass is the last-resort credential. Store its password in a vault or sealed envelope.

Document the offline path

For each host, document:

• The local break-glass user.
• The break-glass password location.
• The SSSD offline cache duration.
• The procedure to bring SSSD back online.

Service accounts

For service-to-service auth, do not rely on central identity. Use:

• Local accounts for system services.
• Certificates (mTLS, SSH certs) for service-to-service.
• Vault tokens (HashiCorp Vault, etc.) for cloud and automation.

A service that fails because AD is down is a service that should not depend on AD.

Network resilience

• Multiple AD Domain Controllers in SSSD config.
• DNS with multiple resolvers.
• Network path redundancy (multiple uplinks).

[domain/example.com]
ad_server = dc1.example.com, dc2.example.com, dc3.example.com

SSSD tries each DC; if one is unreachable, it falls over to the next.

Diagnose a central identity outage

# Is the DC reachable?
ping dc1.example.com
nc -vz dc1.example.com 389
nc -vz dc1.example.com 636

# Is DNS resolving?
dig dc1.example.com
dig _ldap._tcp.example.com SRV

# Is SSSD running?
sudo systemctl status sssd

# What does SSSD say?
sssctl domain-status example.com

# Recent errors?
sudo journalctl -u sssd -n 50

Common patterns:

• DC unreachable: network or DC outage. Fix upstream.
• DNS broken: local resolver issue. Fix DNS.
• SSSD offline: stuck in offline mode. sss_cache -E.
• Wrong realm: SSSD config error. Fix sssd.conf.

Bring SSSD back online

When the central server is restored:

# Force SSSD to retry
sudo sss_cache -E
sudo systemctl restart sssd

# Verify
sssctl domain-status example.com
getent passwd alice@EXAMPLE.COM

If SSSD stays offline after the server is back:

• Check SSSD logs for the underlying error.
• Verify DNS resolves the DC.
• Verify the Kerberos keytab is valid.

The break-glass procedure

When central identity is down and you need to log in:

1. Connect via out-of-band (cloud console, IPMI).
2. Log in as the local breakglass user.
3. Fix central identity (or wait for it to recover).
4. Document the incident.

Knowledge check