critical riskcluster affecting~30 min

Runbook: LDAP or AD unavailable - restore central identity

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the failure: server down, network down, configuration error
· Determine blast radius: how many hosts affected
· Verify local break-glass accounts exist on every host

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Identify the failure layer (server, network, SSSD, DNS)
2Verify local break-glass access on every host
3If server is down: work to restore or failover
4If network is down: route around it
5If SSSD is stuck: restart and invalidate cache
6Once central identity is restored: verify all hosts
7Document the incident timeline

4 · Verification

Confirm the procedure actually fixed the problem.

✓Local break-glass works on every affected host
✓Central identity is reachable from at least one host
✓getent returns expected users
✓Sudo works for the admin group
✓SSH SSO works for at least one test user

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If SSSD restart broke auth, restore /etc/sssd/sssd.conf from backup
↶If SSSD is wedged rather than misconfigured, clear its cache with sss_cache -E, or stop sssd, remove /var/lib/sss/db/*, and start it again - a restart alone reuses the poisoned cache. Confirm with id <a-directory-user> before closing the console
↶Use console access if SSH is broken

6 · Escalation

When the runbook isn't enough, contact:

· If the AD domain itself is down, escalate to AD team
· If multiple hosts are affected, escalate to cluster ops
· If the failure recurs frequently, escalate for permanent fix
· If break-glass accounts are missing, escalate for security review

This runbook triages a central identity outage. Central identity (AD or LDAP) being unreachable makes logins fail. The recovery depends on whether SSSD has cached credentials and whether local break-glass accounts exist.

When to use this runbook

Use this runbook when:

Users cannot log in to a host.
“User is not in the sudoers” errors appear.
SSSD logs show “Server is unreachable”.
Authentication for central users fails.

Inputs

Gather before starting:

Affected hosts (one or many).
Central identity server (AD DC, LDAP server) status.
Network status between hosts and central server.
Recent changes (SSSD config, AD changes, network changes).

Procedure

Step 1: Determine the failure layer

Read-only / Safeping

# Is the central server reachable?
ping dc1.example.com
nc -vz dc1.example.com 389
nc -vz dc1.example.com 636

# Is DNS working?
dig dc1.example.com
dig _ldap._tcp.example.com SRV

# Is SSSD running?
sudo systemctl status sssd
sssctl domain-status example.com

Identify which layer is failing:

Server: AD DC is down or unreachable.
Network: routing or firewall blocks LDAP/Kerberos ports.
DNS: name resolution for the DC fails.
SSSD: SSSD is stuck in offline or misconfigured.

Step 2: Verify local break-glass

Read-only / Safessh

ssh breakglass@host

If break-glass works, you have access. If not, you need console access.

Step 3: For server failure

If the AD DC is down:

Contact the AD team.
Failover to a secondary DC if available.
Wait for the DC to recover.
During the outage, local accounts with cached credentials continue to work.

Step 4: For network failure

If network blocks LDAP/Kerberos:

Check firewall rules for ports 389/636/88.
Check routing.
Work around: SSH to a host in a working network, then connect to the broken host via the bastion.

Step 5: For SSSD failure

Service impact possiblesystemctl restart

# Restart SSSD
sudo systemctl restart sssd

# Invalidate cache
sudo sss_cache -E

# Check logs
sudo journalctl -u sssd -n 50

Common SSSD issues:

Stale offline mode: sss_cache -E.
Bad config: restore from backup.
Permission on sssd.conf: chmod 600, chown root.

Step 6: For DNS failure

If the DC’s hostname does not resolve:

Check the local resolver.
Use IP directly: configure SSSD with ad_server = 10.0.0.5 (IP) instead of hostname.
Fix DNS once the central resolver is back.

Step 7: Verify recovery

Once the central server is reachable:

Read-only / Safesssctl domain-status

sssctl domain-status example.com    # online
getent passwd alice@EXAMPLE.COM
kinit alice@EXAMPLE.COM
klist
ssh alice@server.example.com    # SSO works

All four should succeed.

Step 8: Document

In the incident log:

Symptom (lockouts, auth failures).
Affected hosts.
Failure layer (server, network, DNS, SSSD).
Recovery procedure.
Time to recovery.
Recommendation for prevention.

Common patterns

Symptom	Most likely cause
All users fail, no cached credentials	Server down + cache empty
Existing users work, new users fail	Cache valid for old, expired for new
`sssd` in offline mode after server recovery	`sss_cache -E` or restart
DNS resolution for DC fails	Local resolver issue
All hosts fail simultaneously	Central server or shared network

Escalation

Escalate when:

AD domain itself is down (AD team).
Multiple clusters affected.
Recurring failure.
Break-glass accounts are missing (security review).

Bring: affected hosts, failure layer, recovery actions.