External identity failure implications — when the directory is down

External identity stores fail. LDAP replication breaks. RADIUS servers run out of capacity. TACACS+ configurations regress. When the external store fails, the operators who depend on it cannot authenticate to the firewall. If the firewall is the only path to fixing the external store — and often it is, because the external store is on a network behind the firewall — the operator is locked out exactly when they need to act.

This lesson covers the failure modes that take down external identity stores, what happens to OPNsense when the store is unreachable, how to design break-glass paths, and the operational runbooks that keep operators able to act during identity outages.

Why external identity stores fail

External identity stores are infrastructure, and infrastructure fails. The failure modes specific to identity:

1. Network partition. The management network that carries LDAP/RADIUS/TACACS+ traffic goes down. The firewall can reach the Internet but cannot reach the identity server.
2. Server crash. The LDAP/RADIUS/TACACS+ server process exits. The data is intact on disk but the protocol endpoint is down.
3. Replication lag or split. Multi-master replication diverges. The firewall sees an inconsistent view of the directory.
4. Certificate expiry. The LDAP server certificate (for ldaps://) or the RADIUS server’s TLS configuration expires. TLS handshakes fail. Bind operations fail.
5. Service account credential rotation. The directory administrator rotates the service account password but the firewall is not updated. Bind fails for every operator.
6. Schema change. A directory schema migration changes the attributes the firewall queries. The search returns nothing.
7. Denial of service. The identity server is overloaded or under attack. Responses time out.

What happens to OPNsense when the store is unreachable

OPNsense’s behaviour when the external store is unreachable depends on how the integration is configured:

• Pure external: OPNsense consults the external store for every authentication. When the store is unreachable, every operator authentication fails. The firewall is locked.
• External with local fallback: OPNsense tries the external store first. If the store is unreachable or returns an error, OPNsense falls back to local accounts. Local break-glass accounts continue to work.
• External-only with cache: OPNsense caches recent successful authentications. Cached sessions continue to work for the cache duration. New authentications fail.

The fallback configuration is the production answer. Without it, the firewall is one LDAP outage away from being unreachable.

$ configctl auth test ldap

LDAP server: ldaps://ldap.example.com:636
Bind DN: cn=opnsense-svc,ou=service-accounts,dc=example,dc=com
Bind result: success
Search result: success (1 entry returned)
Test authentication as operator: success (member of: admins, neteng)
LDAP server reachable: yes

Designing break-glass paths

A break-glass path is a documented, practiced way to authenticate when the normal path is unavailable. For external identity failures on OPNsense, the break-glass paths:

1. Local break-glass admin account. A local account that exists on the firewall regardless of the external store. The password is in a sealed envelope in a safe. The account has admin privileges. The account is the first line of break-glass.

2. Out-of-band management access. A console connection (serial, IPMI, KVM over IP) that does not depend on the network the identity store is on. The operator connects via the console, logs in with the local break-glass account, and fixes the firewall.

3. Cached sessions. If the operator authenticated before the identity store failed, their session may still be valid (until the session timeout). A short session timeout means few cached sessions; a long session timeout means more cached sessions but weaker security.

4. Backup identity server. A second LDAP server (or RADIUS, or TACACS+) configured in OPNsense as a fallback. If the primary is down, the secondary takes over.

5. Direct configuration access. SSH with key-based authentication that does not require the external identity store. The operator’s SSH key is in the firewall’s authorized_keys; SSH works regardless of LDAP.

The recovery runbook

When the identity store fails, the recovery must be documented and practiced. A runbook for LDAP outage:

IDENTITY STORE OUTAGE RUNBOOK (LDAP)

Symptoms:
  - Operators report "authentication failed" on the GUI
  - configctl auth test ldap returns bind failure or
    timeout
  - External monitoring shows LDAP server unreachable

Step 1: Confirm scope
  - Is one firewall affected or all firewalls?
  - Is the LDAP server reachable from the firewall?
    (ping, telnet to 636)
  - Is the LDAP server reachable from the operator
    workstation?

Step 2: If one firewall only
  - Check the firewall's network connectivity to the LDAP
    server.
  - Check the firewall's LDAP configuration (bind password,
    certificate trust).
  - Use local break-glass admin account to log in and fix
    the configuration.

Step 3: If all firewalls affected
  - The LDAP server itself is the problem.
  - Engage the directory team.
  - Operators use local break-glass admin accounts to
    continue working.
  - The break-glass accounts have admin privileges — use
    them sparingly, log every action.

Step 4: Recovery
  - LDAP server is restored.
  - Verify configctl auth test ldap returns success.
  - Verify a normal operator can authenticate.
  - Disable any temporary workarounds.
  - Document the incident.

The runbook is the difference between an incident that is resolved in 30 minutes and an incident that lasts 4 hours while the on-call team improvises.

The cascade trap

The cascade trap: the identity store fails, the operator cannot reach the firewall, the firewall is the only path to fixing the identity store, and the operator is locked out. The fix is to design the system so that the operator can reach the firewall when the identity store is down.

The cascade happens in two flavours:

1. Identity on a network the firewall protects. The LDAP server is on a network behind the firewall. The firewall rules normally allow the operator to reach the LDAP server via the firewall. When the LDAP server fails, the operator may also be unable to reach the firewall itself because the firewall’s management plane is tied to the same broken path.
2. Identity on the same authentication path as the firewall. The operator authenticates to the firewall using LDAP. When LDAP fails, the operator cannot authenticate to the firewall. There is no path to fix anything.

The mitigation is to design break-glass paths that do not depend on the broken component. Local accounts. Out-of-band console access. SSH key-based authentication that bypasses LDAP. Backup identity servers on a separate network path.

Summary

• External identity stores fail. The failure modes are network partitions, server crashes, replication problems, certificate expiry, credential rotation, schema changes, and overload.
• Without local break-glass, the firewall is one identity outage away from being unreachable. Always have a local admin account that works regardless of the external store.
• Break-glass paths include local accounts, out-of-band console, cached sessions, backup identity servers, and SSH key-based access.
• Document the recovery runbook and practice it. Test the break-glass path under realistic conditions.