Databases and network devices: identity, rotation and connection pools

The last two things in an estate to get proper credential management are usually the database and the firewall. Both predate the platform tooling, both are configured through their own interfaces, and both punish a careless rotation with an outage that arrives hours after the change. This lesson is about the identity and lifetime questions those two classes of system ask, which are the same questions as everywhere else with the automation removed.

The server identity nobody verifies

A datastore or an appliance presents a certificate; the client decides whether to check it. Plenty of drivers and management tools default to encrypting the connection while accepting whatever certificate arrives, which buys confidentiality against a passive observer and nothing whatsoever against an active one. An attacker who can answer for the address gets a complete, encrypted, fully readable session.

Verification is two independent tests. The chain test asks whether the certificate leads to a trust anchor the client holds. The name test asks whether the identity in the certificate matches the name the client asked for. They fail differently and they are fixed differently, and the fastest diagnosis is to read which one you got. A chain failure reports being unable to get the local issuer certificate. A name failure reports that no alternative certificate subject name matches the target hostname.

HOST=fw-01.example.com
PORT=443
openssl s_client -connect "$HOST:$PORT" -servername "$HOST" </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

The name test is where old internal certificates die. RFC 9525 removed the Common Name fallback: a client must not use the Common Name RDN, or any other RDN in the subject, to identify a service. Identity lives in the subject alternative name and nowhere else. An internal certificate generated years ago by an appliance’s own issuing function, carrying a Common Name and no SAN at all, worked for a decade and then stopped working when a client library was updated. Nothing changed on the server.

Client identity, and what a certificate proves to a datastore

Mutual TLS to a database replaces a password with a key pair. The server verifies the presented client certificate against a CA it trusts and checks its validity window, then maps the identity in that certificate onto a database role. Those are two separate decisions and it is worth being pedantic about them: authentication asks whether the certificate is genuine and current, and authorisation asks which role the identity is allowed to become.

The gains are real. There is no shared password to leak into a configuration file, the credential expires by itself, and a single internal CA can serve every application without any per-application secret being distributed. The costs are equally real. The client private key on the application host is now the credential, with exactly the same file mode, ownership and backup-exposure questions a password had. And revocation only works if the server actually loads and enforces a revocation list, which for a private PKI is a thing you operate rather than a thing you assume.

Rotation meets the connection pool

A connection pool authenticates when it opens a connection and never again. The database associates the session with a role at login and subsequent queries are not re-authenticated. Change the password and nothing happens: every pooled connection keeps working, and the failure appears whenever the pool next opens a new one, which may be minutes or many hours later, at a traffic trough, with no change correlated to it.

flowchart LR
    A["Credential issued\nlease TTL 120s"] --> B["Pool opens connection\nauthenticates once"]
    B --> C["Queries run\nno re-authentication"]
    A --> D["Lease expires\nor is revoked"]
    D --> E["New connection refused\nrole does not exist"]
    C --> E

The dual-credential pattern exists for exactly this reason. Keep two usable credentials at all times, switch the application to the second, drain and verify, then retire the first. That converts a cliff edge into two safe steps, and it is the only way to rotate a shared database password without a maintenance window.

Dynamic credentials make the whole timeline explicit rather than implicit. A secret manager issues a fresh database role per lease and hands back a username, a password and a lease with a duration attached. On the database side the role carries a validity timestamp written by the creation statement, so it expires whether or not anything asks it to.

expire_time     2026-08-26T21:25:43.891514518Z
issue_time      2026-08-26T21:23:43.891514368Z
renewable       true
ttl             1m59s

That lease is two minutes long, which is deliberately shorter than anything a real service would use, and it shows the shape of the problem clearly. If the pool holds connections for longer than the lease and never renews, the application is running on a credential the secret manager considers finished. Revoking the lease removes the role outright, and the next login attempt fails in a way that is at least unambiguous.

psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  role "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423" does not exist

Three numbers therefore have to be designed together: the lease duration, the renewal interval the client uses, and the pool’s maximum connection lifetime. The pool lifetime must be comfortably shorter than the lease, and the application must treat re-authentication as a normal event rather than a fatal error. That is the same requirement the projected ServiceAccount token imposes one layer up, arriving through a different door.

Certificates on firewalls, VPNs and management interfaces

A network appliance typically carries four distinct certificate roles, and confusing them is the source of most appliance certificate incidents. There is the certificate presented by the management web interface and API. There is the identity the VPN service presents to remote clients. There is the set of client certificates the device issues or trusts for those users. And there may be a certificate used by an inspection or captive-portal function, which is a different trust decision again.

These are the worst-monitored certificates in most estates for structural reasons rather than negligent ones. They are not produced by the delivery pipeline, so no CI job knows they exist. They are frequently issued by the appliance’s own built-in CA, whose parameters were chosen once during commissioning. Renewal is a manual sequence in a graphical interface, which means it is not scheduled and not tested. And the inventory that would catch them lives in the platform team’s tooling, which does not scan the management network.

Two further properties deserve stating plainly. An appliance’s built-in CA is a real trust anchor for your remote access, so its private key material sits on the appliance and travels inside every appliance configuration backup; those backups need the handling you would give a CA key, not the handling you would give a config file. And revoking a VPN client certificate only removes access if the concentrator loads and enforces a revocation list, so establish whether yours fails closed or fails open when that list is stale before you rely on revocation as a control.

Monitoring closes the gap, and it has to run from where a client sits rather than on the device itself. A probe that opens the connection sees the chain the device actually serves, including a missing intermediate that a local file inspection would never reveal. The expiry primitive is simple enough to run anywhere: a check for a horizon of ninety days prints that the certificate will expire and exits non-zero, which is all a monitoring integration needs.

Production discipline

1. Read which verification test failed. An issuer error and a name error have different fixes, and neither of them is disabling verification.
2. Reissue legacy internal certificates with a SAN. Any certificate identifying a service by Common Name alone is already failing on updated clients, whatever it did last year.
3. Set the pool lifetime below the credential lifetime. Lease duration, renewal interval and maximum connection age are one design, not three independent settings.
4. Rotate with two credentials, never one. Introduce, switch, drain, verify, retire; a single in-place change is a cliff edge with a delayed trigger.
5. Inventory appliance certificates deliberately. They are invisible to pipeline tooling, so they need an explicit owner, an external probe and a tested out-of-band access path.

Cross-course references

• OPNsense for Production Network & Security Administrators - Parts XXIV (PKI and Certificates) and VIII (Management Plane Security) cover the appliance-side procedures for the certificate roles this lesson treats as expiring credentials with owners.
• VyOS for Production Network Engineers - Parts XLVII (Management Plane Hardening) and XLII (IPsec) cover the device configuration behind the VPN and management identities discussed here.
• Observability for Production Sysadmins - Part LXIV (TLS Monitoring) covers building the external probe and the expiry alert that these appliances are usually missing.

Quiz