You cannot rotate what you do not know exists

A credential inventory is the list of every private key, certificate, token and password an estate depends on, together with enough context to act on each one. Without it, rotation degrades into a best-effort pass over whatever the team happens to remember. Expiry monitoring, compromise response and access review inherit the same blind spot, because each of them begins by enumerating a set and then doing something to every member of it.

Every credential operation is a loop over a set

Consider what “rotate the database password” actually means as a procedure. Generate a new secret. Write it where each consumer reads from. Restart or signal each consumer so it picks the new value up. Confirm every consumer is authenticating with the new value. Retire the old one. Four of those five steps iterate over a list of consumers, and the fifth depends on the fourth having been complete.

The dangerous property is not that the loop errors on a missing member. It is that the loop has no way to know the member exists, so it completes, logs success, and the retirement step then breaks a consumer that nobody knew was in the set. The rotation record shows a clean run. The outage arrives minutes or hours later, attributed to whatever was deployed nearby.

• Rotation iterates over consumers. A consumer absent from the list keeps the retired credential and fails at the moment the old value stops working, which is deliberately later than the change.
• Revocation iterates over issued objects. You can only revoke serial numbers you can name. A certificate issued from a compromised key by a CA you forgot you operated stays valid.
• Expiry response iterates over deployed copies. Renewing the certificate in the register does nothing for the identical certificate copied onto a load balancer two years ago.
• Access review iterates over holders. An unrecorded API token held by a decommissioned service is a standing grant that no review will ever look at.

Each of those is a security control on paper and a partial function in practice. The size of the gap between the set you act on and the set that exists is the real measure of how well the estate is run.

How an estate loses count

Credentials do not arrive through one door. They arrive through several, and each door keeps its own record in its own format, and none of those records is the union.

An ACME client issues on its own schedule and keeps its state on the host that ran it. What certbot leaves behind is a directory of symbolic links pointing at the generation currently in use:

cert.pem      -> ../../archive/web.lab.example/cert1.pem
chain.pem     -> ../../archive/web.lab.example/chain1.pem
fullchain.pem -> ../../archive/web.lab.example/fullchain1.pem
privkey.pem   -> ../../archive/web.lab.example/privkey1.pem

That is a complete and accurate register of one credential, maintained automatically, and legible only to somebody already logged into that particular machine. Multiply it by every host running an ACME client and the estate has an excellent distributed record that nobody can query.

An internal issuing CA has an authoritative log of what it signed, which tells you nothing about certificates signed by the other CA that a platform team stood up for a migration and never decommissioned. A secret manager knows its own objects and nothing about the key that was generated by hand during an incident at three in the morning and pasted into a configuration file.

Appliances and vendor products ship with credentials already installed. Nobody issued those, so nobody recorded them.

Then there is copying. A private key generated once is duplicated into a container image layer, a configuration management repository, a backup archive and an engineer’s laptop. Every copy is independently capable of impersonating the identity, and none of the copies appears in the issuer’s log, because the issuer only saw the first one.

# What one host actually carries, as opposed to what a register claims.
find /etc /opt /srv -type f \( -name '*.pem' -o -name '*.crt' \) 2>/dev/null

# For each candidate, the four facts that make it actionable.
CERT=/etc/ssl/certs/api.example.com.pem
openssl x509 -in "$CERT" -noout -subject -issuer -serial -dates

Running that on a single host is not discovery. It is a demonstration of the asymmetry: the host knows precisely what it has, and the organisation does not, and there is no channel between the two unless somebody builds one.

The gap has a shape

flowchart TD
    A["Public CA via ACME client"] --> E["Credentials in production"]
    B["Internal issuing CA"] --> E
    C["Secret manager or KMS"] --> E
    D["Generated by hand during an incident"] --> E
    E --> F{"Present in the register?"}
    F -- "yes" --> G["Renewed and rotated on schedule"]
    F -- "no" --> H["Discovered when it stops working"]

Four issuance paths feed one production estate, and only some of what they produce reaches the register. The branch on the right is not an exotic edge case: it is the ordinary fate of anything issued outside the path that the tooling watches. The credential works perfectly until its validity window closes or its key is rotated underneath it, and then it becomes an incident whose first fifteen minutes are spent establishing what the thing even is.

Count keys, not only certificates

An inventory built around certificates records the wrong object. A certificate is public. It is metadata that binds a name to a public key and carries a signature saying a CA agreed. Publishing one costs nothing. The private key is the thing that grants the power, and it is the unit that compromise is measured in.

This matters in two directions. First, one key may back several certificates, issued at different times, possibly by different CAs, possibly for different names. Revoking one of them leaves the others usable, so a key-compromise response that iterates over certificates rather than over keys is incomplete by construction. Second, renewing a certificate while reusing the existing key produces a fresh validity window with no reduction in exposure at all. The register has to be able to tell those two events apart, and it can only do that if key material is a first-class object in it rather than a field on a certificate row.

The honest position on completeness is that the gap is never zero. There is no moment at which an estate can declare its register total, because a new credential can be created by anyone with shell access at any time. What can be measured is the fraction of observed credentials that the register already knew about, and the age distribution of the entries it holds. Those two numbers move in response to real work. “The inventory is complete” does not.

Production discipline

1. Treat the register as a precondition, not a deliverable. Before committing to a rotation programme, establish what fraction of the estate the programme can actually reach. A rotation plan over an unmeasured set is a plan to break things in an order nobody chose.
2. Make key material the primary object. Certificates, tokens and derived credentials hang off it. Compromise response follows keys; renewal schedules follow certificates. Conflating them guarantees that one of the two is done wrong.
3. Record the copies, not just the originals. Every place a private key or token has been written is a separate entry with its own owner and its own removal task, including images, backups and configuration repositories.
4. Publish coverage rather than completeness. Report the proportion of discovered credentials that were already known and the number of entries not confirmed within the last review period. Both numbers are actionable; a claim of totality is not.
5. Give every entry an owning rota, never a named individual. People change teams. The credential outlives them, and an ownership field pointing at somebody who left is functionally the same as no owner at all.

Cross-course references

• Linux for Production Sysadmins - Part LXXII (Secrets) covers how credential material is stored, permissioned and read on a single host, which is the ground truth any inventory has to reconcile against.
• Kubernetes for Production Sysadmins - Part LXXVI (Certs) covers the cluster PKI that kubeadm issues and renews, a set of credentials that is easy to omit from an estate register because no human ever requested any of them.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XCIII (CredRotation) covers rotation cadence for pipeline credentials, the operation that fails silently when the consumer list is incomplete.

Quiz