Discovering certificates across an estate

Discovery is the act of turning an estate into a list. There are four practical methods, they answer different questions, and the first thing to understand about them is that none is a superset of the others. Choosing between them is not the task. Understanding the specific shape of each one’s blind spot is, because that residue is what will page you.

What each method can see, and what it structurally cannot

flowchart LR
    A["Endpoint probing"] --> A1["Blind to: ports and networks you cannot reach, client certificates, material not yet deployed"]
    B["Host and image scanning"] --> B1["Blind to: appliances, sealed keystores, any host without access"]
    C["CA issuance records"] --> C1["Blind to: every other issuer, and to whether anything was deployed"]
    D["Certificate transparency"] --> D1["Blind to: all privately trusted issuance"]

Read the diagram as four partial views of one estate. Probing sees what is being served right now to a client standing where the prober stands. Filesystem and image scanning sees material at rest on machines you can log into. Issuance records see what a particular authority signed, whether or not anyone ever installed it. Transparency logs see publicly trusted issuance for your names, including issuance nobody in your organisation requested. Each answers a real question and refuses to answer the others.

Probing the served certificate

The probe is the only method that observes the thing a client actually receives, which makes it the ground truth for anything expiry-related.

HOST=api.example.com
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" </dev/null 2>/dev/null |
  openssl x509 -noout -subject -issuer -serial -dates -ext subjectAltName

A probe against a correctly configured server reports the whole chain it was sent, and the verification result of validating that chain from where the prober is standing:

Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
   i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Two facts arrive together and should be recorded separately. The leaf and its dates are properties of the certificate and are the same for everyone. The verification result is a property of the prober: its trust store, its clock, its network path. A probe from inside the estate can return a clean verification for a chain that fails from outside it, and the register needs to know which vantage point produced each observation.

Probing has hard structural limits. It reaches only ports it knows about, so a scan seeded from the public DNS zone will never see the internal database listener on a non-standard port. It cannot see client certificates at all, because those are presented by clients and only when a server asks for one. And it cannot see material that has been provisioned but not yet served, which is precisely the material that will surface at the next restart.

Reading the stores and the issuers

Scanning hosts finds material the network cannot show you: keys staged for a future cutover, certificates for services that are currently down, client certificates, and the bundles that determine what a host trusts rather than what it presents.

# Every certificate object in a bundle, not only the first one.
openssl storeutl -noout -certs /etc/ssl/certs/fullchain.pem

# Container formats that a naive extension match walks straight past.
find / -xdev -type f \( -name '*.jks' -o -name '*.p12' -o -name '*.pfx' \) 2>/dev/null

The second command matters more than it looks. A great deal of certificate material in a mature estate is not sitting in PEM files. It is inside Java keystores, PKCS#12 bundles and platform certificate stores, and a scanner that only globs for *.pem and *.crt reports a tidy, confidently wrong result. Worse, those container formats are usually password-protected, so the scanner can prove a keystore exists and often cannot say what is in it. An inventory entry that records the existence of a keystore whose contents are unknown is still worth having, because it is a truthful statement of what you know.

Client certificates are the clearest case for scanning hosts. A server probe cannot observe one, because a client certificate is presented by the client and only when a server asks for it, so an estate running mutual TLS between services has an entire class of credentials that network discovery is structurally unable to see. Those certificates expire on exactly the same terms as server certificates, and when they do the failure is a service that has stopped talking to another service rather than a browser warning anybody notices. The only routes to them are the host or image that carries the material and the CA that issued it.

Issuance records are the complement. An internal CA can produce an exact list of what it signed, with serials, subjects and validity windows, and no scan can match that for completeness within that one authority’s scope. The two things it cannot tell you are whether any given certificate was ever deployed, and anything at all about certificates signed by the other issuers in the estate. Every CA anyone stood up, every cloud provider’s managed issuer, every vendor appliance signing its own material: each keeps its own log, and the union has to be assembled deliberately.

Public logs, and the shadow estate they reveal

Certificate transparency is a set of append-only logs of issued certificates. Because mainstream browsers require evidence that a publicly trusted certificate was logged, public issuance for your domains is effectively recorded in public, and querying those logs for your own names is the one discovery method that does not depend on any access to your own infrastructure.

That property makes it uniquely good at one specific job: finding certificates your organisation did not know it had. A team that obtained a certificate for a marketing subdomain through a personal account, a supplier who issued one for a name under your zone, a long-forgotten staging host still holding a valid certificate. None of those appear in your CA’s records or on hosts you administer, and all of them appear in the logs.

Coverage as a measurement, not a claim

Combine all four methods and there is still a residue: hosts nobody can reach, appliances with no shell, keystores nobody has the password for, certificates on contractor laptops, material inside images that are built but not yet deployed. Discovery cannot be finished, only run again.

What can be done is to measure it. Every observation carries the method that produced it, so the register can report how many credentials were seen by more than one method, how many by exactly one, and how many are known only by declaration with no observation behind them at all. That last figure is the honest measure of how much of the estate is running on trust rather than evidence, and it is the number to put in front of a leadership team that has asked whether the inventory is complete. The answer is no. The useful answer is here is what fraction is evidenced, here is what is improving, and here is the segment nothing can currently see.

Production discipline

1. Record the vantage point with every probe result. A verification outcome is a statement about the prober as much as about the server, and results from inside and outside the estate are different observations of different things.
2. Seed probing from a name source, never from addresses alone. Name-based selection means an address scan reports the default virtual host and hides the rest of what that listener serves.
3. Scan for container formats, not just PEM extensions. Keystores and PKCS#12 bundles hold a large share of real estate material, and an extension-only walk produces a confident undercount.
4. Assemble issuance records from every issuer deliberately. Each authority is authoritative only for itself, so the list of issuers you query is itself an inventory that needs maintaining.
5. Publish coverage by method. Report what fraction of entries have at least one observation, and never describe the result of any single discovery run as the estate.

Cross-course references

• Observability for Production Sysadmins - Part XI (Blackbox) covers probing endpoints from a defined vantage point, the same mechanism this lesson uses for discovery rather than for alerting.
• Linux for Production Sysadmins - Part LXXI (TLS) covers the on-host certificate and trust store layout that a filesystem walk has to understand before its results mean anything.
• Kubernetes for Production Sysadmins - Part CXIV (TLS) covers where certificate material lives inside a cluster, which is a store that neither a host walk nor a network probe reaches.

Quiz