A support bundle uploaded twelve days ago contains the live server private key

Reported symptoms

The certificate for api.example.com comes from a public CA. An ACME client on web-01 obtains it, configuration management copies it to web-02 and web-03, and the whole arrangement has renewed itself on a forty-five day cycle since last November without anyone having to think about it. There is no story here. That is worth saying, because there is still no story at 08:52 UTC when this begins.

An engineer who joined three weeks ago opens support ticket 4471. She is not investigating anything. She wants to see how the reverse proxy is configured, somebody has attached a diagnostic archive taken from web-01, and unpacking it is quicker than asking for access to the host. What she finds is this:

• etc/ssl/private/api.example.com/privkey.pem, complete and unencrypted, inside an archive created on 14 August at 07:26 UTC.
• The archive was uploaded to a third-party vendor support portal fifteen minutes after it was made, during a latency investigation that eventually turned out to be a routing problem.
• The same archive is attached to the internal ticket, where anyone with a staff login has been able to download it ever since.
• Nothing is broken. Every client validates, every host is healthy, and not one monitoring signal has moved, because nothing about a copied key is visible from the inside.

She says so in the platform channel at 08:58 UTC. The incident is declared at 09:01.

Then comes the part that makes this scenario worth reading slowly. At 09:05 an engineer runs the renewal, because the obvious answer to a leaked certificate is a new certificate. It works. The tooling reports success, a certificate with a different serial and a later expiry lands on all three hosts, the reload is clean, and the channel calms down. Nothing in that sequence is false. The exposure at 09:20 is exactly what it was at 08:52.

Evidence provided

Begin with what was uploaded, because the scope of the incident is decided by the contents of one archive.

$ tar -tzf /srv/tickets/4471/web-01-diag-2026-08-14.tar.gz | grep -E 'ssl|nginx.conf'

etc/nginx/nginx.conf
etc/nginx/conf.d/api.example.com.conf
etc/ssl/certs/api.example.com/fullchain.pem
etc/ssl/private/api.example.com/privkey.pem

The certificate being in there is unremarkable; a certificate is a public document. The fourth line is the incident. The next question is whether it is the key that is live right now or a copy that has since been superseded, and there is a definitive answer to that.

$ openssl pkey -in /run/incident/from-bundle/privkey.pem -pubout | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Now the same value taken the other way round, from the certificate the server is presenting to the network at this moment.

$ echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null | openssl x509 -noout -pubkey | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Two identical digests, one from a private key sitting in a vendor support portal and one from a live production service. This pair is the whole diagnosis. A private key and a certificate belong to each other precisely when their public keys match, so the archived key is not a historical artefact, it is the operative key of the service, and it has been outside the organisation for twelve days.

How long that matters for depends on the certificate.

$ openssl x509 -in /etc/ssl/certs/api.example.com/fullchain.pem -noout -serial -dates

serial=03A7F41C6B8E250D9247B0FC5E1A6338D40B
notBefore=Aug  1 09:14:00 2026 GMT
notAfter=Sep 15 09:14:00 2026 GMT

And here is the reading taken at 09:20, after the renewal that everybody believed had resolved it.

$ openssl x509 -in /etc/ssl/certs/api.example.com/fullchain.pem -noout -serial -dates

serial=0491C3E8207B6D45F0A83E19C7526BB0D82F
notBefore=Aug 26 09:05:00 2026 GMT
notAfter=Oct 10 09:05:00 2026 GMT

Measured the other way, on exactly the same file, the picture is different.

$ openssl x509 -in /etc/ssl/certs/api.example.com/fullchain.pem -noout -pubkey | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

One more reading, because it decides how much of the response can lean on revocation.

$ openssl x509 -in /etc/ssl/certs/api.example.com/fullchain.pem -noout -ext authorityInfoAccess

X509v3 Authority Information Access:
  CA Issuers - URI:http://issuer.example.net/ca.crt

There is no OCSP responder URI in that extension. That is no longer unusual on the public web. Let’s Encrypt removed the OCSP URI from its certificates on 7 May 2025 and switched its responder off altogether on 6 August 2025, so a runbook step that runs an OCSP query against a certificate from that CA has nothing to talk to.

Work the evidence before reading on

Everything in this incident is working correctly, including the renewal, and that is what makes it a good scenario to sit with rather than skim.

1. Two digests, taken twenty minutes apart across a renewal, are identical while the serial number changes. State in one sentence what the renewal actually replaced.
2. The certificate had twenty days left when the key was found. In what sense is that number the exposure, and in what sense is it not?
3. The Certificate Transparency search came back clean and the access logs look ordinary. Name precisely what those two negative results rule out, and what they leave open.
4. The archive is on a vendor portal, on an internal ticket, and in whatever was downloaded from either. Which parts of the copy count can you establish, and which will you have to write down as unknown?

Before continuing: the key is compromised from the moment it was copied, not from the moment it is used. Say what follows from that about how much of the remediation can be justified by the absence of any sign of misuse.

Root cause

An ordinary tool could read the key

The diagnostic bundler archives the reverse proxy configuration directory and the files it references, which is a reasonable thing for a support bundle to do and is precisely how the key got in. The script runs as an identity that can read the key, because it walks the same tree the reverse proxy walks and the reverse proxy must be able to read it.

No rule was broken. The engineer at 07:26 on 14 August was doing exactly what the runbook says to do during a latency investigation: capture the state, attach it to the ticket, give the vendor something to work with. The upload happened fifteen minutes later under the pressure of an active problem, and nobody opened the archive to see what was in it, because nobody opens an archive to see what is in it.

The general shape is worth naming. Any automated collector whose scope is a directory will eventually collect something that was never meant to leave, and the moment it does, the material is wherever collectors send things.

Renewal preserved the key it should have replaced

The second defect is invisible until you measure it, and the evidence measures it twice. The ACME client is configured to reuse its existing key pair, so every renewal since 3 November 2025 has produced a new certificate over the same private key.

That single setting does three things, none of them good. It means the key in the 14 August archive is the key in production twelve days later. It means the 09:05 renewal produced a second certificate the holder of that key can present, rather than making the archived material worthless. And it means the key is not only in the archive: it is in every backup of web-01 taken since November, in every configuration snapshot, and in any earlier diagnostic bundle built by the same script.

This is the defect that made the incident durable. The first defect produced one bad copy on one afternoon. The second guaranteed that the copy stayed useful, that the obvious remediation would not touch it, and that there is no certificate lifetime anywhere in the system that bounds the exposure by itself.

Revocation is the reflex, and it is not the control

The instinct after a key leak is to revoke, and revoking is correct. It is also much weaker than most operators expect, and a response that quietly depends on it has a gap in it.

The evidence already shows one reason: the certificate carries no OCSP responder URI, because that CA no longer runs one. The wider reason is client behaviour. Chrome does not perform online revocation checks by default and disabled revocation checking for extended validation certificates in 2022. Firefox uses CRLite, a pushed dataset rather than a live query, and does not make OCSP requests for domain-validated certificates that chain to the Mozilla root store. The overall effect is that a certificate you have revoked will continue to be accepted by a large share of the clients on the internet until it expires.

So revoke, and record it, and mean it. Then plan as though it will not be honoured, because for public TLS that is the honest default.

Resolution

1. Record the moment the archive was created, 07:26 UTC on 14 August, as the start of the compromise. Not the moment of discovery, not the moment of upload. Every window you report and every decision about what else was exposed is measured from the copy, because that is when the key stopped being under your control.
2. Generate a new private key. This is the step the incident is actually about, and it is not a renewal, not a reissue and not a redeploy of the same material. Create a fresh key pair on the host that will use it, with the same algorithm and parameters as before unless you have a reason to change them.
3. Take the public key digest of the new key and compare it against the recorded digest of the compromised one before you request anything. Two different values is the gate for continuing. If they match you have reused the key, and everything after this point would be theatre.
4. Issue a certificate against the new key and deploy it to web-01, web-02 and web-03, reloading each service so the running process picks it up. Treat the reload as the cutover: until a process has re-read its key, that host is still serving the compromised one however correct the files on disk look.
5. Revoke the old certificate, and be clear with yourself about why. It is the correct entry to place on the record, it protects clients that do check, and in a private PKI where you operate and enforce a CRL it is a genuine control. On the public web it will not stop most clients accepting the old certificate, so it is a step you take in addition to fast replacement rather than instead of it.
6. Enumerate the copies and pursue each one. The vendor portal attachment, the internal ticket attachment, every earlier bundle produced by the same script, the backups of web-01 taken since November, and any image built from that configuration tree. Ask the vendor in writing to delete the attachment and to confirm the deletion with a date.
7. Ask what else was in the archive. The key is the headline and the archive is the incident: 214 configuration files went to the same two places, and anything credential-shaped among them joins its own rotation queue with its own owner.
8. Change the bundler before the next latency investigation, not in a follow-up ticket. Give it an explicit list of files to collect rather than a directory to walk, and run it as an identity that cannot read the key directory at all.
9. Turn off key reuse in the ACME client configuration so the next renewal produces a new key pair, and add a deployment check that compares the new certificate public key digest against the previous one and fails when they match.
10. Write the exposure window down as a single number, from 07:26 UTC on 14 August to the moment the new key went live, and put it at the top of the report. Twelve days is the finding. Everything else is detail.

Verification

1. Read the public key digest from the socket rather than from disk, from a host outside the deployment path. The server is the only authority on what it is presenting; a file that is correct on disk and has not been re-read by the process looks identical to a finished deployment in every other check.
2. Repeat that reading against web-01, web-02 and web-03 individually rather than through the load balancer. Three digests, all different from the recorded compromised value, is the result. Two out of three is an unfinished rollout, and the load balancer will happily hide the third.
3. Confirm the certificate now in service carries the expected new serial and the expected validity window, so that the record of what is deployed matches the record of what was issued.
4. Scan the estate for the old serial and require a count of zero. Include any host that terminates TLS for this name, including ones the change plan did not mention, because the copy of the key was never limited to the hosts on the list.
5. Confirm the revocation was accepted by the CA and record the timestamp. Then record, in the same place, that you cannot demonstrate any particular client honours it, so nobody later reads the revocation entry as proof of containment.
6. Run the corrected diagnostic bundler and list the archive it produces. The key file must be absent. A procedure change nobody has executed is a document, and this one is easy to execute.
7. Search Certificate Transparency for the domain again after the replacement, and set up the monitoring that repeats the search continuously. The check that matters is not the one you ran during the incident, it is the one that will run next month.
8. State in the report what could not be established: whether the key was ever used, how many copies exist, and who holds them. An incident report that omits its own unknowns invites the reader to assume there were none.

Prevention

• Generate a new key on every issuance, by default. Key reuse is usually a configuration option that somebody enabled for a reason that made sense at the time, and it silently converts a ninety-second operation into a permanent asset. A fresh key pair costs milliseconds and caps the age of the thing you least want copied.
• Gate the deployment on the digest. Compare the public key digest of each newly issued certificate against the previous one and fail the deployment when they match. This is three lines of script and it is the difference between a rotation and a renewal being indistinguishable from the outside.
• Give collectors an allow list, never a directory. Diagnostic bundlers, log shippers, configuration backups and support tools all walk trees, and a tree that contains a key will eventually ship one. Enumerate the files to collect, run the collector as an identity with no read access to the key directory, and test that it refuses.
• Hold keys where they cannot be copied. A key generated inside an HSM or a PKCS#11 token, marked non-exportable, cannot be in a support archive at all. That converts this entire class of incident from something you respond to into something that cannot be expressed.
• Keep validity short and renewal automatic. The remaining life of a leaked certificate is the one part of the exposure you can bound in advance, and it is bounded by policy rather than by luck. Forty-five day certificates renewed on schedule cap it at weeks; an annual certificate caps it at nothing you would want to put in a report.
• Monitor Certificate Transparency for every name you own. Page within 15 minutes on any issuance nobody requested. It is the only external channel that will tell you a key of yours is being used by someone else, and it costs nothing to run.
• Measure the replacement, then rehearse it. Time a full key and certificate replacement across the fleet, hold it under 60 minutes, and repeat the drill quarterly. That measured number, not the revocation entry, is what will actually be protecting you the next time an archive goes somewhere it should not.