A client certificate nobody issued was accepted by the payments service

Reported symptoms

The internal hierarchy is a two-tier arrangement that has been running since 2023: an offline root, an issuing CA on ca-1, and 2,411 certificates underneath it. About four hundred of those are client certificates used for mutual TLS between internal services. The payments service is one of the consumers, and it keeps a monthly list of the client identities that have presented a certificate to it, because someone asked for that report during an audit two years ago and nobody ever turned it off.

At 07:48 UTC an engineer reads the August list. There are eighty-one identities on it. Eighty of them are services she recognises. The eighty-first is deploy-agent-07.

• There is no deploy-agent-07. The naming convention for deploy agents was retired in 2024 and the highest number ever allocated was four.
• The certificate validated. It was accepted, the request was served, and no error, warning or alert was generated at any point, because from the payments service point of view nothing unusual happened.
• Its serial number is not in the CA database. That file records every certificate the CA software has issued since the hierarchy was built.
• Its notBefore is 19 August, which lands inside a window an unrelated intrusion investigation has already flagged: root access on ca-1 that the team does not own.

The last bullet is the one that changes the shape of the day, and it takes about ten minutes for someone to say the thing out loud. This is not a rogue certificate that needs revoking. If the intruder had root on ca-1, they had the CA signing key, and a certification authority key does not produce one certificate. It produces as many as its holder wants, at any time, for any name, with no record anywhere that you control.

The incident is declared at 08:04 UTC as a certification authority compromise. Nothing in the estate is failing, and nothing will fail on its own.

Evidence provided

Start with the certificate itself, because everything else is a consequence of what it proves.

$ openssl x509 -in /run/incident/deploy-agent-07.pem -noout -subject -issuer -serial -dates

subject=CN=deploy-agent-07
issuer=O=Example Internal, CN=Example Internal Issuing CA G2
serial=6C08B1F9A32D74E05B96C1470FA82D3E81
notBefore=Aug 19 02:51:00 2026 GMT
notAfter=Nov 17 02:51:00 2026 GMT

Now the same certificate measured against the record that is supposed to know about it.

$ grep -c '6C08B1F9A32D74E05B96C1470FA82D3E81' /etc/pki/ca/db/index.txt; wc -l /etc/pki/ca/db/index.txt

0
2411 /etc/pki/ca/db/index.txt

That pair is the diagnosis, and it is worth being explicit about what it means. The CA database is written by the CA software when the CA software signs something. A certificate signed by the same key, outside that software, on a different machine, at a time of the holder choosing, appears nowhere in it. So the absence of a row is not an anomaly to investigate. It is the expected result when somebody else has the key.

The next reading rules out the comfortable hypothesis that this is a malformed or somehow marginal certificate.

$ openssl verify -CAfile /etc/pki/ca/root-g1.crt -untrusted /etc/pki/ca/issuing-ca-g2.crt /run/incident/deploy-agent-07.pem

/run/incident/deploy-agent-07.pem: OK

It is a completely valid certificate. There is nothing wrong with it and nothing to detect. It differs from the 2,411 certificates you did issue in exactly one respect, which is that you did not ask for it, and that fact is not encoded anywhere in the certificate.

The authority to make more of them is stated plainly in the intermediate.

$ openssl x509 -in /etc/pki/ca/issuing-ca-g2.crt -noout -ext basicConstraints,keyUsage

X509v3 Basic Constraints: critical
  CA:TRUE, pathlen:0
X509v3 Key Usage: critical
  Certificate Sign, CRL Sign

The blast radius is the set of things that depend on this intermediate, and that is a number the inventory can produce.

$ for f in /etc/pki/inventory/*.pem; do openssl x509 -in "$f" -noout -issuer; done | sort | uniq -c

   2411 issuer=O=Example Internal, CN=Example Internal Issuing CA G2

Then the reading that decides whether revocation is available as a lever at all.

$ openssl x509 -in /etc/pki/services/payments-api/cert.pem -noout -text | grep -c 'X509v3 CRL Distribution Points'

0

$ openssl x509 -in /etc/ssl/certs/edge-public.pem -noout -text | grep -c 'X509v3 CRL Distribution Points'

1

The internal certificates were issued without a distribution point for eight years, so there is no URI in them, no published list, and no client that has ever been configured to fetch one. The public certificate on the edge has a distribution point because a public CA put it there, which is a useful reminder that this was always a choice somebody made rather than a property of X.509.

Work the evidence before reading on

The interesting thing about this incident is how little of it is a detection problem and how much of it is an accounting problem.

1. The rogue certificate verifies successfully against your production trust anchor. Name every property of it that distinguishes it from a certificate you issued, and then say which of those properties a client can check.
2. The CA database contains 2,411 rows and none of them is this certificate. What is that file actually a record of, and what would a record that answered the question you want look like?
3. Forensics place unauthorised root on ca-1 from 5 August. The rogue certificate is dated 19 August. Which date should the scope statement use, and why is it neither of those two?
4. The internal leaves carry no CRL distribution point. Work through what that removes from the response, and decide what is left.

Before continuing: name the smallest set of certificates you can honestly describe as trustworthy at 08:04 UTC, and justify the boundary you drew.

Root cause

The signing key was online, on a host, in a copyable form

The CA runs as an ordinary service on ca-1. Its private key is a file on that host disk, encrypted with a passphrase, and the passphrase is fetched from the secret manager when the service starts so that the machine can restart unattended. Both halves of that arrangement were deliberate and both were documented.

The consequence is arithmetic rather than subtle. Root on ca-1 yields the key file directly and the passphrase either from the running process memory or by asking the secret manager with the credentials that are on the host for exactly that purpose. There is no additional obstacle between an intruder with root and the ability to sign. Nineteen days is a long time to have that.

ca-1 was also a general-purpose host in every way that matters: it ran a package manager, a monitoring agent, a configuration management client and a log shipper, each of which is an update path and an attack surface. A machine whose only job is to hold a signing key has a much smaller list, and a signing key inside a token that will not export it has a list of one.

The issuance record describes your CA, not your namespace

This is the property that makes a CA compromise different in kind from a leaked server key, and it is worth being blunt about.

When a server key leaks, you know exactly what is exposed: one certificate, one name, one service, and a validity window you can read. The quantity is one and you can hold it in your head.

When a CA key leaks, the exposed set is every certificate the holder chooses to create, for any name in your namespace, dated however they like, at any point from the moment they took the key until you stop trusting the intermediate. Your CA database bounds what your software issued. Nothing bounds what their software issued, and there is no observation you can make of a certificate that reveals which of the two it was. Serial numbers are chosen by the signer, so they can duplicate yours or avoid them.

That is why the certificate found on the payments service is not the incident. It is a single accidental sighting that tells you the incident exists, and it tells you nothing whatsoever about its size.

Nothing in the estate could stop trusting one intermediate

The third property is what turns the response from a repair into a programme. The obvious containment for a compromised intermediate is to distrust that intermediate while keeping the root, which is much cheaper because the trust anchor does not change and no client needs anything new.

It requires a working revocation path, and there is not one. The internal certificates carry no CRL distribution point, so a client has no URI to fetch. No CRL has ever been published. No service in the estate has been configured to require one, and several of the runtimes involved would need explicit configuration to enforce it even if a list existed. Publishing a CRL today and announcing that the intermediate is revoked would change the behaviour of exactly nothing.

That leaves one mechanism that genuinely removes trust: change the trust anchor. Every trust store in the estate must gain a new root and lose the old one, which is the expensive, slow, coverage-driven work that makes this a multi-week transition rather than an afternoon.

Resolution

1. Stop issuance immediately by disabling the CA service, and say plainly in the channel what that does and does not achieve. It stops your own population growing and freezes the record, which matters for the investigation. It does not inconvenience whoever holds the key, because they have never needed your server to sign anything.
2. Preserve before you rebuild. Take a disk image of ca-1, copy the CA database, the service logs, the host logs and the key file itself, record a hash of each artefact, and hand the set to whoever owns the investigation. The key file is evidence: it establishes what was available to be taken. Do not reinstall the host while it is the only copy of that.
3. Fix the compromise window and then widen it deliberately. Forensics place unauthorised root from 02:14 UTC on 5 August; the scope statement should use the earliest moment access was possible rather than the earliest moment it was observed, and it should say which of the two each date is.
4. Decide whether the root is also in scope, and answer it with evidence rather than hope. If the root key was ever online, or its passphrase was reachable from ca-1, or its custody records have gaps, treat it as compromised and replace the anchor. Here the root is offline in a safe with signed custody records, so it is sound, and that materially reduces the work that follows.
5. Determine whether you can enforce revocation before choosing a strategy, because that single question decides everything after it. A sound root plus enforceable revocation means a new intermediate under the same root and no change to any trust store. No enforceable revocation means the anchor has to change. In this estate the leaves carry no distribution point and no client checks, so the anchor changes.
6. Build the replacement hierarchy on hardware that does nothing else, with the new keys generated inside an HSM or a PKCS#11 token and marked non-exportable, and with an independent append-only issuance record configured before the first certificate is signed. Do not restore any part of the old CA into the new one.
7. Distribute the new trust anchor to every trust store in the estate before anything serves a chain that requires it. Both anchors present, old chain still served, is the state you want during distribution. Ordering this the other way round is its own incident and is the reason this step is written before the reissuance step rather than beside it.
8. Reissue the 2411 certificates from the new hierarchy, grouped by deployment mechanism rather than by service, and prove one of each mechanism from the socket before rolling the rest of that mechanism.
9. Remove the old anchor and distrust the old intermediate only when the last consumer holds the new one and the last service serves the new chain. This is the step that actually ends the compromise, and it is last for a reason: it is the only irreversible one, and doing it early converts a controlled transition into an outage.
10. Rotate everything else ca-1 could reach while you are here: the secret manager credentials held on that host, the backup encryption key, the monitoring agent token, and any credential in the configuration management tree for that machine. A host held for nineteen days did not only give up the CA key.
11. Communicate on a schedule, starting with a written statement of the realistic elapsed time. In this estate the plan runs from the incident on 26 August to removal of the old anchor on 16 October, roughly seven weeks, with trust distribution taking the largest single block. Publish that up front. A plan that quietly slips from a promised fortnight loses more trust than an honest seven weeks ever costs.

Verification

1. Prove the new hierarchy works from a client whose trust store contains only the new root. Connect to a sample of services, read the chain each one presents, and confirm the handshake validates and terminates at the new anchor. This is the easy half of the verification and it is not the half that matters.
2. Prove the old hierarchy no longer works, which is the half that matters. Present the recovered rogue certificate to a production client with the production trust store and confirm it is refused. Then do the same with a legitimate certificate that the old CA issued, and confirm that it is refused too, because a distrust that only rejects the certificate you already knew about is not a distrust.
3. Run openssl verify against the rogue certificate with the new root as the only CAfile and confirm it fails to find an issuer. Record the exact output in the report, because this is the single artefact that demonstrates containment.
4. Sample every trust store class separately rather than averaging over hosts. The operating system store, the JVM store, the container base image, the network appliances and the developer laptops are five different distribution problems with five different failure modes, and a percentage across all of them tells you nothing about any of them.
5. Scan the estate for any chain still terminating at the old anchor and require a count of zero. Include hosts nobody put on the transition list, because the inventory that missed a certificate is the inventory you are relying on.
6. Confirm the new CA key is genuinely non-exportable by attempting an export and recording the refusal. A property nobody has tested is a property from a datasheet.
7. Verify the control that was missing. Issue a certificate from the new CA, confirm it appears in the independent issuance record within the expected window, then present a certificate that is not in that record and confirm the detection fires to the rota that will be carrying the pager next quarter.
8. Confirm the rotated credentials from ca-1 are actually in use rather than merely reissued, by checking each consuming system rather than by asking its owner.

Prevention

• Keep the signing key where a root shell cannot reach it. An HSM or PKCS#11 token that generates the key internally and will not export it changes the worst case from unbounded to countable, because the device can tell you how many signatures it produced and you can compare that number with your records.
• Keep an independent, append-only issuance record. Store it somewhere the CA host cannot write to, and monitor the estate against it rather than trusting it passively. Alert within 15 minutes on any certificate observed in service whose serial is absent from the record. That one rule would have surfaced this on 19 August rather than 26 August.
• Issue every certificate with a distribution point, and make something enforce it. Publish the list, configure the clients you control to fetch and hard-fail, and then measure the proportion that genuinely does. Report that proportion. A revocation path you cannot measure is a revocation path you do not have, and its absence is what makes an incident like this take seven weeks.
• Constrain intermediates by name. An intermediate limited to part of your namespace cannot sign outside it, which bounds a compromise by policy instead of by trust. Confirm the runtimes you depend on honour the constraint before you count it as a control.
• Separate the roles and record the custody. Root offline, a quorum required to use it, two named people required to issue an intermediate, and access to any CA host reviewed weekly rather than annually. Nineteen days of unnoticed root access is itself a finding independent of what it was used for.
• Rehearse the anchor replacement annually and write down the clock. Run the whole transition in a lab hierarchy once a year and record how long each phase took. That measured figure is what you will be asked for in the first hour of a real compromise, and the alternative is guessing in front of people who will hold you to the guess.