The issuing CA expires in twenty-three days and nothing is alerting

Reported symptoms

The internal PKI has been boring for five years. One offline root, one issuing CA on ca-1, and 412 service certificates with ninety-day validity, renewed nightly by a job that has reported success every night for fourteen months. There is an expiry dashboard. It is green.

At 09:12 UTC an engineer opens a chain for a completely unrelated reason. A colleague has asked why a new service will not connect to the internal message broker, and the quickest way to answer is to print the whole chain and look at it. The leaf is fine. The broker is fine. What is not fine is the middle certificate, and it takes a second read to register why:

• notAfter=Sep 18 09:00:00 2026 GMT on the issuing CA. That is twenty-three days away.
• The dashboard, refreshed in the next tab, reports 412 certificates and a fleet minimum of 38 days remaining. No warnings. It has never had a warning.
• Nothing is broken. Every service is serving TLS, every client is validating, and the nightly renewal job ran clean at 02:00.
• The internet-facing edge is unaffected. Those certificates come from a public CA through separate automation, which is exactly why nobody thinks of the internal hierarchy as a thing that has an expiry date.

And one detail that has been sitting in the ticket queue for three days. A staging host was left with its clock deliberately advanced for a rehearsal, and since then it has refused every internal TLS connection. The ticket was closed as expected behaviour for that host. It was not expected behaviour. It was a preview.

Evidence provided

Start with the certificate everyone has been ignoring.

$ openssl x509 -in /etc/pki/ca/issuing-ca-g1.crt -noout -subject -issuer -serial -dates

subject=O=Example Internal, CN=Internal Issuing CA G1
issuer=O=Example Internal, CN=Internal Root CA G1
serial=5A3C71E0F94B2D8617C0AE35B98F41D2
notBefore=Sep 18 09:00:00 2021 GMT
notAfter=Sep 18 09:00:00 2026 GMT

Now the same reading for a certificate it signed, taken from the service inventory at random.

$ openssl x509 -in /etc/pki/services/payments-api/cert.pem -noout -subject -issuer -serial -dates

subject=CN=payments-api.internal.example.com
issuer=O=Example Internal, CN=Internal Issuing CA G1
serial=3B8F27C0D14E6A95F072B83DC4610E7A29D5B146
notBefore=Aug 12 06:41:00 2026 GMT
notAfter=Nov 10 06:41:00 2026 GMT

Read those two blocks together and the arithmetic is uncomfortable. This leaf was issued on 12 August 2026 with a validity window that ends on 10 November, which is fifty-three days after its own issuer stops being valid. Nothing refused to sign it.

The monitoring primitive the dashboard is built on tells the same story twice, and gets it right both times.

$ openssl x509 -in /etc/pki/ca/issuing-ca-g1.crt -noout -checkend 2592000; echo exit=$?

Certificate will expire
exit=1

$ openssl x509 -in /etc/pki/services/payments-api/cert.pem -noout -checkend 2592000; echo exit=$?

Certificate will not expire
exit=0

openssl x509 -checkend is the primitive the whole dashboard is built on, and both readings above are correct. One certificate answers green and one answers red, and only the green one is on the dashboard. That is the incident, expressed in two commands.

Two more readings rule out the hypotheses the team reached for first. The chain is not already broken:

$ openssl verify -CAfile /etc/pki/ca/root-g1.crt -untrusted /etc/pki/ca/issuing-ca-g1.crt /etc/pki/services/payments-api/cert.pem

/etc/pki/services/payments-api/cert.pem: OK

And the anchor is not the problem:

$ openssl x509 -in /etc/pki/ca/root-g1.crt -noout -subject -dates

subject=O=Example Internal, CN=Internal Root CA G1
notBefore=Sep 15 08:40:00 2021 GMT
notAfter=Sep 15 08:40:00 2031 GMT

Finally, the scale. Walking every certificate in the inventory and matching its authority key identifier against the issuing CA subject key identifier gives the blast radius as a number.

$ openssl x509 -in /etc/pki/ca/issuing-ca-g1.crt -noout -ext subjectKeyIdentifier

X509v3 Subject Key Identifier:
  9E:41:C7:0B:2D:58:A3:6F:14:E0:8B:97:D2:35:6A:C1:40:F8:B3:2E

412 of the 412 certificates in the inventory carry that value. There is no partial exposure here and no subset to prioritise.

Work the evidence before reading on

Nothing has failed. That is the interesting part, and it is the part that makes this scenario worth sitting with rather than skimming.

1. The dashboard reports a fleet minimum of 38 days and the issuing CA has 23. Both numbers were measured correctly. What question is the dashboard answering, and what question did the team believe it was answering?
2. A leaf issued on 12 August carries a notAfter fifty-three days past its issuer notAfter. Which component should have refused that request, and what would it have had to compare?
3. On 18 September at 09:00 UTC, what changes for a leaf whose own notAfter is 10 November? Describe it in terms of what a client does during path validation, not in terms of what the dashboard shows.
4. A staging host with an advanced clock has been failing internal TLS for three days and the ticket was closed. What was that host actually measuring, and why is a wrong clock a time machine here?

Before continuing: the estate has 412 monitored certificates and one unmonitored one. State the property that made the unmonitored certificate invisible to a monitoring system that was otherwise working correctly.

Root cause

The inventory contains endpoints, and a CA is not an endpoint

The expiry check works by iterating a list of service names, resolving each one, connecting, and reading the certificate at the end of the chain. Every part of that is defensible. It is the right shape for a service inventory, it catches the failure mode teams meet most often, and it produced 412 correct answers every five minutes for fourteen months.

It also structurally cannot see a certification authority. An issuing CA has no listener. It never appears at position zero in a chain. Nothing in the estate resolves its name. It exists as a file on ca-1 and as a copy stapled into every server chain, and neither of those is something the collector was built to enumerate. The dashboard was not misconfigured and no threshold was set wrongly. The set being measured simply did not contain the object that mattered.

This is the defect that turned a scheduling problem into an incident. A rotation date missed by three weeks is recoverable with a calendar entry. A rotation date discovered by accident, twenty-three days out, by someone who was looking at something else, is a different situation, and the only reason it is not a worse one is that the engineer read the whole chain instead of just the leaf.

Nothing compared the leaf notAfter with the signer notAfter

The second defect is quieter and it is visible directly in the evidence. The signing automation takes a certificate signing request, applies a profile with ninety days of validity, and signs. It never asks how long the signer itself has left.

So on 12 August the automation happily produced a certificate valid until 10 November under a CA valid until 18 September, and it will keep doing that every night until the moment it cannot. Worse, from about 20 June onwards every certificate this CA issued has carried a validity window it cannot honour, and each one of those issuances was a free opportunity to notice. Ninety days before an intermediate expires, the intermediate silently becomes the binding constraint on every certificate beneath it, and no part of this system observed that boundary being crossed.

Expiry propagates down the chain, not up it

The reason the fleet fails simultaneously rather than gradually is worth stating precisely, because the intuition runs the other way. Certificates are usually thought of as failing individually and on their own schedule, which is why a ninety-day estate feels staggered and safe.

Path validation does not work that way. A client building a chain from the leaf to the anchor checks the validity period of every certificate it traverses, using the same current time for all of them. A leaf with two months left, sitting under an issuer that expired an hour ago, produces a chain that cannot be validated. The leaf dates are irrelevant to the outcome. At 09:00 UTC on 18 September, all 412 services stop being trustable inside the same second, and no amount of leaf renewal fixes any of them.

The staging host was showing the answer

The host with the advanced clock was not exhibiting a quirk. It was performing the experiment. A client evaluating a chain against a clock set past 18 September rejects that chain, which is exactly what will happen everywhere on the real date. The ticket recorded the symptom accurately and drew the wrong conclusion from it, because the estate had no reason to associate a clock complaint with a CA that was not on any dashboard.

Resolution

1. Confirm the root is usable before planning anything else. Locate the root key material, confirm the operators holding its shares are available inside the next twenty-three days, and book the signing ceremony now. Every option below requires a root signature, and discovering a key-custody problem in week three would leave no time to solve it.
2. Freeze the leaf validity profile at its current ninety days and do not extend it. A longer leaf under an expiring issuer increases the number of certificates that must be reissued after the rotation without moving the deadline. This is a one-line change to hold and it prevents the fleet drifting further out of alignment while the plan is written.
3. Choose the rotation shape deliberately, and record the reason. Re-issuing the existing intermediate with the same subject distinguished name, the same public key, a new serial and a longer validity keeps every one of the 412 existing leaves valid, because chain building matches on issuer name and key identifier and the leaf signatures were made by that key. Generating a new key and a new intermediate does not, and obliges a full reissue before 18 September.
4. Take the re-issue path only if the key passes its own review. The key must be sound, held where your policy says CA keys live, and not already overdue for replacement. If any of those fails, this rotation is the moment to fix it, because you will not get a second scheduled opportunity for five years.
5. Sign the new intermediate with a validity that exceeds your longest leaf validity by a wide margin, and record its rotation date in the same change. A CA created without a rotation date in the calendar is the exact defect being repaired.
6. Distribute the new intermediate to every service that serves a chain. Servers present the intermediate; clients do not fetch it. Until the new intermediate is deployed to a service, that service still presents the old one and still carries the old deadline, so treat the deployment as the real cutover rather than the signing.
7. Roll the deployment by mechanism, not by service. Configuration-managed hosts, container images, the ingress tier and the appliance fleet each have a different path for a chain file and a different reload story. Prove one service per mechanism from the socket before proceeding to the rest of that mechanism.
8. Add the CA certificates to the monitoring inventory in the same change, not in a follow-up ticket. The rotation removes the deadline; the monitoring rule is what stops the next one arriving unannounced in 2031.
9. Reopen the staging clock ticket and close it against this incident, so that the next engineer searching for an internal TLS failure on a fast-clocked host finds the cause rather than a fourth closure.

Verification

1. Read the chain from the socket, not from the file you deployed. For one service per deployment mechanism, connect from a client host and print the certificates the server actually presents. The intermediate must carry the new serial and the new notAfter. A correct file on disk that the process has not reloaded looks identical to a working deployment in every check except this one.
2. Confirm the handshake validates end to end from a client that trusts only the root. The verify return code must be 0, which proves the client built a complete path to the anchor using the intermediate the server supplied.
3. Ask the new intermediate the question that started the incident. Run openssl x509 -noout -checkend against it with a horizon longer than your longest leaf validity; it must report that the certificate will not expire. Run the same check against every other CA certificate in the estate while you are there.
4. Prove the old intermediate is gone from the serving path. Scan the estate and count services still presenting the old serial. That count must be zero before the rotation is called complete, and it is the only number that distinguishes a finished rollout from a mostly-finished one.
5. Verify the monitoring change from the rendered dashboard rather than the configuration file. Each CA certificate must appear with a days-remaining figure. A configuration file the collector has not reloaded is a change that has not happened.
6. Prove the new rule can fire. Point it at a deliberately short-dated CA in staging and confirm it warns at the warning threshold and pages at the paging threshold, to the rota that will actually be carrying the pager in 2031.
7. Re-run the issuance guard against a request that would produce a leaf outliving its signer, and confirm the request is rejected rather than signed. A guard that has never rejected anything has never been exercised.

Prevention

• Monitor certification authorities as objects, with their own thresholds. A leaf with thirty days left is routine. An issuing CA with thirty days left is a fleet-wide deadline. Warn at 25 per cent of the original validity remaining, page at 90 days, escalate to the named platform owner at 30.
• Build the inventory from what is served, not from a list. Scan listening sockets, walk each presented chain to its anchor, and record every certificate encountered on the way. A hand-maintained list can only contain what someone remembered to add, and the certificate nobody remembered is the one that hurts.
• Refuse to sign a leaf that would outlive its signer. The check is a single comparison at issuance time: reject when the requested notAfter falls after the signer notAfter minus one full leaf validity. It converts a silent policy gap into a failed request with three months of warning.
• Record a rotation date when the CA is created, in the same change. Not in a wiki page, in the calendar the team already reads. Five years is longer than the average tenure of the person who ran the ceremony.
• Rehearse an intermediate rotation annually in a lab hierarchy. The value is not the practice, it is the number: after the rehearsal you know how long the rotation takes, which turns the paging threshold from a guess into a measurement.
• Write the arithmetic down where the profile lives. Ninety days of leaf validity under a five-year intermediate means the intermediate becomes the binding constraint for the last ninety days of its life. Anyone changing either number should see that sentence.