The renewal replaced the certificate and left the key behind

Reported symptoms

Three nodes, edge-01 to edge-03, terminate TLS for app.lab.example behind a load balancer. They share one certificate and one key, renewed on a ninety day cycle by a pipeline that has run without incident since the estate was built. edge-02 is the odd one out in exactly one respect that nobody has thought about for two months: it was rebuilt from an older base image in June, and its entry in the inventory was carried across by hand.

The renewal ran at 21:19 UTC on 26 August and reported three successes. Nothing happened for seven hours.

At 04:12 the following morning the unattended patch window restarted services on edge-02. The node left the pool and did not return.

• The TLS front end exits within a second of every start attempt. That rules out a slow start, a port conflict, and anything that waits on another unit.
• The certificate on the node is current, correct and issued the previous evening, with a notAfter in November. Expiry is eliminated in the first two minutes and the conversation turns to the kernel that was upgraded in the same window.
• Certificate monitoring is green across all three nodes. It reads the validity dates out of the file, and the file is faultless.
• edge-01 and edge-03 are serving the same certificate happily. Neither has been restarted since June.
• A configuration test passes on edge-01 and fails on edge-02, which locates the fault on one host without naming it.

The detail that reframes everything arrives eleven minutes later, when somebody finally lists the private key directory and finds two key files where there should be one.

Evidence provided

$ openssl x509 -in /etc/ssl/certs/app.lab.example.crt -noout -subject -issuer -serial -dates

subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

The certificate is not the problem, and running the same command on edge-01 returns the same serial. Whatever went wrong at 21:19 did not go wrong to the certificate.

$ CRT=/etc/ssl/certs/app.lab.example.crt
KEY=/etc/ssl/private/app.lab.example.key
openssl x509 -in "$CRT" -noout -pubkey | openssl sha256
sudo openssl pkey -in "$KEY" -pubout | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 1c4f0b7a9d2e6835ab41d907c5e8f2b06d3a7419fe58cc02b6d1e93a4f70852d

$ CRT=/etc/ssl/certs/app.lab.example.crt
KEY=/etc/ssl/private/app.lab.example.key
openssl x509 -in "$CRT" -noout -pubkey | openssl sha256
sudo openssl pkey -in "$KEY" -pubout | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Two digests that differ mean the key cannot produce the public key the certificate carries. Two that agree mean it can. Nothing else in this incident is as short or as decisive, and the second capture is what makes the first one trustworthy: the method is sound and only one node is affected.

$ sudo ls -l --time-style=long-iso /etc/ssl/private/

$ sudo openssl pkey -in /etc/ssl/private/app.key -pubout | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

That digest matches the certificate. The key the node needs has been sitting on the node all along, four directory entries away from where the service was looking for it.

$ grep -rn 'private_key_path' /etc/deploy/inventory/host_vars/

$ grep -A3 'reload' /var/log/deploy/renewal-2026-08-26T2119.log

$ echo | openssl s_client -connect edge-02.lab.example:443 -servername app.lab.example 2>/dev/null | openssl x509 -noout -serial -dates

serial=4E9C0A71B35D62F8C0148ADE7719B4C2903F51D6
notBefore=Jun 27 21:19:00 2026 GMT
notAfter=Sep 25 21:19:00 2026 GMT

Work the evidence before reading on

Nothing here is corrupt, expired or unauthorised. Every file is a valid file and every process did what it was told. The interesting question is why a defect created at 21:19 cost nothing until 04:12.

1. The certificate on edge-02 is the same certificate as on edge-01, down to the serial. What does that eliminate, and what does it leave standing?
2. The two digests on edge-02 differ and the two on edge-01 agree. Write down, in one sentence, what a digest taken from a certificate and a digest taken from a key file each represent, and why their equality is the only test that matters here.
3. There is a second key file on the host, dated the evening of the renewal, whose digest matches the certificate. What single misconfiguration explains both its existence and its location?
4. The archived probe from 03:00 shows edge-02 serving a June certificate while edge-01 served an August one. That divergence existed for seven hours. Which of the checks the estate already runs could have seen it, and why did none of them?

Before continuing: say what the running process on edge-02 was holding in memory between 21:19 and 04:12, and why nothing on disk could tell you.

Root cause

One node was told to put the key somewhere else

The renewal produces a matched pair and distributes both files. The certificate path is the same in every host entry. The private key path is not: edge-02 was rebuilt from an older base image in June, its inventory entry was transcribed by hand, and it kept a legacy value of /etc/ssl/private/app.key. The unit on that node, like the units on its siblings, reads the canonical path named after the service.

So at 21:19 the certificate was replaced at the path the service reads, and the key was written to a path the service has never heard of. What remained at the canonical path was the key from the June issuance. At the only location that matters, the certificate had moved on and the key had not.

That is the whole mechanical fault, and it is worth being precise about why it is fatal. A certificate contains a public key. A private key file contains, or can derive, the matching public key. Presenting a certificate obliges the server to prove it holds the corresponding private key. A server given a certificate and an unrelated key cannot make that proof, and every implementation worth using refuses the pair at load time rather than failing later in a handshake.

Nothing checked whether the reload worked

The pipeline issues a reload after writing the files and moves straight on. On edge-02 the reload failed, because the daemon was asked to adopt a pair it could not use, and it did the correct thing: it rejected the new configuration and carried on with the one already loaded. The exit status was never read. Three deployments were reported, two of which were real.

This is why the node kept working. The process had loaded a valid certificate and its matching key in June, and it continued serving them. Everything about that behaviour is defensible, and the combination of a daemon that refuses to break itself and a script that never asks how it went is precisely how a fault becomes latent.

Monitoring watched the file, not the wire

Certificate monitoring on this estate opens the file and reads the validity dates. After 21:19 the file on edge-02 was the August certificate, so the check reported November and stayed green, while the node went on serving a June certificate that expires on 25 September. For seven hours the three nodes disagreed about what they were presenting to clients and no instrument in the estate could express that. One probe per node, reading the serial off the wire, would have shown two values where there should be one at 21:24, in the change window, with a human still awake.

Resolution

1. Copy before you edit. Take a root only copy of both key files into a directory outside the deployment path, because one of them may be the only surviving copy of a key something else still depends on, and the other is the one you are about to promote. This is a minute of work that removes the possibility of turning one broken node into a reissuance.
2. Decide which key belongs to the certificate by digest, not by name, date or plausibility. Compare openssl x509 -pubkey piped through openssl sha256 against openssl pkey -pubout piped the same way, for each candidate key. Exactly one will match, and that one is the key regardless of what it is called or when it was written.
3. Install the matching key at the path the unit reads, with an explicit owner, group and mode rather than a copy followed by a chmod. Write to a temporary file in the same directory and rename it into place, so a partially written private key is never visible to anything that might read it.
4. Start the service and confirm it stays running rather than confirming it started. A unit that comes up and exits again within a second is telling you the pair is still wrong, and a second attempt will not change that.
5. Do not reissue. Requesting a new key and certificate at 04:20 needs the CA, needs approval, spends issuance budget, and forces a change to two healthy nodes because the new key will not match the certificate they already hold. Do not copy the key across from edge-01 either: it works, and it moves private key material through an interactive session, a shell history and a temporary file in exchange for nothing the local file does not already give you. The key you need is on the host.
6. Confirm the load balancer has taken the node back and that traffic is being served, before you touch the cause. Restoring capacity and correcting the inventory are two changes, and doing them together at 04:40 is how a second incident starts.
7. Correct the host variable so the private key path is derived from the certificate path rather than stored independently, then sweep the whole estate for the legacy value. Any host built from the same base image carries it, and its next renewal is the next occurrence of this incident.
8. Record the times: renewal at 21:19, reload failure at 21:19, restart at 04:12, diagnosis at 04:30. The seven hour gap between the defect and its consequence is the finding worth taking to the review, not the mismatch itself.

Verification

1. Take the certificate off the wire from a host outside the pool and read the serial that edge-02 is presenting. The files on disk have already told you everything they can; only the handshake proves the key matched, the process loaded it and the socket bound.
2. Confirm the serial is the August one rather than the June one. A node that starts successfully while still holding a stale certificate in memory is a distinct and equally invisible failure, and this is the observation that separates them.
3. Capture the same value from edge-01 and edge-03 and confirm all three now agree. Disagreement between peers was available as a signal for seven hours and nothing could see it, so proving it has closed is worth thirty seconds.
4. Read the unit main start timestamp and confirm it falls after the change. A node can hold a perfectly correct pair on disk and continue serving whatever it loaded in June, and the file listing looks identical in both cases.
5. Confirm the load balancer health check passes from the balancer point of view rather than from a request issued on the node itself. A node can answer perfectly on loopback and still be failing the check that decides whether it gets traffic.
6. Run the digest comparison on every host in the estate, not only the one that failed. The drift that caused this is invisible from edge-02, and any host carrying the same legacy variable is simply waiting for its next renewal.
7. Deliberately deploy a mismatched pair in staging and confirm the pipeline now refuses it. The correspondence gate is the control that makes all of the above unnecessary, and an untested gate is a comment.

Prevention

• Gate the deployment on correspondence. Two commands and one string comparison decide whether the certificate and the key belong together. A deployment that stops when they differ cannot produce this incident, and it costs less to write than the incident review did.
• Derive the key path from the certificate path. One variable with a suffix substitution cannot drift between hosts. Two independent variables in an inventory eventually will, and this pair drifted quietly for two months while everything appeared to work.
• Read the exit status of the reload, then prove the process re-read the files. Within 120 seconds of a deployment, compare the serial the process is serving against the serial just written. A reload that returns an error is worth catching; a reload that returns success and changes nothing is worth catching too.
• Probe each node individually, every 60 seconds, past the load balancer. A check that opens the file confirms a copy succeeded. A probe that completes a handshake confirms the service works, and it is the only one of the two that would have noticed anything on the evening of the 26th.
• Alert on disagreement between peers. If any node serves a different serial from the others for more than 5 minutes, page. That condition existed from 21:24 and would have cost one engineer ten minutes instead of costing the estate a third of its capacity during a patch window.
• Restart deliberately after a key change, inside the change window. A latent load failure discovered at 04:12 by unattended patching is the same defect that would have surfaced harmlessly at 21:20 in front of the person who caused it. Choosing when to find out is most of what change windows are for.
• Use the platform courses for the surrounding mechanics. Linux for Production Sysadmins - Part LXXIV (Drift) covers hosts that diverge after a rebuild, which is the fault underneath this one, and Part LXXI (TLS) covers the service side certificate and key configuration. Observability for Production Sysadmins - Part LXIV (TLSMonitoring) covers measuring the certificate a service actually serves rather than the file it happens to hold. Git, CI/CD & GitOps for Infrastructure Engineers - Part LVIII (DeployStrategies) covers checking the result of a deployment step instead of its issuance.