Skip to main content
RunBook Academy

← All break/fix scenarios in Secrets, PKI & Certificates

intermediatepki-key-material~40 min

The service would not start, and the fix made the key public

Reported symptoms

  • ●At 22:52 UTC customers report that app.lab.example is refusing connections, twelve minutes after an automated certificate renewal deployment that the pipeline recorded as successful
  • ●The unit is in a failed state and every restart attempt fails again within a second, so this is not a slow start or a resource problem
  • ●The journal entry that precedes the failure names the private key path and comes from the service TLS setup rather than from systemd, which several people initially read as a corrupt key file
  • ●The certificate deployed alongside the key is current, correct and issued that same evening, so nothing about expiry or the wrong certificate explains it
  • ●A sibling service on the same host, deployed by the same pipeline eight months earlier, is running normally with its own certificate and key
  • ●At 22:58 an engineer runs chmod 644 on the key to rule permissions out, the service starts immediately, and the incident is closed at 23:05 with the cause recorded as a permissions issue
  • ●At 09:15 the following morning a routine host scan reports a world readable private key under /etc/ssl/private, on a host carrying fourteen interactive accounts and a shared build agent

Evidence

  • · openssl x509 against the deployed certificate shows the subject, the issuing CA, a full length serial and a notBefore of that same evening, so the certificate itself is current and correct
  • · The public key digest taken from the key file and the public key digest taken from the certificate are identical, which proves the pair corresponds and rules out a mismatched deployment
  • · systemctl status reports the unit in a failed state with a result of exit-code and a main process that lived for well under a second
  • · The journal entry immediately before the exit names the private key path and reports that the process could not open it, and the message is emitted by the service TLS setup rather than by systemd
  • · ls -l on the key shows read and write for the owner only, with the owner and group both root, while the unit runs as the unprivileged app user
  • · systemctl show reports User and Group of app for the failing unit, so the process that must open the key is not the user that owns it
  • · A read test performed as the app user against the key path fails, and the same test against the certificate path succeeds, which locates the failure precisely at the key
  • · stat across the directory shows the sibling service key at mode 0400 owned by its own service account, deployed eight months earlier by a different mechanism
  • · After the chmod, a find for files under the private key directory carrying any read bit for group or other returns the key, which is the finding the morning scan raised
  • · The deploy script copies both files with cp and does not check the exit status of the restart it issues afterwards, which is why the pipeline reported success against a service that never came up
Diagnosis and resolutionclick to reveal

Root cause

Two defects, and the second one is far more expensive than the first. The first is an ordinary deployment defect. The renewal script copies the certificate and the key into place with cp, which preserves nothing useful: the resulting owner is whoever ran the script, and the resulting mode comes from that process umask. The key therefore arrived owned by root with read and write for root only. The service runs as an unprivileged account, so it could not open the file, and it exited during TLS setup before it ever bound a socket. The sibling service on the same host was unaffected because its key had been placed eight months earlier with an explicit mode and owner, by a person rather than by this script. Compounding it, the deploy script issues a restart and never checks whether it worked, so the pipeline reported a successful deployment against a service that was down. The second defect is the response. Making a private key readable by everyone does start the service, because the process can now open the file, and it does so by removing the only property that made the key worth protecting. This host carries fourteen interactive accounts and a shared build agent, so for ten hours and seventeen minutes any of them could have read the key that authenticates the service. The incident was closed at 23:05 as a permissions issue, which was accurate and was also the moment an availability incident quietly became a key exposure requiring rotation.

Remediation

Decide the exposure question before restoring anything, because it determines whether this ends with a chmod or with a new key. Take the exact window from the file modification time, the journal and the shell history, then establish who could have read the file during it. On a host with other interactive accounts, a shared build agent, a backup that runs as root, or any process that archives configuration, assume the key left the host. A private key that could have been copied is a rotated key, and no amount of re-permissioning changes that. Restore correct ownership and mode immediately regardless, because that closes the window even if rotation follows. Use install with an explicit owner, group and mode rather than chmod after the fact, and confirm the directory itself does not grant access more broadly than the file does. Then rotate properly if exposure is credible. Generate a new key, issue a new certificate for it, deploy both, restart, and only then revoke the old certificate. Be honest about what revocation buys you: in an internal PKI it works only where a CRL is actually distributed and checked, and for a publicly trusted certificate mainstream clients soft fail or do not check at all, so the effective control is a short lifetime and fast reissuance rather than the revocation itself. Fix the deployment path in the same change. Replace cp with install carrying explicit owner and mode, write to a temporary file in the same directory and rename it into place so a partial copy is never visible, and check the exit status of the restart so a failed service can never be reported as a successful deployment. Never make a private key readable by everyone to get a service started, and never leave that state in place while you look for the real cause.

Verification

Ask the running service what it is serving rather than reading the files that were just written. Connect from another host and take the certificate off the wire, confirming both a clean verification result and, critically, the serial of the new certificate rather than the old one. That single observation proves the key was readable, the service started, the correct pair was deployed and the process reloaded, none of which a file listing can establish. Confirm the process is genuinely new by comparing the unit main start timestamp against the time of the change, since a service that never restarted can present a perfectly correct file on disk and continue serving what it loaded hours earlier. Prove the exposure is closed rather than merely improved. A read test performed as an unrelated unprivileged account must fail, and a search of the private key directory for any file carrying a read bit for group or other must return nothing. Run that search across every host in the fleet rather than the one that failed, because the deployment defect is shared and the others have simply not been renewed yet. If the key was rotated, confirm the old certificate no longer appears anywhere it could still be served, and confirm the revocation actually reaches a client that checks it. An entry on a CRL that nothing consults is paperwork rather than a control.

Prevention

Place key material with a tool that takes ownership and mode as arguments. install with an explicit owner, group and mode makes the result independent of who ran the script and of their umask, and it is a one word change from cp. Better still, stop handing the key to the service. A service manager credential mechanism keeps the file root owned and unreadable by the service account, passing a copy through a private directory only that process can see. It removes the class rather than making it rarer. Assert the invariant after every deployment. A search for any file under the private key directory carrying a read bit for group or other must return nothing, run as a pipeline gate and again on a fifteen minute schedule, with a file integrity watch alerting within five minutes of any mode change. Make the smoke test open a connection rather than check a file. Verifying that the certificate file exists with the right dates verifies that a copy succeeded; completing a handshake and comparing the served serial verifies that the service works. Fail the pipeline when the restart fails. A step whose exit status is never examined reports success independently of reality, which is why a customer found this outage first. Renew early enough that none of this happens at night. A warning at 30 days and a page at 7 keeps certificate work inside office hours, where a chmod is likelier to be questioned.

Reported symptoms

app-api on web-01 terminates TLS itself and runs as the unprivileged app account. Its certificate is renewed by a pipeline that has run every ninety days for two years. The host is shared: it carries fourteen interactive accounts and a build agent that several teams use.

At 22:40 UTC on 26 August the renewal deployment ran and the pipeline recorded a success. At 22:52 the first customer report arrives: the service is refusing connections.

  • The unit is in a failed state. Every restart attempt fails again within a second, which rules out a slow start, a port conflict or anything resource related.
  • The journal entry immediately before the exit names the private key path and reports that the process could not open it. Because the message names the key file, two people immediately start looking for a corrupt or truncated key.
  • The certificate deployed alongside it is current and correct, with a notBefore from that same evening.
  • A sibling service on the same host, using the same pipeline and its own certificate, is running normally and has not been touched.
  • At 22:58, to rule permissions out, an engineer runs chmod 644 on the key. The service starts immediately. The site returns. The incident is closed at 23:05 with the cause recorded as a permissions issue.

That is where it would have ended. At 09:15 the next morning the routine host scan flags a world readable private key under /etc/ssl/private. The key had been readable by every account on a shared host for ten hours and seventeen minutes, and the incident that produced that state had been closed as resolved nine hours earlier.

Evidence provided

Read-only / Safeweb-01 at 22:55 - the certificate is current and is not the problem
$ openssl x509 -in /etc/ssl/certs/app.lab.example.crt -noout -subject -issuer -serial -dates
subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

Illustrative output

Read-only / Safeweb-01 - the pair corresponds, so this is not a mismatched deployment
$ KEY=/etc/ssl/private/app.lab.example.key
CRT=/etc/ssl/certs/app.lab.example.crt
sudo openssl pkey -in "$KEY" -pubout | openssl sha256
openssl x509 -in "$CRT" -noout -pubkey | openssl sha256
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Illustrative output

Read-only / Safeweb-01 at 22:54 - the unit state, before anybody changes anything
$ systemctl status app-api.service --no-pager
Read-only / Safeweb-01 - who is complaining, and about what
$ sudo journalctl -u app-api.service -n 20 --no-pager
Read-only / Safeweb-01 at 22:55 - the file the process cannot open
$ ls -l /etc/ssl/private/app.lab.example.key /etc/ssl/certs/app.lab.example.crt
Read-only / Safeweb-01 - the account the service actually runs as
$ systemctl show app-api.service -p User -p Group
Read-only / Safeweb-01 - the one line that separates the key from everything else
$ sudo -u app test -r /etc/ssl/private/app.lab.example.key; echo $?; sudo -u app test -r /etc/ssl/certs/app.lab.example.crt; echo $?
Read-only / Safeweb-01 - the assertion that would have caught both halves of this
$ sudo find /etc/ssl/private -type f -perm /0044 -printf '%M %u:%g %p\n'

Work the evidence before reading on

Two things went wrong nine hours apart and the second one was recorded as the fix for the first. Pull them apart before reading on.

  1. The journal names the key path, which is why people looked for a corrupt key. Read the message source again and say which process emitted it, and what that rules out.
  2. The certificate is current and the digests match. What two popular explanations does that pair of observations eliminate in one go?
  3. ls -l shows a key that root can read and nobody else can. That is exactly what a private key should look like on most hosts. What makes it wrong on this one?
  4. The chmod worked. Say precisely what it changed, what it did not change, and what a host with fourteen interactive accounts turns that into.

Before continuing: state what property of the key was traded for what property of the service at 22:58, and say what has to happen as a result.

Root cause

The deployment copied instead of installing

The renewal script moves the certificate and the key into place with cp. That command carries no opinion about ownership or mode: the resulting file is owned by whoever ran the script, and its mode comes from that process umask. Run from an automation account with root privileges, the key arrived owned by root with read and write for root and nothing for anyone else.

That is a perfectly reasonable mode for a private key in general. It is the wrong one here, because the service runs as app. The process started, tried to open the key during TLS setup, was refused by the kernel, and exited before binding a socket.

The sibling service was unaffected for a mundane reason: its key had been placed eight months earlier by a person, by hand, with an explicit mode and owner. The defect has been in the script all along and only appears on the ninety day boundary when a key is actually replaced.

Nothing checked whether the restart worked

The script issues a restart and does not look at the result. The pipeline therefore recorded a successful deployment at 22:40 against a service that was already down, and the outage was found twelve minutes later by a customer rather than by the system that had just caused it.

An unchecked exit status is not a small omission here. It is the difference between a deployment that rolls itself back at 22:41 and one that hands the problem to whoever is awake.

The fix removed the property that mattered

Making the key readable by everyone does start the service. The process can open the file, TLS setup completes, and the socket binds. Every observable symptom clears within seconds, which is exactly what makes the action so easy to accept at eleven at night.

What it removed is confidentiality. A private key is the entire basis on which a client believes it is talking to this service; the certificate is public and is meant to be. On a host with fourteen interactive accounts and a shared build agent, ten hours and seventeen minutes of world readability means the honest position is that the key may have left the host.

The incident was closed as a permissions issue. That description was accurate and it hid the fact that an availability incident had been converted into a key exposure.

Resolution

  1. Settle the exposure question before anything else, because it decides whether this ends with an install or with a new key. Take the window from the file modification time, the journal and the shell history, then list who could have read the file during it: interactive accounts, shared build agents, backup jobs running as root, and any configuration archiving process.
  2. On a shared host, assume the key left. A private key that could have been copied is a rotated key, and re-permissioning it afterwards restores the file mode without restoring the secrecy. This is a judgement call only where the set of principals with access was genuinely empty.
  3. Restore correct ownership and mode immediately regardless of the rotation decision, because that closes the window while you think. Use sudo install -o app -g app -m 0400 rather than chmod after the fact, and check the directory above the file grants no more access than the file does.
  4. Restart the unit and confirm it stays running, then confirm from another host that the service is answering. A unit that enters a failed state twice in a row is telling you something the file listing is not.
  5. If exposure is credible, rotate properly: generate a new key, issue a new certificate for it, deploy both with install, restart, and only then revoke the old certificate. Deploy before revoking, so that revocation never becomes the thing standing between you and a working service.
  6. Be honest about what the revocation buys. In an internal PKI it works only where a CRL is genuinely distributed and checked, and for a publicly trusted certificate mainstream clients soft fail or do not check at all. The effective control for a compromised key is a short lifetime and fast reissuance, with revocation as a secondary measure rather than the plan.
  7. Fix the deployment path in the same change: replace cp with install carrying an explicit owner, group and mode, write to a temporary file in the same directory and rename it into place so a partial copy is never readable, and check the exit status of the restart so a failed service can never be reported as a successful deployment.
  8. Never leave a private key world readable while you look for the real cause, and never record a chmod as the resolution of an incident. Both are how a twenty minute outage becomes a rotation nobody planned.
  9. Run the same assertion across every host in the fleet before closing, because the deployment defect is shared and the other hosts have simply not reached their renewal date yet.

Verification

  1. Ask the running service what it is serving. From another host, take the certificate off the wire and confirm a clean verification result, and then confirm the serial is the new one rather than the old. That single observation proves the key was readable, the service started, the correct pair was deployed and the process picked it up.
  2. Confirm the process is genuinely new: systemctl show app-api.service -p ExecMainStartTimestamp must report a time after the change. A service that never restarted can present a perfectly correct pair on disk while continuing to serve what it loaded hours ago.
  3. Prove the exposure is closed rather than narrowed. sudo -u nobody test -r against the key must fail, and sudo find /etc/ssl/private -type f -perm /0044 must return nothing.
  4. Run that same find on every host in the fleet, not only the one that failed. The deployment defect is shared, and a host that has not renewed recently is carrying the same script with the fault simply not yet triggered.
  5. If the key was rotated, confirm the old certificate is no longer present anywhere it could still be served, including load balancer configuration, container images built that week, and any backup restored into a test environment.
  6. Confirm the revocation actually reaches a client that checks it, rather than confirming that the serial appears on a list. An entry on a CRL nothing consults is paperwork, and it is worth knowing which of the two you have.
  7. Deliberately break a deployment in a staging environment and confirm the pipeline now fails rather than reporting success, since the unchecked restart is the defect most likely to reappear in a different script.

Prevention

  • Place key material with a tool that takes ownership and mode as arguments. install -o app -g app -m 0400 makes the result independent of who ran the script and of their umask. It is a one word change from cp in nearly every deployment that gets this wrong.
  • Stop handing the key to the service at all. A service manager credential mechanism keeps the file root owned and unreadable by the service account, passing a copy through a private directory only that process can see. It removes the class rather than making it less frequent.
  • Assert the invariant after every deployment. A search for any file under the private key directory carrying a read bit for group or other must return nothing, run as a pipeline gate and again on a fifteen minute schedule, with a file integrity watch alerting within five minutes of a mode change on those paths.
  • Make the smoke test open a connection. Checking that the certificate file exists with the right dates verifies that a copy succeeded. Completing a handshake and comparing the served serial against the one just issued verifies that the service works.
  • Fail the pipeline when the restart fails. A deployment step whose exit status is never examined reports success independently of reality, which is why a customer found this outage before the system that caused it did.
  • Renew early enough to stay in office hours. A warning at 30 days and a page at 7 keeps certificate work away from midnight, where a chmod is much likelier to be questioned before it is accepted as a resolution.
  • Use the platform course for the mechanics. Linux for Production Sysadmins - Part IV (Users) covers ownership and the service account model this rests on, Part XXX (Capabilities) covers dropping privilege without needing the key to be widely readable, and Part XXXI (Audit) covers recording who changed a file mode and when, which is the evidence this incident needed the next morning.