Rotating certificates and private keys across a fleet

Every certificate rotation has a step that is easy to forget because nothing depends on remembering it until much later: the running process has to be told. Until then, the file on disk and the certificate being served are two different objects, and only one of them is visible to the tooling most teams check.

Renewal, rekey and trust rotation are three operations

These are frequently discussed as one activity, and they have different triggers, different blast radiuses and different exit criteria.

• Renewal issues a new certificate for the existing public key. The private key never changes and never leaves the host. The trigger is approaching expiry, and the risk is low.
• Rekey generates a new private key, produces a new certificate signing request from it, and obtains a certificate for the new public key. The trigger is a key that may have been exposed, a policy on key age, or an algorithm change.
• Trust rotation changes which authority the certificate chains to. It is a different problem entirely, and the next two lessons cover it.

The distinction matters operationally because renewal is idempotent from the client’s point of view while rekey is not. A renewed certificate presents the same public key, so anything that pinned the key keeps working. A rekeyed certificate presents a new one, so pinning breaks, and any peer that cached the public key has to be updated. If the reason for the rotation was a suspected key compromise, renewal is not a remedy: the exposed private key still works with the still-valid old certificate until that certificate expires or is revoked.

The file on disk is not the certificate being served

A service reads its certificate and private key when it starts, or when it is told to reload its configuration, and keeps them in memory afterwards. Writing a new file changes the filesystem. It does not change the process.

HOST=api.example.com
CERT=/etc/ssl/live/api/fullchain.pem

# What is on disk:
openssl x509 -in "$CERT" -noout -fingerprint -sha256

# What is actually being presented on the wire:
openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

Two identical fingerprints mean the deployment reached the running process. Two different fingerprints mean the renewal succeeded and the rotation did not, which is the exact state that produces an expiry outage on a host whose certificate file was updated weeks earlier. The check costs one round trip and is the only one that answers the question.

The fingerprint is the right thing to compare, and the expiry date is not. Two certificates issued minutes apart for the same name can carry the same notAfter to the day while being entirely different objects with different serials and, after a rekey, different public keys. A check that reads only the expiry therefore passes on a host that is still serving the previous certificate, which is precisely the host the check exists to find. The fingerprint is a hash of the whole encoded certificate, so it changes whenever anything about it changes, and it can be compared without parsing anything.

The same connection also tells you whether the chain the service sends is complete, which is worth reading at the same time because a rekey often changes the intermediate as well:

Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
   i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

The leaf at position 0 is issued by the certificate at position 1, and that certificate is issued by an authority the client already trusts. A capture that shows only position 0 is a service that will fail for every client that does not happen to have the intermediate cached from somewhere else.

The reload step, and how it fails quietly

Reloading is not restarting. A reload asks the running process to re-read its configuration and adopt new material while keeping its listening sockets and finishing the requests already in flight. A restart drops both. Reload is what you want, and it introduces a failure mode that restart does not have.

flowchart LR
    A["New material written to disk"] --> B{"Process signalled to reload?"}
    B -- "no" --> C["Old certificate still served\nfrom memory"]
    C --> D["Expiry outage weeks later"]
    B -- "yes" --> E{"New configuration valid\nand key readable?"}
    E -- "no" --> F["Reload refused,\nold configuration kept running"]
    F --> C
    E -- "yes" --> G["New workers serve the new material"]
    G --> H{"Served fingerprint matches file?"}
    H -- "no" --> C
    H -- "yes" --> I["Stage passes"]

The branch worth studying is the refused reload. When a service cannot parse the new configuration or cannot read the new private key, the correct behaviour is to keep running the configuration it already has, and that is what mature servers do. The result is a host that is still serving, still healthy, and still presenting the old certificate. Where the service unit’s reload action is a bare signal, the command’s exit status reports only that the signal was delivered, so the automation records a success.

This is why the configuration test belongs in the deployment path, before the signal, and why the served-certificate check belongs after it. Neither alone is sufficient: the test proves the new configuration would load, the fingerprint proves it did.

Deploy beside, then flip

Overwriting key material in place destroys the thing you would roll back to, and does it at the exact moment the risk is highest. Write the new material under a new name, verify it, and then move it into position, because a move within a filesystem is atomic and a partially written certificate file is not.

STAGE=/etc/ssl/staged/api-2026-08
LIVE=/etc/ssl/live/api

install -o root -g root -m 0644 "$STAGE/fullchain.pem" "$LIVE/fullchain.pem.new"
install -o root -g root -m 0600 "$STAGE/privkey.pem" "$LIVE/privkey.pem.new"

mv "$LIVE/fullchain.pem.new" "$LIVE/fullchain.pem"
mv "$LIVE/privkey.pem.new" "$LIVE/privkey.pem"

nginx -t && systemctl reload nginx

The mode on the private key is part of the deployment, not an afterthought. A key installed with the wrong owner or mode produces a refused reload, which lands on the quiet branch of the diagram above.

Staging the fleet

A certificate rollout is a change to every host that terminates the service, so it gets the same treatment as any other fleet change: a canary, then a proportion, then the remainder, with each stage gated on evidence.

# Gate each stage from outside the load balancer, one host at a
# time. Fail the stage, not the fleet.
for H in web-01 web-02 web-03; do
  printf '%s ' "$H"
  openssl s_client -connect "$H.example.com:443" -servername api.example.com </dev/null 2>/dev/null \
    | openssl x509 -noout -enddate -fingerprint -sha256
done

Reading each backend directly, rather than through the load balancer, matters because a balancer will happily hide a single broken member behind a health check that only tests whether the port answers. The vantage point should also be outside any corporate proxy that might terminate TLS itself, or you will be verifying the proxy’s certificate rather than the service’s.

The size of the first stage should be set by what you are willing to lose rather than by what is convenient to deploy. One host out of twelve is roughly eight per cent of capacity and, on a service that is not running close to its limit, an amount of error budget a team can absorb while it works out what went wrong. Half the fleet is not, and neither is a single batch that leaves no intermediate state to stop at. Retaining the previous certificate and key on each host through every stage is what makes the rollback a move and a reload rather than a reissue, and it is the reason the deployment writes beside the live material instead of over it.

Production discipline

1. Decide renewal or rekey from the trigger. Approaching expiry calls for renewal; suspected exposure of the private key calls for a new key, and renewal does nothing for it.
2. Verify on the wire, never on the filesystem. The fingerprint the service presents is the only reading that reflects what clients experience.
3. Test the configuration before signalling and check the fingerprint after. A refused reload is silent, keeps the host healthy, and keeps the old certificate.
4. Write beside, then move. Never overwrite live key material; the previous file is the rollback, and a partial write is an outage.
5. Stage the fleet and read each member directly. A load balancer will hide a single failed host from any check that goes through it.

Cross-course references

• Observability for Production Sysadmins - Part LXIV (TLSMonitoring) covers probing the expiry of the served certificate, the continuous version of the manual check in this lesson.
• Linux for Production Sysadmins - Part XXXI (Audit) covers recording who changed key material on a host, which is what turns an unexplained fingerprint into an answerable question.
• Kubernetes for Production Sysadmins - Part CXIV (TLS) covers how an ingress controller picks up rotated material, where the reload step is performed by a controller rather than by the operator.

Quiz