Deployment and reload: the step that actually breaks

The renewal job reported success. The files on disk carry a fresh serial and a validity window months into the future. The service is still presenting the old certificate, and at some point it will present an expired one. This is the most common way a fully automated certificate pipeline still produces an outage, and it happens because writing a file and changing a running process are unrelated events.

Where a certificate lives once the process has started

A service that terminates TLS does not read the certificate for each connection. At start-up, or when it processes a configuration reload, it opens the certificate and key files, parses them from PEM into internal structures, builds a TLS context from them and closes the file descriptors. From that moment the material exists as parsed objects in the process address space. The path it came from is irrelevant until something asks the process to load its configuration again.

That single fact explains almost every post-renewal surprise. Replacing the file underneath a running process changes nothing that the process can observe. Neither does replacing the directory it sat in, nor repointing a symlink.

Symlinks make this vivid. A widely used ACME client stores each issuance in a versioned archive directory and exposes a stable path made of symlinks:

cert.pem      -> ../../archive/web.lab.example/cert1.pem
chain.pem     -> ../../archive/web.lab.example/chain1.pem
fullchain.pem -> ../../archive/web.lab.example/fullchain1.pem
privkey.pem   -> ../../archive/web.lab.example/privkey1.pem

Renewal writes a new numbered file into the archive and repoints the symlink. The configured path never changes, which is exactly the point of the design, and the running process still holds the parsed contents of the file the symlink used to reference. Nothing about this is a defect in the tool; it is the operator’s job to close the loop with a reload.

Reload, restart, and the difference that matters

Both a reload and a restart cause a process to read certificate material again. They differ in what happens to traffic while that occurs, and choosing the wrong one turns a silent maintenance step into a visible blip.

• A reload keeps the listening sockets. The supervising process never closes the port. New connections are accepted throughout, and connections already in flight are allowed to complete against the old configuration before the workers handling them exit.
• A restart closes and reopens the listeners. Every established connection is severed, and there is a window in which connections are refused rather than queued. On a busy service this is visible in client error rates.
• Not every service supports a reload for this purpose. Some runtimes load a keystore once at start-up and offer no way to replace it in place, which is a property to discover during rehearsal rather than during a renewal.

The safe sequence is always the same: validate the configuration first, then reload, then verify. Validating first matters because a reload that fails on a syntax error can leave a supervisor believing the unit is healthy while the old workers continue serving the old certificate.

# 1. Prove the configuration parses before asking anything to change.
nginx -t

# 2. Ask the running master to re-read configuration and certificates.
systemctl reload nginx

# 3. Confirm the swap from outside the process, not from the filesystem.
HOST=app.lab.example
PORT=443
openssl s_client -connect "$HOST:$PORT" -servername "$HOST" </dev/null 2>/dev/null \
    | openssl x509 -noout -serial -dates

Deploying the wrong file is the other half of the problem

Renewal tooling writes several files, and only one of them is usually the right thing to point a TLS server at. The leaf on its own is not enough: the server is responsible for sending the intermediate certificates that let a client build a path to a trust anchor it already has. A configuration that references the leaf-only file works during testing on a machine that happens to have the intermediate cached, and fails for real clients.

The symptom is unmistakable once you know it. Against a server sending only the leaf, a client cannot complete the path:

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

With the intermediate included the same connection shows two entries in the chain and a clean result:

Certificate chain
 0 s:CN=app.lab.example
   i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
   i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

This class of failure is particularly nasty after a renewal because the configuration did not change. The file it referenced changed content, and if the automation started writing a leaf-only file where it previously wrote a bundle, or an operator hand-copied the wrong artefact during a manual renewal, the chain silently becomes incomplete at the exact moment everyone assumes the work is finished.

The hook that runs only when something changed

Automation usually offers two hook points, and confusing them causes either a reload storm or no reload at all. One hook runs after every renewal attempt, successful or not. The other runs only when new material was actually installed. Reload belongs in the second.

flowchart LR
    A["Scheduled attempt"] --> B{"New certificate\nissued?"}
    B -- "no" --> C["Exit quietly\nno reload"]
    B -- "yes" --> D["Write files\natomically"]
    D --> E["Deploy hook\nvalidate and reload"]
    E --> F["Verify from the socket\ncompare serial"]
    F -- "mismatch" --> G["Alert: material on disk\nnot in the process"]

Reloading on every attempt is not merely wasteful. On a service where reload has any cost at all, doing it twice daily for months teaches the team that reloads are risky, which makes the necessary reload feel like a change requiring approval. Reloading only on change keeps the operation rare, boring and trusted.

Three properties make a deploy hook safe. It must be idempotent, so that running it twice is harmless. It must validate before it reloads, so that a bad file never reaches a running process. And it must exit non-zero when the reload fails, so the renewal is recorded as incomplete rather than successful.

Proving the process is serving the new certificate

Verification has to use a channel independent of the one that made the change. Reading the file back proves the write succeeded, which was never in doubt. The observation that settles it is a handshake against the address clients use, followed by a comparison of the serial with the serial in the newly installed file.

Do this from outside every proxy and load balancer in the path, then repeat it from inside. Modern estates frequently terminate TLS in more than one place, and it is entirely possible for a renewal to land on the origin while a caching layer, an ingress controller or a service mesh sidecar continues to present its own older copy.

Remember also that a client resuming an earlier session does not receive a certificate at all. Resumption re-establishes keys from previously negotiated state, so a client with valid resumption material may keep transacting for some time without ever seeing the new certificate. That is a reason to verify with a fresh full handshake rather than concluding from a happy application that the swap has propagated.

Production discipline

1. Make verification part of the renewal, not a follow-up task. The pipeline that installs the certificate should also assert that the socket serves it, and should fail if it does not.
2. Enumerate every process that reads the file. A single path can be loaded by a web server, a mail daemon, a metrics exporter and a sidecar, and each keeps its own parsed copy.
3. Validate configuration before every reload. A reload that aborts on a parse error leaves the old material in place and the supervisor reporting success.
4. Alert on the gap between file and socket. Comparing the serial on disk with the serial on the port is a cheap check that catches missed reloads, stale proxies and unmanaged copies.
5. Rehearse the restart path too. For services that cannot reload certificates, the change is a restart, and its cost has to be known before the day it becomes urgent.

Cross-course references

• Linux for Production Sysadmins - Part LXXI (TLS) covers service-level TLS configuration on the host, including the unit behaviour that decides whether a reload reaches the process.
• Kubernetes for Production Sysadmins - Part CXIV (TLS) covers how certificate material reaches workloads through mounted volumes, where the reload problem reappears as a pod that never restarted.
• Observability for Production Sysadmins - Part XI (Blackbox) covers probing an endpoint from outside, which is the independent channel this lesson relies on for verification.

Quiz