Designing renewal automation you can trust

Automated renewal is usually declared finished at the moment a client first obtains a certificate without help. That is one of four stages working. The other three are the running process picking up the new file, somebody being told when a run fails, and somebody being told when runs stop happening at all. Each is a distinct mechanism with its own failure mode, and an estate that has built only the first has automated the easy part and left the outage intact.

Four stages, four separate mechanisms

flowchart LR
    A["Scheduled trigger"] --> B["Order and issue"]
    B --> C["Write to disk"]
    C --> D["Reload the process"]
    D --> E["Served on the wire"]
    A -. "never fired" .-> Z["Silent expiry"]
    B -. "failed" .-> Y["Alert on failure"]
    D -. "skipped" .-> X["Old certificate\nstill served"]

The happy path runs left to right and each dotted branch is a different incident. A trigger that never fires produces no output at all, so nothing in the logs distinguishes it from a healthy quiet night. An order that fails produces an error which is only useful if a human receives it. A reload that is skipped produces a perfectly renewed file and an unchanged wire. These cannot be covered by one check, because the observation that detects each one is different.

Test the renewal, not the configuration

A configuration test proves syntax. A renewal test proves the whole ACME exchange, including the challenge path, the account credentials and the server’s willingness to issue. Certbot provides one that runs against staging and installs nothing:

$ certbot renew --dry-run
Simulating renewal of an existing certificate for web.lab.example
Congratulations, all simulated renewals succeeded:
  /etc/letsencrypt/live/web.lab.example/fullchain.pem (success)

Be precise about what that proved. It exercised the order, the authorization and the challenge for a real name, against a real server, using the real account, and it did so without consuming a production rate-limit token. It did not prove that a renewed certificate reaches the running process, because no certificate was installed and the live files were not touched. Treating a successful dry run as end-to-end coverage is the single most common gap in otherwise careful automation.

Run it on a schedule of its own, well before the renewal window opens. A dry run that starts failing three weeks before the first real renewal is a warning; the same failure discovered when the certificate has four days left is an incident.

The reload is where renewals actually fail

A certificate on disk changes nothing. The process that terminates TLS read the file when it started and holds the parsed material in memory. Renewal without reload leaves you with a fresh file, a happy client log, and an expired certificate on the wire, which is the worst combination because every local check passes.

The file layout makes this easy to miss. A certbot deployment keeps the current material as symbolic links into a versioned archive:

cert.pem      -> ../../archive/web.lab.example/cert1.pem
chain.pem     -> ../../archive/web.lab.example/chain1.pem
fullchain.pem -> ../../archive/web.lab.example/fullchain1.pem
privkey.pem   -> ../../archive/web.lab.example/privkey1.pem

Renewal writes a new numbered file into archive/ and repoints the symlink. The path in your service configuration never changes, its modification time barely moves, and a process holding an open descriptor on the old target is entirely unaware. So prove it from the outside instead:

# Independent proof: what the file holds versus what is served.
HOST=api.example.com
CERT=/etc/letsencrypt/live/api.example.com/fullchain.pem

openssl x509 -in "$CERT" -noout -serial -dates

openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null |
    openssl x509 -noout -serial -dates

Matching serials mean the reload landed. Different serials mean it did not, and the certificate the world sees is the old one. That comparison is the only assertion in the pipeline that cannot be satisfied by a file copy.

Failure must alert, and absence is the harder signal

There are two categories to cover and they need different plumbing. A run that fails can announce itself, and on a systemd host that is a single directive:

[Unit]
Description=Renew ACME certificates
OnFailure=acme-renew-alert.service

[Service]
Type=oneshot
ExecStart=/usr/bin/certbot renew --quiet --deploy-hook /usr/local/sbin/acme-deploy

OnFailure activates the alert unit whenever the renewal unit exits non-zero, which covers the client failing, the challenge failing and the deploy hook refusing to reload a broken configuration. What it cannot cover is the unit never running. A timer that was masked during maintenance, a host rebuilt from an image that predates the automation, or a package upgrade that disabled the unit all produce silence, and silence is indistinguishable from success to anything that only watches for errors.

The mechanism for that is a heartbeat with an expiry, often called a dead-man’s switch. The renewal job reports success to an external endpoint after every run, and the external system alerts when a report has not arrived within a window longer than the schedule. The state that triggers the page is the absence of a message, so it fires correctly when the job, the host or the network has disappeared entirely. The endpoint must live outside the estate the job belongs to, or a failure that takes out both takes out the alarm with them.

Monitor from outside, and from more than one vantage

The final stage is watching the outcome rather than the process. Two signals, measured independently:

• Remaining validity, observed on the wire. Probe the service the way a client does, from outside your network, and read the notAfter of what is actually served. Express the threshold as a fraction of the certificate’s own lifetime, or as the ARI window, so it stays sensible when the lifetime drops from 90 days to 64 and then to 45.
• Time since the last successful renewal. A number that should reset on every cycle. When it exceeds one renewal interval plus a margin, something upstream of the certificate is broken and you have days rather than hours to fix it.

Both must be collected outside the failure domain of the thing they watch. A probe that runs on the same host as the renewal job, through the same egress proxy, with the same trust store, goes blind for exactly the reasons that break the renewal.

Production discipline

1. Assert on the wire, never on the file. The renewal is complete when an external observation shows the new serial, and not one step earlier.
2. Give the dry run its own schedule. Discovering a broken challenge path three weeks early costs nothing; discovering it inside the final week costs an incident.
3. Alert on staleness as well as on failure. A job that stopped running emits nothing, so the alarm must be armed by the absence of a heartbeat.
4. Keep the watcher out of the watched system. Shared hosts, shared proxies and shared trust stores make the monitor fail in step with the thing it monitors.
5. Rehearse the manual path. Automation reduces how often humans renew certificates, which means the manual procedure decays; exercise it deliberately so it exists on the day the automation cannot run.

Cross-course references

• Observability for Production Sysadmins - Part LXIII (Synthetic) covers multi-step journey probes, the external observation model that proves what a service is really serving rather than what its files contain.
• Linux for Production Sysadmins - Part XXXVI (Scheduling) covers auditing what actually runs on a host and detecting job failure, which is the layer beneath the heartbeat described here.
• Kubernetes for Production Sysadmins - Part CXIV (TLS) covers certificate rotation and renewal inside a cluster, where the reload problem reappears as a workload that must be restarted to notice a changed Secret.

Quiz