Monitoring certificate expiry so it actually catches it

Certificate expiry is the most predictable outage in infrastructure. The deadline is written into the certificate at issuance, it does not move, and it is readable by anyone. Estates still fall over on it, and almost always for one of three reasons: the check measured the wrong copy, the warning arrived too late to act on, or the check quietly stopped running months ago.

Measure the handshake, not the file

A renewal writes new material to disk. The process serving TLS loaded its certificate at startup and holds it in memory, and it continues presenting the old one until it is reloaded or restarted. Between those two moments the filesystem and the network disagree, and a check that reads the filesystem reports the reassuring answer.

The layout that ACME clients use makes this sharper than it sounds. Certbot keeps the current material in a live/ directory whose entries are symbolic links into an archive/ directory, and at each renewal the link target is repointed at the new generation. A file-based check follows the link, reads a certificate valid for another two months, and reports health, while the running server still holds the generation that expires on Thursday.

The same divergence appears wherever material is copied. A name served by four backends behind a load balancer has five copies of the certificate: one per backend and one at the terminating balancer. Checking one host proves something about that host. It proves nothing about the name.

• The file is intent. The handshake is fact. Only the handshake observes what a client receives, and only the client’s experience determines whether there is an outage.
• Check every name, not every host. Discovery produces names; monitoring probes names. A host inventory answers a different question and will miss the balancer entirely.
• Record the vantage point. The same name can present different material to an internal prober and an external one, and both results are worth having for different reasons.

The primitive is an exit code

OpenSSL will answer the expiry question directly rather than handing you a date to parse and compare. The -checkend option takes a number of seconds and answers whether the certificate is still valid that far into the future:

openssl x509 -checkend 0          Certificate will not expire   exit 0
openssl x509 -checkend 7776000    Certificate will expire       exit 1

The printed sentence is for humans. The exit status is the interface: zero means the certificate survives the whole window, one means it does not. There is no date arithmetic to get wrong, no locale to misparse, and no timezone conversion to reason about.

The trap is that a failed measurement also produces a non-zero status, and a script that treats every non-zero result as “expiring” will page on a DNS failure and, more dangerously, a script that treats every non-zero result as noise will swallow a genuine deadline. Capture the certificate first, prove it parsed, and only then ask the question:

HOST=api.example.com
PORT=443
WINDOW=$(( 30 * 24 * 60 * 60 ))

PEM=$(openssl s_client -connect "$HOST:$PORT" -servername "$HOST" </dev/null 2>/dev/null |
  openssl x509 2>/dev/null)

if [ -z "$PEM" ]; then
  printf 'unknown %s:%s no certificate retrieved\n' "$HOST" "$PORT"
  exit 2
fi

if printf '%s\n' "$PEM" | openssl x509 -noout -checkend "$WINDOW"; then
  printf 'ok %s:%s\n' "$HOST" "$PORT"
else
  printf 'expiring %s:%s\n' "$HOST" "$PORT"
fi

Three outcomes, not two. ok, expiring and unknown are different states requiring different responses, and collapsing the third into either of the others is how monitoring lies.

Lead time comes from the renewal path

A threshold is not a round number somebody liked. It is the worst-case duration of everything that must happen between the alert firing and new material being served, plus margin.

Add up the real path. Detection latency, since the check runs on an interval. Routing to whoever can act. The change window, if production changes need one. Issuance, which may be seconds over ACME or days through a vendor ticket queue, and which for a DNS based challenge includes propagation. Deployment and the reload that makes the process pick the material up. Verification. Then add an allowance for the alert firing at 17:00 on the Friday before a public holiday, because eventually it will.

Two thresholds follow naturally. The long one is a ticket: enough time for the ordinary process, raised as work rather than as an interruption. The short one is a page: the point past which the ordinary process can no longer complete before the deadline, so somebody has to act now.

The monitor is a system that fails silently

Every failure of an expiry monitor produces the same observable symptom, which is nothing at all. No alert fires when the checks have stopped, when the name list is stale, when a probe cannot resolve and the code skips it, or when a silence created during last quarter’s incident was never removed. Silence is the normal output of a healthy expiry monitor, so silence carries no information unless you make it carry some.

flowchart LR
    A["Name list from the register"] --> B["Probe the served certificate"]
    B --> C{"Certificate retrieved?"}
    C -- "no" --> D["unknown: alert as a measurement failure"]
    C -- "yes" --> E{"checkend within window?"}
    E -- "no" --> F["expiring: ticket or page by threshold"]
    E -- "yes" --> G["ok: emit a fresh timestamped result"]
    G --> H["Watchdog: alert if no fresh results arrive"]

The watchdog at the end of that chain is the part teams omit. Every run emits a result for every name, including the unknowns, with a timestamp. A separate rule fires when fresh results stop arriving, and another fires when the number of names being checked drops, because a stale name list is invisible from inside the pipeline. The mechanics of expressing those rules, the absence conditions, the grouping and the routing, belong to the alerting stack rather than to this course.

Finally, test the alarm. A pipeline that has never fired in anger is an untested code path. Point the check at a deliberately short-lived certificate in a non-production environment on a schedule and confirm the page arrives, at the right severity, to the rota that is actually on call this week.

Production discipline

1. Probe names from the register and reconcile the two. The monitoring configuration and the credential inventory must be generated from the same source, or the gap between them becomes the set of endpoints nobody watches.
2. Keep three outcomes end to end. Preserve unknown from the probe through to the alert rule. Every collapse into two states discards the distinction between a deadline and a blind spot.
3. Derive thresholds, then write them down. Record the measured worst-case renewal duration next to the threshold it produced, so the next person can tell whether the number is still right.
4. Alert on the pipeline, not only through it. Freshness of results and the count of monitored names are first-class signals with their own rules.
5. Rehearse the page. Exercise the whole path to a live rota on a schedule, because every untested component of an alerting path is a component that has never worked.

Cross-course references

• Observability for Production Sysadmins - Parts LXIV (TLSMonitoring) and XC (MetaMonitoring) cover the metric pipeline that carries these three outcomes and the rules that detect a monitoring system which has stopped reporting.
• Linux for Production Sysadmins - Part XXIV (Time) covers clock synchronisation on the host, which is the input the expiry arithmetic in this lesson depends on entirely.
• Kubernetes for Production Sysadmins - Part LXXVI (Certs) covers cluster certificates that renew on their own schedule and need the same served-certificate check applied to their endpoints.

Quiz