Lab 24: Monitor certificate expiry before it breaks

Objective

Certificate expiry is the most preventable outage in infrastructure, and the reason it keeps happening is not that nobody monitors it. It is that the monitor watches a file while the users talk to a socket, and the two are allowed to disagree for weeks without anything reporting a problem.

By the end of this lab you will have a check that runs, emits a machine-readable result, and carries its severity in its exit status. More importantly you will have built the case against your own check: you will renew a certificate on disk, watch the file check go green, and then prove with a fingerprint comparison that the service is still serving the old one. You will also build a chain where the leaf has a year left and the issuing CA has a month, which a leaf-only monitor reports as perfectly healthy.

Architecture

A two-tier certificate authority, three leaf certificates with deliberately different windows, and one nginx container serving whichever leaf you copy into place. The check runs on your host and looks at the same estate from two directions.

flowchart LR
    R["root CA\n3650 days"] --> I["issuing CA\n30 days"]
    I --> L1["app.crt\n365 days"]
    I --> L2["soon.crt\n20 days"]
    I --> L3["urgent.crt\n1 day"]
    L2 --> N["nginx serving\nserve.crt"]
    F["file check\nreads the PEM on disk"] --> V{"do they agree?"}
    N --> E["endpoint check\nreads what TLS returns"]
    E --> V

The two inputs to that decision are the whole lab. One path reads a file, which is cheap, works offline and tells you what the configuration management system intended. The other completes a TLS handshake, which is slower, needs reachability and tells you what the process is actually holding in memory. Only the second one is what a user meets.

Requirements

• OpenSSL 3.5.x on the host. Anything from 3.0 will run the commands; the output quoted is from 3.5.
• Docker 29.x, able to pull nginx:1.29-alpine and publish a port on 127.0.0.1.
• GNU coreutils, specifically date -d and csplit. On a BSD or macOS host the date arithmetic in Task 9 needs a different flag set.
• About 40 MB of disk and a working directory under $HOME.
• No out-of-band access requirement. Nothing here touches SSH, the firewall, the primary interface or /etc/fstab, and no certificate is installed into the host trust store. The only service involved runs in a container on 127.0.0.1:18424.

Scenario

Your team was paged at 03:10 because an internal API stopped answering. The certificate had expired. The post-incident review found that the expiry monitor had been green throughout, because it was reading /etc/ssl/certs/api.example.com.pem, which the renewal job had rewritten six weeks earlier. Nothing had reloaded the service, so the process was still holding the certificate it had loaded at start-up.

The action item landed on you: build a check that could not have been green that night, and be able to explain precisely what it measures.

Tasks

Task 1 — Record the starting state

LAB="$HOME/rbpki-lab-24"
rm -rf "$LAB"
mkdir -p "$LAB/pki"
cd "$LAB"

{
  echo "containers:"
  docker ps -a --filter name=rbpki- --format '{{.Names}}'
  echo "networks:"
  docker network ls --filter name=rbpki- --format '{{.Name}}'
} > "$LAB/state.pre-lab"
cat "$LAB/state.pre-lab"

Both lists must be empty. A pre-existing rbpki- resource means Cleanup would remove something that is not yours, so rename this lab’s resources before continuing.

Task 2 — Build a certificate authority with a short-lived issuer

LAB="$HOME/rbpki-lab-24"
cd "$LAB/pki"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out root.crt

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out srv-ca.key
openssl req -new -key srv-ca.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA" \
  -out srv-ca.csr

cat > srv-ca.ext <<'CAEXT'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
CAEXT

openssl x509 -req -in srv-ca.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 30 -extfile srv-ca.ext -out srv-ca.crt
chmod 600 root.key srv-ca.key

The issuing CA is given 30 days on purpose. That is the trap this lab exists to expose: nothing about a leaf certificate tells you when its issuer runs out, and an expired intermediate breaks every certificate beneath it at once.

Now issue three leaves from one key, differing only in validity.

LAB="$HOME/rbpki-lab-24"
cd "$LAB/pki"

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
chmod 600 app.key
openssl req -new -key app.key -sha256 -subj "/CN=app.lab.example" \
  -addext "subjectAltName=DNS:app.lab.example" -out app.csr

cat > app.ext <<'APPEXT'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example
APPEXT

for spec in "app.crt 365" "soon.crt 20" "urgent.crt 1"; do
  set -- $spec
  openssl x509 -req -in app.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
    -sha256 -days "$2" -extfile app.ext -out "$1"
done
ls -l app.crt soon.crt urgent.crt

Read one of them back so you know what the fields are called.

$ openssl x509 -in app.crt -noout -subject -issuer -serial -dates

subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

Your serial and dates will differ. notAfter is what every threshold in this lab compares against, and subject plus issuer are what turn an alert from “something expired” into “this expired, issued by that”.

Task 3 — The primitive, and why the exit status is the answer

openssl x509 -checkend N asks a single question: will this certificate still be valid N seconds from now.

$ openssl x509 -in app.crt -noout -checkend 0

Certificate will not expire

$ openssl x509 -in app.crt -noout -checkend 7776000

Certificate will expire

Two commands, two sentences, and the useful part is neither sentence. -checkend exits 0 when the certificate survives the horizon and 1 when it does not, so a check reads the status and never parses the text. Parsing the text is how a monitor breaks on an OpenSSL upgrade that rewords a message.

Run it across all three leaves and watch the pattern.

LAB="$HOME/rbpki-lab-24"
cd "$LAB/pki"
for c in app.crt soon.crt urgent.crt; do
  for horizon in 0 604800 2592000; do
    if openssl x509 -in "$c" -noout -checkend "$horizon" >/dev/null; then
      printf '%-12s horizon %-8s SURVIVES\n' "$c" "$horizon"
    else
      printf '%-12s horizon %-8s EXPIRES\n' "$c" "$horizon"
    fi
  done
done

604800 is seven days and 2592000 is thirty. urgent.crt fails both, soon.crt fails only the thirty-day horizon, and app.crt survives both. That is a two-threshold alerting policy expressed entirely in exit statuses.

Task 4 — Turn the primitive into a check

LAB="$HOME/rbpki-lab-24"
cat > "$LAB/cert-expiry-check.sh" <<'CHECK'
#!/usr/bin/env bash
# Reports one certificate's expiry status as a single JSON object.
# Usage: cert-expiry-check.sh TARGET KIND [SNI]
#   KIND is "file" (TARGET is a path) or "endpoint" (TARGET is host:port).
# Exit: 0 ok, 1 warning, 2 critical, 3 the check itself could not answer.
set -uo pipefail

WARN_SECONDS=${WARN_SECONDS:-2592000}
CRIT_SECONDS=${CRIT_SECONDS:-604800}

target=$1
kind=$2
pem=$(mktemp)
trap 'rm -f "$pem"' EXIT

fail() {
  printf '{"target":"%s","kind":"%s","status":"unknown","reason":"%s"}\n' \
    "$target" "$kind" "$1"
  exit 3
}

if [ "$kind" = file ]; then
  cp "$target" "$pem" || fail "cannot read file"
else
  host=${target%:*}
  port=${target##*:}
  sni=${3:-$host}
  openssl s_client -connect "$host:$port" -servername "$sni" </dev/null 2>/dev/null \
    | openssl x509 > "$pem" 2>/dev/null || fail "no certificate retrieved"
fi
[ -s "$pem" ] || fail "empty certificate"

not_after=$(openssl x509 -in "$pem" -noout -enddate | cut -d= -f2)
subject=$(openssl x509 -in "$pem" -noout -subject | cut -d= -f2-)
sha256=$(openssl x509 -in "$pem" -noout -fingerprint -sha256 | cut -d= -f2)

status=ok
code=0
if ! openssl x509 -in "$pem" -noout -checkend "$CRIT_SECONDS" >/dev/null; then
  status=critical
  code=2
elif ! openssl x509 -in "$pem" -noout -checkend "$WARN_SECONDS" >/dev/null; then
  status=warning
  code=1
fi

printf '{"target":"%s","kind":"%s","status":"%s","not_after":"%s","subject":"%s","sha256":"%s"}\n' \
  "$target" "$kind" "$status" "$not_after" "$subject" "$sha256"
exit "$code"
CHECK
chmod +x "$LAB/cert-expiry-check.sh"

"$LAB/cert-expiry-check.sh" "$LAB/pki/app.crt" file
echo "exit: $?"
"$LAB/cert-expiry-check.sh" "$LAB/pki/urgent.crt" file
echo "exit: $?"

Three design decisions in that script are worth stating out loud, because they are what makes it usable rather than merely correct.

• The severity lives in the exit status. 0, 1 and 2 map onto the ok, warning and critical convention that check runners already understand, and 3 is reserved for “the check could not answer”, which is a different thing from “the certificate is bad” and must never be silently folded into it.
• The output is one JSON object per line. A line-delimited stream survives concatenation, grep, and being appended to by several runs, which a single pretty-printed document does not.
• The fingerprint is in the output. That field is what lets you compare a file result against an endpoint result later, and it is the only field in the record that identifies the exact certificate rather than describing it.

Task 5 — Serve a certificate and check the endpoint

LAB="$HOME/rbpki-lab-24"
cat "$LAB/pki/soon.crt" "$LAB/pki/srv-ca.crt" > "$LAB/pki/serve.crt"

cat > "$LAB/nginx.conf" <<'NGINXCONF'
events {}
http {
  server {
    listen 8443 ssl;
    server_name app.lab.example;
    ssl_certificate     /certs/serve.crt;
    ssl_certificate_key /certs/app.key;
    location / { return 200 "lab ok\n"; }
  }
}
NGINXCONF

docker network create rbpki-net-24
docker run -d --name rbpki-nginx24 --network rbpki-net-24 \
  --network-alias app.lab.example \
  -p 127.0.0.1:18424:8443 \
  -v "$LAB/pki:/certs:ro" \
  -v "$LAB/nginx.conf:/etc/nginx/nginx.conf:ro" \
  nginx:1.29-alpine
sleep 3
docker ps --filter name=rbpki-nginx24 --format '{{.Names}} {{.Status}}'

"$LAB/cert-expiry-check.sh" 127.0.0.1:18424 endpoint app.lab.example
echo "exit: $?"

The check returns "status":"warning" and exits 1, because soon.crt has twenty days left and the default warning horizon is thirty. Note the third argument. Without it the check would send 127.0.0.1 as the server name.

Confirm the handshake itself is sound while you are here.

$ openssl s_client -connect 127.0.0.1:18424 -servername app.lab.example -CAfile root.crt </dev/null 2>/dev/null | grep -E 'Protocol|Cipher|Verify return code'

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Task 6 — Renew the file, do not reload the service

This is the outage from the scenario, reproduced in two commands.

LAB="$HOME/rbpki-lab-24"
cat "$LAB/pki/app.crt" "$LAB/pki/srv-ca.crt" > "$LAB/pki/serve.crt"

echo "--- file check, after renewal ---"
"$LAB/cert-expiry-check.sh" "$LAB/pki/serve.crt" file
echo "exit: $?"

echo "--- endpoint check, same moment ---"
"$LAB/cert-expiry-check.sh" 127.0.0.1:18424 endpoint app.lab.example
echo "exit: $?"

The file check reports "status":"ok" and exits 0. The endpoint check still reports "status":"warning" and exits 1, because nginx loaded its certificate at start-up and has not been told anything has changed. Both answers are correct about what they measured. Only one of them is about the service.

Task 7 — Detect the drift, then close it

Compare the fingerprint on disk against the fingerprint on the wire. The comparison needs no thresholds and no clock.

LAB="$HOME/rbpki-lab-24"
disk=$(openssl x509 -in "$LAB/pki/serve.crt" -noout -fingerprint -sha256 | cut -d= -f2)
wire=$(openssl s_client -connect 127.0.0.1:18424 -servername app.lab.example \
        </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2)

{
  echo "before reload"
  echo "  disk: $disk"
  echo "  wire: $wire"
  if [ "$disk" = "$wire" ]; then echo "  verdict: MATCH"; else echo "  verdict: DRIFT"; fi
} | tee "$LAB/drift-evidence.txt"

The verdict is DRIFT. Reload nginx and repeat.

LAB="$HOME/rbpki-lab-24"
docker kill -s HUP rbpki-nginx24
sleep 2

disk=$(openssl x509 -in "$LAB/pki/serve.crt" -noout -fingerprint -sha256 | cut -d= -f2)
wire=$(openssl s_client -connect 127.0.0.1:18424 -servername app.lab.example \
        </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2)

{
  echo "after reload"
  echo "  disk: $disk"
  echo "  wire: $wire"
  if [ "$disk" = "$wire" ]; then echo "  verdict: MATCH"; else echo "  verdict: DRIFT"; fi
} | tee -a "$LAB/drift-evidence.txt"

"$LAB/cert-expiry-check.sh" 127.0.0.1:18424 endpoint app.lab.example
echo "exit: $?"

The verdict is now MATCH and the endpoint check returns "status":"ok". Two identical fingerprints mean the process is serving the file you inspected; two different ones mean your renewal has not landed, whatever the file’s dates say.

This comparison is worth running on its own schedule. It is the only check in this lab that detects the failure while there is still a year of validity left, rather than thirty days before the outage.

Task 8 — Check the chain, not just the leaf

The served bundle contains two certificates. Confirm that before measuring anything.

LAB="$HOME/rbpki-lab-24"
cd "$LAB/pki"
openssl storeutl -noout -certs serve.crt

Now split the bundle and apply the same thirty-day horizon to every certificate in it.

LAB="$HOME/rbpki-lab-24"
cd "$LAB/pki"
rm -f chain-*.pem
csplit -sz -f chain- -b '%02d.pem' serve.crt '/-----BEGIN CERTIFICATE-----/' '{*}'

{
  for c in chain-*.pem; do
    subj=$(openssl x509 -in "$c" -noout -subject | cut -d= -f2-)
    end=$(openssl x509 -in "$c" -noout -enddate | cut -d= -f2)
    if openssl x509 -in "$c" -noout -checkend 2592000 >/dev/null; then
      verdict=OK
    else
      verdict=EXPIRING
    fi
    printf '%-10s %-9s %-28s %s\n' "$c" "$verdict" "$end" "$subj"
  done
} | tee "$LAB/chain-report.txt"

chain-00.pem is the leaf with a year to run and reports OK. chain-01.pem is the issuing CA with thirty days and reports EXPIRING. A monitor that reads only the first certificate in the bundle would have called this endpoint healthy.

The consequence is worse than a single expiry, because an intermediate serves every certificate under it. When it lapses, every leaf it signed fails at the same instant, and the failure a client reports names the leaf rather than the issuer.

$ openssl verify -CAfile root.crt app.crt

error 20 at 0 depth lookup: unable to get local issuer certificate

Supply the intermediate and the same leaf verifies.

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt

app.crt: OK

Add a rule to your monitoring policy from this: alert on the minimum remaining validity across the whole chain, and label the alert with the subject of the certificate that produced the minimum. A single number per endpoint that silently means “the leaf” is how an intermediate expiry becomes a surprise.

Task 9 — Emit metrics a monitoring system can scrape

LAB="$HOME/rbpki-lab-24"
cat > "$LAB/cert-expiry-textfile.sh" <<'PROM'
#!/usr/bin/env bash
# Renders one certificate's expiry as node_exporter textfile-collector metrics.
# Usage: cert-expiry-textfile.sh TARGET KIND [SNI]
set -uo pipefail

target=$1
kind=$2
pem=$(mktemp)
trap 'rm -f "$pem"' EXIT

if [ "$kind" = file ]; then
  cp "$target" "$pem" || exit 3
else
  host=${target%:*}
  port=${target##*:}
  sni=${3:-$host}
  openssl s_client -connect "$host:$port" -servername "$sni" </dev/null 2>/dev/null \
    | openssl x509 > "$pem" 2>/dev/null || exit 3
fi
[ -s "$pem" ] || exit 3

not_after=$(openssl x509 -in "$pem" -noout -enddate | cut -d= -f2)
end_epoch=$(date -u -d "$not_after" +%s)
now_epoch=$(date -u +%s)

echo "# HELP ssl_cert_not_after_seconds Unix time at which the certificate stops being valid."
echo "# TYPE ssl_cert_not_after_seconds gauge"
printf 'ssl_cert_not_after_seconds{target="%s",kind="%s"} %s\n' "$target" "$kind" "$end_epoch"
echo "# HELP ssl_cert_remaining_seconds Seconds of validity remaining at collection time."
echo "# TYPE ssl_cert_remaining_seconds gauge"
printf 'ssl_cert_remaining_seconds{target="%s",kind="%s"} %s\n' \
  "$target" "$kind" "$((end_epoch - now_epoch))"
PROM
chmod +x "$LAB/cert-expiry-textfile.sh"

"$LAB/cert-expiry-textfile.sh" 127.0.0.1:18424 endpoint app.lab.example

Two gauges, not one. The absolute expiry time lets the alerting system compute the remaining window itself and keeps working if a scrape is delayed. The remaining-seconds gauge is what a human reads on a dashboard. Exporting only the second one makes every graph a straight downward line whose slope depends on scrape timing rather than on anything real.

Task 10 — Capture the deliverables

LAB="$HOME/rbpki-lab-24"
cd "$LAB"

{
  "$LAB/cert-expiry-check.sh" "$LAB/pki/app.crt" file
  "$LAB/cert-expiry-check.sh" "$LAB/pki/soon.crt" file
  "$LAB/cert-expiry-check.sh" "$LAB/pki/urgent.crt" file
  "$LAB/cert-expiry-check.sh" "$LAB/pki/srv-ca.crt" file
  "$LAB/cert-expiry-check.sh" 127.0.0.1:18424 endpoint app.lab.example
} > "$LAB/expiry-report.json"
cat "$LAB/expiry-report.json"

ls -l cert-expiry-check.sh cert-expiry-textfile.sh expiry-report.json \
      chain-report.txt drift-evidence.txt

Validation

• "$LAB/cert-expiry-check.sh" "$LAB/pki/urgent.crt" file exits 2 and its JSON carries "status":"critical". An exit of 0 means the certificate was issued with the wrong -days value.
• "$LAB/cert-expiry-check.sh" "$LAB/pki/soon.crt" file exits 1 with "status":"warning". An exit of 2 means your default thresholds were edited.
• "$LAB/cert-expiry-check.sh" 127.0.0.1:18424 endpoint app.lab.example exits 0 after the reload in Task 7. An exit of 3 means the container is not running or the port is not published.
• grep -c MATCH "$LAB/drift-evidence.txt" returns 1 and grep -c DRIFT returns 1. Two DRIFT lines mean the reload did not take effect; two MATCH lines mean the renewal in Task 6 did not actually change the file.
• chain-report.txt contains exactly two rows, one OK and one EXPIRING, and the EXPIRING row names the issuing CA in its subject.
• expiry-report.json has five lines, each parsing as a JSON object. Confirm with while read -r l; do printf '%s' "$l" | python3 -m json.tool >/dev/null || echo BAD; done < expiry-report.json.
• The five deliverables exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-24/
├── state.pre-lab
├── cert-expiry-check.sh
├── cert-expiry-textfile.sh
├── expiry-report.json
├── chain-report.txt
├── drift-evidence.txt
├── nginx.conf
└── pki/
    ├── root.key, root.crt
    ├── srv-ca.key, srv-ca.crt, srv-ca.ext
    ├── app.key, app.csr, app.ext
    ├── app.crt, soon.crt, urgent.crt
    ├── serve.crt
    └── chain-00.pem, chain-01.pem

You can now answer the question the post-incident review actually asked, which was not “do we monitor certificate expiry”. It was “what does our monitor measure”. The answer has three parts: it must read the certificate the service is serving rather than the one on disk, it must send the same server name a client sends, and it must report the minimum remaining validity across the whole chain rather than the leaf’s.

Troubleshooting

The endpoint check exits 3 with no certificate retrieved. The container is not listening yet or the port is not published. Confirm with docker ps --filter name=rbpki-nginx24 and docker logs rbpki-nginx24. nginx refuses to start if serve.crt and app.key do not correspond, which happens if the bundle was assembled from a leaf that a different key signed.

nginx exits immediately. Read docker logs rbpki-nginx24. The two common causes are a serve.crt that does not exist yet because Task 5 was run before Task 2 finished, and a bind mount pointing at a directory rather than at nginx.conf.

The drift comparison prints two empty fingerprints. The s_client pipeline produced nothing, so the shell compared two empty strings and called it a match. Run the openssl s_client command on its own first and confirm it prints a PEM block.

date: invalid date in Task 9. date -d is a GNU extension. On a BSD or macOS host, parse the notAfter string with date -j -f '%b %e %H:%M:%S %Y %Z' instead.

Every certificate reports critical. The host clock is wrong, or the certificates were issued while it was. Compare date -u against a known-good source before changing any thresholds, because -checkend believes the local clock without question.

Cleanup

LAB="$HOME/rbpki-lab-24"

# 1. Stop and forget the service and its network.
docker rm -f rbpki-nginx24
docker network rm rbpki-net-24

# 2. Compare against the Task 1 capture before deleting the evidence.
cat "$LAB/state.pre-lab"
docker ps -a --filter name=rbpki- --format '{{.Names}}'
docker network ls --filter name=rbpki- --format '{{.Name}}'

# 3. Remove the lab directory, CA keys included.
rm -rf "$LAB"

To confirm restoration, run docker ps -a --filter name=rbpki- and docker network ls --filter name=rbpki-: with the --format arguments used above both must print nothing at all, matching the empty state.pre-lab capture from Task 1. Then run test -d "$HOME/rbpki-lab-24" && echo "still present" || echo "removed". No certificate was installed into the host trust store, so /usr/local/share/ca-certificates and /etc/ssl/certs need no attention.

Production notes

• Thresholds must fit the lifetime. A thirty-day warning was generous when certificates lasted a year. The public TLS maximum has been 200 days since 2026-03-15, drops to 100 days from 2027-03-15 and to 47 days from 2029-03-15. A thirty-day warning on a 47-day certificate fires during the normal renewal window, which trains people to ignore it. Express thresholds as a fraction of the certificate’s own lifetime, or accept that the real fix is automation rather than alerting.
• Monitor the renewal, not only the expiry. The certificate approaching expiry is a late symptom. The renewal job that stopped running, the ACME account that lost its authorisation, the reload hook that silently returns non-zero: each of those is observable weeks earlier and each is a better page.
• Probe from where the client is. A check that runs on the same host as the service bypasses the load balancer, the terminating proxy and any CDN in front of it, all of which hold their own certificates. Run at least one probe from outside every layer that terminates TLS.
• Observability for Production Sysadmins - Part LXIV (TLS Monitoring) covers probe_ssl_earliest_cert_expiry, the blackbox_exporter metric that turns the endpoint check in this lab into a Prometheus target with alert rules and hysteresis, and Part XI (Blackbox Monitoring) covers the probe configuration that decides which SNI and which module each target uses. Prefer a maintained exporter to a bespoke script wherever one exists; the value of the script above is that you now know what the exporter is doing.

What You Learned

• -checkend is a threshold, and the exit status is the answer. 0 means the certificate survives the horizon, 1 means it does not, and two horizons give you warning and critical without any date arithmetic.
• A file check and an endpoint check answer different questions. The file tells you what the renewal job wrote. The endpoint tells you what the process is serving. A renewal with no reload keeps the first green for as long as the service stays up.
• A fingerprint comparison detects the drift early. It needs no threshold and no clock, and it fires while a year of validity remains rather than thirty days before the outage.
• The leaf is not the chain. An intermediate with less validity than the certificates beneath it takes all of them down at once, and a client’s error names the leaf. Alert on the minimum across the chain, labelled with the subject that produced it.
• Send the right SNI, or measure the wrong certificate. A probe that connects by IP receives the default virtual host’s certificate, which is usually healthy and usually not the one your users are served.
• A check that stops running looks exactly like a healthy estate. Alert on the age of the measurement and on the disappearance of a target, not only on the value.