Lab 9: Diagnose an expired certificate

Objective

Certificate expiry is the most preventable outage in infrastructure and one of the most common. It has a known date, months of warning, and a one-line detection command, and it still takes services down every week. This lab is about closing the gap between those two facts.

You will manufacture the failure rather than wait for it, by issuing a certificate whose validity window closed thirty days ago. Serving it teaches the first uncomfortable lesson: the server starts normally, reports a healthy configuration test, and answers on port 443. Expiry is judged entirely by clients, at connection time, against their own clocks.

Then you will build the detection. openssl x509 -checkend answers “will this certificate still be valid in N seconds” with an exit code, which is exactly the shape a monitor needs. You will wrap it in a two-tier check, and then find the two things it cannot see: a certificate that is not valid yet, and a host whose clock is wrong. Both produce failures that look identical to expiry from the client’s side, and neither is detected by the check everybody deploys.

Architecture

One nginx container serves app.lab.example on port 443, published to the host on 8449. Three certificates are issued from the same request and the same key, differing only in their validity windows: one already closed, one open now, one not yet open. The clients on the host judge whichever is deployed against the host clock, and the container carries a clock of its own that the lab compares.

flowchart TD
    K["one key, one CSR"] --> A["expired.crt\nnotAfter 30 days ago"]
    K --> B["app.crt\nvalid now, 90 days"]
    K --> C["future.crt\nnotBefore in 10 days"]
    A --> V{"client checks the window\nagainst its own clock"}
    B --> V
    C --> V
    V -- "now outside the window" --> F["handshake completes\nverification refuses"]
    V -- "now inside the window" --> P["request served"]

The diagram places the decision where it actually happens. Three certificates carry identical subjects, identical names, identical chains and identical keys. The only variable is a pair of timestamps, and the only judge is whatever clock the client happens to be running. That is why an expiry incident and a clock incident are indistinguishable from the client’s error message, and why the first question in either case is what time the two ends think it is.

Requirements

• OpenSSL 3.5.x, providing openssl x509 with -not_before, -not_after and -checkend.
• GNU coreutils date, for the relative date arithmetic. The -d 'N days ago' form is a GNU extension and is not available on BSD or macOS date without adjustment.
• Docker with permission to run containers and publish a port. The lab pulls nginx:1.29-alpine.
• curl built against OpenSSL.
• Host TCP port 8449 free.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. It never changes the system clock; the container’s clock is read, not set.

Labs 4 and 5 built the two-tier certificate authority this lab issues from. If you still have root.crt, srv-ca.crt and srv-ca.key, copy them into $LAB/pki and skip the first half of Task 2. Otherwise Task 2 rebuilds an equivalent authority.

Scenario

At 02:00 a batch job that has run every night for two years fails. By 08:30 a customer-facing endpoint is failing too, and the errors mention certificates. Nobody deployed anything. The certificate on the server is the same file it has been for a year, unchanged and unmoved.

That is the shape of an expiry incident: nothing changed, and it broke anyway, because the thing that changed was the date. The follow-up question, once the certificate is replaced, is why nobody knew it was coming, and that question is answered by whatever check you build in this lab rather than by the certificate itself.

Tasks

Task 1 — Prepare the lab directory and record the starting state

LAB="$HOME/rbpki-lab-09"
rm -rf "$LAB"
mkdir -p "$LAB/pki" "$LAB/site" "$LAB/html"
cd "$LAB"

{
  echo "--- containers before the lab"
  docker ps -a --format '{{.Names}}'
  echo "--- host UTC time at lab start"
  date -u
} > "$LAB/state.pre-lab"

docker rm -f rbpki-web-09 2>/dev/null || true

echo 'lab ok' > "$LAB/html/index.html"
cat "$LAB/state.pre-lab"

The lab start time is recorded alongside the container inventory because every validity window in this lab is computed relative to it. If you return to the lab tomorrow, the certificates you issued today will have moved through their windows, and the recorded time is what lets you reconcile what you see with what you expected.

Task 2 — Build the authority and one certificate per validity window

LAB="$HOME/rbpki-lab-09"
cd "$LAB/pki"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" -out root.crt

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out srv-ca.key
openssl req -new -key srv-ca.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA" -out srv-ca.csr
cat > srv-ca.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF
openssl x509 -req -in srv-ca.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 1825 -extfile srv-ca.ext -out srv-ca.crt

# One key and one request, reused for all three certificates.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
openssl req -new -key app.key -sha256 -subj "/CN=app.lab.example" -out app.csr
cat > app.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

# The certificate that is valid right now.
openssl x509 -req -in app.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
  -sha256 -days 90 -extfile app.ext -out app.crt
chmod 600 root.key srv-ca.key app.key
openssl x509 -in app.crt -noout -subject -dates

One key and one certificate signing request serve all three certificates. That is not a shortcut, it is the point: the key is irrelevant to expiry, the names are irrelevant to expiry, and the only thing that will differ between a certificate that works and one that does not is a pair of timestamps written by the issuer.

Task 3 — Issue a certificate whose window has already closed

LAB="$HOME/rbpki-lab-09"
cd "$LAB/pki"

# A window that opened 60 days ago and closed 30 days ago.
NOT_BEFORE=$(date -u -d '60 days ago' +%Y%m%d%H%M%SZ)
NOT_AFTER=$(date -u -d '30 days ago' +%Y%m%d%H%M%SZ)
echo "issuing with notBefore=$NOT_BEFORE notAfter=$NOT_AFTER"

openssl x509 -req -in app.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
  -sha256 -not_before "$NOT_BEFORE" -not_after "$NOT_AFTER" \
  -extfile app.ext -out expired.crt

cat expired.crt srv-ca.crt > expired-fullchain.pem
openssl x509 -in expired.crt -noout -dates
date -u

-not_before and -not_after take an explicit YYMMDDHHMMSSZ or YYYYMMDDHHMMSSZ timestamp, and -not_after overrides -days, so the two flags are used without it. Manufacturing an expired certificate this way is far better than waiting for one or than changing the system clock: it is deterministic, it is reversible, and it leaves nothing on the host to undo.

Read the two dates beside date -u and confirm the arithmetic yourself. The window opened sixty days ago and closed thirty days ago, so the current time is thirty days past notAfter. Every symptom that follows is a consequence of that single relationship, and being able to state it in one sentence is what turns an incident into a diagnosis.

Task 4 — Serve it, and watch the server not care

LAB="$HOME/rbpki-lab-09"

cat > "$LAB/site/default.conf" <<'EOF'
server {
    listen 443 ssl;
    server_name app.lab.example;

    ssl_certificate     /etc/nginx/certs/expired-fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/app.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    root /usr/share/nginx/html;
    index index.html;
}
EOF

docker run -d --name rbpki-web-09 -p 8449:443 \
  -v "$LAB/site:/etc/nginx/conf.d:ro" \
  -v "$LAB/pki:/etc/nginx/certs:ro" \
  -v "$LAB/html:/usr/share/nginx/html:ro" \
  nginx:1.29-alpine
sleep 2

docker exec rbpki-web-09 nginx -t
docker ps --filter name=rbpki-web-09 --format '{{.Names}} {{.Status}}'

cd "$LAB/pki"
curl -sS --resolve app.lab.example:8449:127.0.0.1 --cacert root.crt \
  https://app.lab.example:8449/ > "$LAB/expired-client.out" 2>&1
echo "curl exit status: $?"
head -2 "$LAB/expired-client.out"

nginx -t passes and the container stays up. There is no log line, no warning and no refusal to start. A server does not validate its own certificate’s dates, because doing so would be a policy decision it has no standing to make, and because in a long-running process the certificate can pass its expiry without anything triggering a re-read.

Read the message curl produced from expired-client.out. It exits 60, as it did for the chain failure in lab 7 and the name failure in lab 8, and the text is what distinguishes them. Run the same connection under openssl s_client -connect 127.0.0.1:8449 -servername app.lab.example -CAfile root.crt and read the verify error and the Verify return code line it prints; both name the expiry directly. Record what you see rather than what you expected to see, because the exact wording varies between client versions and the wording is the part people quote in tickets.

Task 5 — The detection primitive

LAB="$HOME/rbpki-lab-09"
cd "$LAB/pki"

# Is it valid right now? A window of zero seconds.
openssl x509 -in expired.crt -noout -checkend 0
echo "expired.crt exit: $?"

openssl x509 -in app.crt -noout -checkend 0
echo "app.crt exit: $?"

# Will it still be valid in 90 days? 7776000 seconds.
openssl x509 -in app.crt -noout -checkend 7776000
echo "app.crt in 90 days exit: $?"

$ openssl x509 -in expired.crt -noout -checkend 0
openssl x509 -in app.crt -noout -checkend 0
openssl x509 -in app.crt -noout -checkend 7776000

Certificate will expire
Certificate will not expire
Certificate will expire

Three answers, in the order the commands were run. The expired certificate reports Certificate will expire, which reads oddly for something that expired a month ago; the message means “the certificate will have expired by the end of the window you asked about”, and a window of zero seconds means now. The valid certificate reports Certificate will not expire. The same valid certificate, asked about ninety days ahead, reports Certificate will expire, because a ninety-day certificate issued today does not survive ninety more days.

The exit code is the part that matters. -checkend exits 0 when the certificate is still valid across the whole window and 1 when it is not, which makes it directly usable as a test in a script with no output parsing at all. Parsing the prose is the mistake to avoid: the wording is a human convenience and the contract is the exit status.

Task 6 — Build a two-tier expiry check

LAB="$HOME/rbpki-lab-09"

cat > "$LAB/check-expiry.sh" <<'EOF'
#!/bin/sh
# Two-tier certificate expiry check.
# Usage: check-expiry.sh CERTFILE
# Exit 0 healthy, 1 warn (inside 30 days), 2 page (inside 7 days).
set -u
CERT="$1"
WARN_SECONDS=2592000   # 30 days
PAGE_SECONDS=604800    # 7 days

NOT_AFTER=$(openssl x509 -in "$CERT" -noout -enddate | cut -d= -f2)

if ! openssl x509 -in "$CERT" -noout -checkend "$PAGE_SECONDS" >/dev/null; then
  echo "PAGE $CERT expires within 7 days or has expired (notAfter $NOT_AFTER)"
  exit 2
fi
if ! openssl x509 -in "$CERT" -noout -checkend "$WARN_SECONDS" >/dev/null; then
  echo "WARN $CERT expires within 30 days (notAfter $NOT_AFTER)"
  exit 1
fi
echo "OK $CERT valid beyond 30 days (notAfter $NOT_AFTER)"
exit 0
EOF
chmod +x "$LAB/check-expiry.sh"

"$LAB/check-expiry.sh" "$LAB/pki/app.crt";     echo "exit: $?"
"$LAB/check-expiry.sh" "$LAB/pki/expired.crt"; echo "exit: $?"

Two thresholds rather than one, because they answer different questions. Thirty days is enough notice to raise a change, chase an approval and schedule a window, so it belongs in a ticket queue. Seven days means the automation has already failed and a human must act today, so it belongs on a pager. A single threshold forces you to pick which of those two failures you would rather have.

The check reads a file, which is the right place to start and the wrong place to stop. A file on disk is not what clients see, and the most common way this monitor gives false comfort is by watching a renewed file while a long-running process serves the old certificate from memory. The production form of this check connects to the service and reads the certificate from the wire, and lab 24 builds exactly that.

Task 7 — The blind spot: notBefore, and clocks

LAB="$HOME/rbpki-lab-09"
cd "$LAB/pki"

# A window that has not opened yet.
FUTURE_START=$(date -u -d '10 days' +%Y%m%d%H%M%SZ)
FUTURE_END=$(date -u -d '100 days' +%Y%m%d%H%M%SZ)
openssl x509 -req -in app.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
  -sha256 -not_before "$FUTURE_START" -not_after "$FUTURE_END" \
  -extfile app.ext -out future.crt

openssl x509 -in future.crt -noout -dates

# The detection primitive, asked about a certificate that is not valid yet.
openssl x509 -in future.crt -noout -checkend 0
echo "checkend exit for a not-yet-valid certificate: $?"

# The verifier, asked the same thing.
openssl verify -CAfile root.crt -untrusted srv-ca.crt future.crt
echo "verify exit for a not-yet-valid certificate: $?"

# The two clocks that will judge these windows.
{
  echo "host UTC:      $(date -u)"
  echo "container UTC: $(docker exec rbpki-web-09 date -u)"
} | tee "$LAB/clock-comparison.txt"

$ openssl x509 -in future.crt -noout -checkend 0

Certificate will not expire

-checkend reports Certificate will not expire and exits 0. It is correct and it is useless: the option only ever inspects notAfter, so a certificate whose window opens in ten days passes the check while every client rejects it. openssl verify on the same file fails, because path validation checks both ends of the window.

This is not a hypothetical. A certificate issued by a CA whose clock runs ahead, or issued with a deliberate backdating offset that was miscalculated, is live on a server and rejected by clients while every expiry dashboard shows green. Any monitor built only on -checkend inherits the blind spot, and the repair is to check the whole window: openssl verify answers both questions at once, which is why the production check should run a verification rather than an expiry test.

The clock comparison in the same block is the other half. A certificate is judged by the verifier’s clock, so a host running thirty days fast rejects a perfectly valid certificate with a message that names expiry, and a host running slow accepts one that everybody else refuses. When an expiry symptom appears without an expiry date to match it, compare the clocks before you touch the certificate.

Task 8 — Repair, and capture the deliverables

LAB="$HOME/rbpki-lab-09"
cd "$LAB/pki"

# Serve the certificate that is actually valid.
cat app.crt srv-ca.crt > fullchain.pem
sed -i 's#expired-fullchain.pem#fullchain.pem#' "$LAB/site/default.conf"
docker exec rbpki-web-09 nginx -t
docker exec rbpki-web-09 nginx -s reload
sleep 2
curl -sS --resolve app.lab.example:8449:127.0.0.1 --cacert root.crt \
  https://app.lab.example:8449/

{
  echo "host UTC now: $(date -u)"
  for CERT in expired.crt app.crt future.crt; do
    echo "--- $CERT"
    openssl x509 -in "$CERT" -noout -dates
  done
} > "$LAB/validity-windows.txt"

{
  for CERT in expired.crt app.crt future.crt; do
    printf '%s checkend 0 -> ' "$CERT"
    openssl x509 -in "$CERT" -noout -checkend 0 || true
    printf '%s verify     -> ' "$CERT"
    openssl verify -CAfile root.crt -untrusted srv-ca.crt "$CERT" 2>&1 | tail -1
  done
} > "$LAB/expiry-evidence.txt"

ls -l "$LAB/validity-windows.txt" "$LAB/check-expiry.sh" \
      "$LAB/expiry-evidence.txt" "$LAB/clock-comparison.txt"

The sed -i edits a file on the host inside a bind-mounted directory, which is why the configuration directory rather than the single file is mounted: replacing a file inside a mounted directory is visible to the container immediately, while replacing a mounted file itself usually is not. The reload picks up the new chain without dropping connections.

The evidence file deliberately puts -checkend and openssl verify side by side for all three certificates, because the interesting row is future.crt, where the two disagree.

Validation

• openssl x509 -in expired.crt -noout -dates shows a notAfter roughly thirty days before the current date -u, and a notBefore roughly sixty days before it.
• openssl x509 -in expired.crt -noout -checkend 0 prints Certificate will expire and exits 1.
• openssl x509 -in app.crt -noout -checkend 0 prints Certificate will not expire and exits 0, and the same command with -checkend 7776000 prints Certificate will expire and exits 1.
• While the expired certificate is deployed, curl exits 60 and its message names the expiry. A pass here is the failure appearing.
• docker exec rbpki-web-09 nginx -t reports the configuration successful while the expired certificate is deployed. The server having no objection is part of the result.
• check-expiry.sh pki/app.crt exits 0, and check-expiry.sh pki/expired.crt exits 2.
• openssl x509 -in future.crt -noout -checkend 0 prints Certificate will not expire and exits 0, while openssl verify on the same file exits 2.
• After the repair, curl returns lab ok and exits 0.
• The four deliverables exist and are non-empty.

A failed validation is usually a date problem in the shell rather than in the certificate. If openssl x509 -req rejects the timestamp, date produced a format the flag does not accept: confirm with date -u -d '30 days ago' +%Y%m%d%H%M%SZ that you are getting fourteen digits followed by Z. If expired.crt turns out to be valid, -days was left on the command line and overrode nothing, because it is -not_after that wins - re-run with -days removed.

Expected Outcome

$HOME/rbpki-lab-09/
├── html/
│   └── index.html
├── pki/
│   ├── app.crt
│   ├── app.csr
│   ├── app.key
│   ├── expired.crt
│   ├── expired-fullchain.pem
│   ├── fullchain.pem
│   ├── future.crt
│   ├── root.crt
│   ├── srv-ca.crt
│   └── srv-ca.key
├── site/
│   └── default.conf
├── check-expiry.sh
├── clock-comparison.txt
├── expired-client.out
├── expiry-evidence.txt
├── state.pre-lab
└── validity-windows.txt

You can now manufacture an expiry failure on demand, which means you can test a monitor instead of trusting it. You can also state precisely what your expiry check does not cover, which is the part that turns a green dashboard into a false sense of safety.

Troubleshooting

openssl x509 -req reports an error about the notBefore or notAfter value. The timestamp must be YYMMDDHHMMSSZ or YYYYMMDDHHMMSSZ with the trailing Z. A date format string missing %S or the literal Z produces a value the parser rejects.

date: invalid option or an unrecognised -d argument. The relative date syntax is a GNU coreutils extension. On BSD or macOS use date -u -v-30d '+%Y%m%d%H%M%SZ' instead.

The expired certificate is served but curl succeeds. fullchain.pem is still referenced in default.conf from an earlier run, or the container was started before the configuration was written. Confirm what is on the wire with openssl s_client -connect 127.0.0.1:8449 -servername app.lab.example and read the dates it reports.

check-expiry.sh reports PAGE for a certificate that has months left. The argument was a chain file rather than a leaf, and openssl x509 reads only the first certificate in a file. Point it at the leaf, or at the file whose first certificate is the leaf.

The two clock lines in Task 7 look different but are not. The container inherits the host kernel’s clock, so the instants are identical; the two date implementations simply format them differently, and a 12-hour host format beside a 24-hour container format is not skew. Compare the values, not the strings. A genuine difference of hours means one of the two is reporting local time, which is why both commands pass -u.

Cleanup

LAB="$HOME/rbpki-lab-09"

# 1. Stop and forget the service.
docker rm -f rbpki-web-09 2>/dev/null || true

# 2. Compare against the record made in Task 1.
cat "$LAB/state.pre-lab"
echo "--- containers now"
docker ps -a --format '{{.Names}}'
echo "--- host UTC now"
date -u

# 3. Remove the lab directory, keys included.
rm -rf "$LAB"

# 4. Assert the host is as you found it.
docker ps -a --format '{{.Names}}' | grep -c '^rbpki-web-09$' || echo "container gone"
test -d "$LAB" && echo "LAB DIRECTORY STILL PRESENT" || echo "lab directory gone"

The container inventories must differ only by the removal of rbpki-web-09. The two UTC times printed in steps 2 and 3 should differ only by the time you spent on the lab; a larger gap means something adjusted the host clock during the session, which is worth understanding before you close the lab out. Both assertions in step 4 must report the resource gone.

Production notes

• Monitor the service, not the file. A renewed certificate on disk that no process has read is the most common way an expiry monitor reports healthy through an outage.
• Monitor the intermediates too. They expire on their own schedule, they are not part of any renewal you run, and their expiry takes down every leaf beneath them at once.
• Two thresholds, not one. Thirty days raises a change; seven days pages a human. Both numbers should be shorter than the certificate lifetime by a wide margin, and both need revisiting as lifetimes shrink.
• Treat clock discipline as part of certificate operations. NTP failure on a fleet produces certificate errors, authentication failures and log confusion together, and the certificate error is usually the one that gets reported first and investigated wrongly.

What You Learned

• Servers do not enforce their own certificate’s dates. nginx started, tested clean and served an expired certificate without a single complaint.
• -checkend is a horizon question answered by an exit code. Zero means valid across the whole window, one means not, and the prose message is a convenience rather than a contract.
• -checkend never reads notBefore. A certificate that is not valid yet passes the check every expiry monitor is built on, and only a full verification catches it.
• Expiry is judged by the verifier’s clock. The same certificate is valid or invalid depending on who is asking, which makes clock skew and expiry indistinguishable from the error message alone.
• Manufacture the failure to test the detection. -not_before and -not_after produce a deterministic expired certificate with nothing to undo afterwards, and changing the system clock to achieve the same thing breaks far more than it proves.
• Shrinking lifetimes make manual renewal untenable. The published caps fall to 47 days in 2029, which removes any process that depends on somebody remembering.