Lab 14: Prove renewal and reload

Objective

Certificate renewal is not the hard part. Every ACME client on the market renews reliably, and certbot renew running from a timer will keep a directory of certificates fresh for years without supervision. The outage happens anyway, because renewal writes files and a running process does not read files it has already read.

By the end of this lab you will have watched a certificate be replaced on disk while the service continued to present the old one, and you will have measured that gap rather than reasoned about it. You will hold four pairs of numbers: the serial on disk and the serial on the wire, taken before renewal, after renewal, after a reload, and after a hook made the reload automatic.

The discipline is a habit of checking. After any certificate change, the question is never “did the file update” but “what is the process serving”. Those are answered by different commands and only one of them is authoritative.

Architecture

Pebble is the authority. nginx serves the site and also serves the ACME challenge from a webroot directory, which means renewals never require the service to stop. certbot runs as a separate container that shares two volumes with nginx: the webroot, so its challenge tokens are reachable, and the certificate directory, so its output is what nginx reads.

flowchart LR
    CB["certbot\n--webroot"] -- "writes token" --> W["shared webroot"]
    W --> NX["nginx :80\n.well-known"]
    P["Pebble"] -- "fetches token" --> NX
    CB -- "writes certN.pem" --> A["archive/"]
    A --> LV["live/ symlinks"]
    LV -- "read at start-up\nand at reload only" --> NX2["nginx :443\nin-memory certificate"]

Notice which arrow is dashed in practice rather than in the diagram: the one from live/ into the running listener. That path is traversed when nginx starts and when it is reloaded, and at no other time. Everything else in this picture happens on a timer without anybody watching.

Requirements

• Docker, with permission to create a network and run containers. The images ghcr.io/letsencrypt/pebble:latest, certbot/certbot:latest, nginx:1.29-alpine and alpine:3.22 are pulled on first use.
• OpenSSL 3.5.x on the host, to issue the certificate Pebble presents.
• Outbound network access to pull the images and to run one apk add.
• About 60 MB of disk under $HOME.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. Nothing is published to a host port.

Scenario

A service has been renewing its certificate automatically for a year. Last month it went down anyway, on a date that turned out to be exactly ninety days after the last time anyone restarted it. Your job is to reproduce that failure deliberately, in a place where it costs nothing, and then to close it.

Tasks

Task 1 — Record the starting state and stand up the authority

LAB="$HOME/rbpki-lab-14"
rm -rf "$LAB"
mkdir -p "$LAB/letsencrypt" "$LAB/lib" "$LAB/log" "$LAB/conf.d" \
         "$LAB/webroot" "$LAB/www" "$LAB/hooks" "$LAB/state"
cd "$LAB"

docker ps -a --format '{{.Names}}' | sort > "$LAB/state/containers.pre-lab"
docker network ls --format '{{.Name}}' | sort > "$LAB/state/networks.pre-lab"
docker network create rbpki-net14

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Renewal Test Root" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" -out root.crt

cat > pebble.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:pebble.lab.example
EOF

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out pebble.key
openssl req -new -key pebble.key -sha256 -subj "/CN=pebble.lab.example" -out pebble.csr
openssl x509 -req -in pebble.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 90 -extfile pebble.ext -out pebble.crt

# 644 on a private key is wrong everywhere except here. This key belongs to a
# throwaway lab authority, it is read by a container running as a different uid,
# and it is deleted at cleanup. On any host that matters a private key is 600,
# owned by the identity that reads it, and never world-readable.
chmod 644 pebble.crt pebble.key

cat > pebble-config.json <<'EOF'
{
  "pebble": {
    "listenAddress": "0.0.0.0:14000",
    "managementListenAddress": "0.0.0.0:15000",
    "certificate": "/pebble/pebble.crt",
    "privateKey": "/pebble/pebble.key",
    "httpPort": 80,
    "tlsPort": 443,
    "ocspResponderURL": "",
    "externalAccountBindingRequired": false
  }
}
EOF

docker run -d --name rbpki-pebble14 --network rbpki-net14 \
  --network-alias pebble.lab.example \
  -v "$LAB/pebble-config.json:/pebble/config.json:ro" \
  -v "$LAB/pebble.crt:/pebble/pebble.crt:ro" \
  -v "$LAB/pebble.key:/pebble/pebble.key:ro" \
  ghcr.io/letsencrypt/pebble:latest -config /pebble/config.json

If Pebble does not stay running, read docker logs rbpki-pebble14 and check the image’s own flag surface with docker run --rm ghcr.io/letsencrypt/pebble:latest -help before changing anything else.

Task 2 — Start the service on port 80 with a webroot

The service comes up before it has a certificate. That order matters: an ACME client using the webroot method needs a web server already answering on port 80 for the name being validated, which is why the bootstrap sequence for a new site is always HTTP first, certificate second, TLS third.

cd "$LAB"
printf 'renewal lab\n' > www/index.html

cat > conf.d/http.conf <<'EOF'
server {
    listen 80;
    server_name web.lab.example;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        root  /usr/share/nginx/html;
        index index.html;
    }
}
EOF

docker run -d --name rbpki-web14 --network rbpki-net14 \
  --network-alias web.lab.example \
  -v "$LAB/conf.d:/etc/nginx/conf.d:ro" \
  -v "$LAB/webroot:/var/www/certbot" \
  -v "$LAB/letsencrypt:/etc/letsencrypt:ro" \
  -v "$LAB/www:/usr/share/nginx/html:ro" \
  nginx:1.29-alpine

docker run -d --name rbpki-tools14 --network rbpki-net14 \
  -v "$LAB:/lab" alpine:3.22 sleep infinity
docker exec rbpki-tools14 apk add --no-cache curl openssl
docker exec rbpki-web14 nginx -t

Task 3 — Write the deploy hook before the first issuance

The hook is a plain executable. Writing it now, rather than after the outage, means every renewal from here on leaves a record of itself.

cd "$LAB"
cat > hooks/record-deploy.sh <<'EOF'
#!/bin/sh
# certbot runs this only when a certificate has actually been renewed.
printf '%s deploy hook fired for %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  "$RENEWED_LINEAGE" >> /var/www/certbot/deploy-hook.log
EOF
chmod +x hooks/record-deploy.sh
: > webroot/deploy-hook.log

RENEWED_LINEAGE is set by certbot for the hook and names the live/ directory of the certificate that was just replaced. A production hook uses it to decide which service to reload when one host holds several certificates.

Task 4 — Issue the first certificate and turn on TLS

cd "$LAB"
docker run --rm --network rbpki-net14 \
  -v "$LAB/letsencrypt:/etc/letsencrypt" \
  -v "$LAB/lib:/var/lib/letsencrypt" \
  -v "$LAB/log:/var/log/letsencrypt" \
  -v "$LAB/webroot:/var/www/certbot" \
  -v "$LAB/hooks/record-deploy.sh:/hooks/record-deploy.sh:ro" \
  -v "$LAB/root.crt:/root.crt:ro" \
  -e REQUESTS_CA_BUNDLE=/root.crt \
  certbot/certbot:latest certonly --webroot -w /var/www/certbot \
    --server https://pebble.lab.example:14000/dir \
    --agree-tos --register-unsafely-without-email --non-interactive \
    --deploy-hook /hooks/record-deploy.sh \
    -d web.lab.example

cat > conf.d/tls.conf <<'EOF'
server {
    listen 443 ssl;
    server_name web.lab.example;

    ssl_certificate     /etc/letsencrypt/live/web.lab.example/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/web.lab.example/privkey.pem;

    root  /usr/share/nginx/html;
    index index.html;
}
EOF

docker exec rbpki-web14 nginx -t
docker exec rbpki-web14 nginx -s reload
sleep 1

$ docker exec rbpki-tools14 curl -sS --cacert /lab/root.crt https://web.lab.example/

The TLS server block names the live/ path. That is the only path any service configuration should ever name, and the rest of this lab is about what that choice does and does not buy you.

Task 5 — Establish the baseline: disk and wire agree

Two independent readings of the same fact. One reads a file; the other opens a connection and reads what the process actually sent.

cd "$LAB"
cat > compare.sh <<'EOF'
#!/bin/sh
echo "-- disk --"
docker exec rbpki-tools14 openssl x509 \
  -in /lab/letsencrypt/live/web.lab.example/cert.pem -noout -serial -dates
echo "-- wire --"
docker exec rbpki-tools14 sh -c 'openssl s_client -connect web.lab.example:443 -servername web.lab.example < /dev/null 2>/dev/null | openssl x509 -noout -serial -dates'
EOF
chmod +x compare.sh

{
  echo "=== stage 1: after first issuance ==="
  ./compare.sh
} > served-vs-disk.txt
cat served-vs-disk.txt

At stage 1 the two serials are identical, because nginx loaded the file moments ago. Write down that serial. Everything that follows is measured against it.

Task 6 — Simulate a renewal, and notice what it did not do

$ docker run --rm --network rbpki-net14 \
-v "$LAB/letsencrypt:/etc/letsencrypt" \
-v "$LAB/lib:/var/lib/letsencrypt" \
-v "$LAB/log:/var/log/letsencrypt" \
-v "$LAB/webroot:/var/www/certbot" \
-v "$LAB/hooks/record-deploy.sh:/hooks/record-deploy.sh:ro" \
-v "$LAB/root.crt:/root.crt:ro" \
-e REQUESTS_CA_BUNDLE=/root.crt \
certbot/certbot:latest renew --dry-run

Simulating renewal of an existing certificate for web.lab.example
Congratulations, all simulated renewals succeeded:
/etc/letsencrypt/live/web.lab.example/fullchain.pem (success)

cd "$LAB"
{
  echo "=== archive after the dry run ==="
  docker exec rbpki-tools14 ls -1 /lab/letsencrypt/archive/web.lab.example/
  echo "=== deploy hook log after the dry run ==="
  cat webroot/deploy-hook.log
} > renew-dry-run.txt
cat renew-dry-run.txt

The archive still contains only the first numbered set. That is the point of a dry run: it exercises the account, the network path, the challenge and the authority’s policy, and it deliberately keeps none of the result.

Now look at the hook log. Did the dry run fire the deploy hook or not? Do not take an answer from documentation, including this page. Read the file, write down what you found, and note that a renewal pipeline whose only test is a dry run has tested everything except the step that reloads the service.

Task 7 — Force a real renewal and prove the service did not notice

cd "$LAB"
docker run --rm --network rbpki-net14 \
  -v "$LAB/letsencrypt:/etc/letsencrypt" \
  -v "$LAB/lib:/var/lib/letsencrypt" \
  -v "$LAB/log:/var/log/letsencrypt" \
  -v "$LAB/webroot:/var/www/certbot" \
  -v "$LAB/hooks/record-deploy.sh:/hooks/record-deploy.sh:ro" \
  -v "$LAB/root.crt:/root.crt:ro" \
  -e REQUESTS_CA_BUNDLE=/root.crt \
  certbot/certbot:latest renew --force-renewal --cert-name web.lab.example

{
  echo "=== archive after the forced renewal ==="
  docker exec rbpki-tools14 ls -1 /lab/letsencrypt/archive/web.lab.example/
  echo "=== live symlink targets ==="
  docker exec rbpki-tools14 ls -l /lab/letsencrypt/live/web.lab.example/
} > archive-listing.txt
cat archive-listing.txt

The archive now holds a second numbered set and the live/ symlinks point at it. Confirm that with archive-listing.txt before going on, because the whole demonstration depends on the file on disk genuinely having changed.

cd "$LAB"
{
  echo "=== stage 2: after renewal, before reload ==="
  ./compare.sh
} >> served-vs-disk.txt
tail -8 served-vs-disk.txt

$ docker exec rbpki-tools14 sh -c 'openssl s_client -connect web.lab.example:443 -servername web.lab.example < /dev/null 2>/dev/null | openssl x509 -noout -serial'

This is the finding. The disk serial is new. The wire serial is the one you wrote down at stage 1. Renewal succeeded, certbot reported success, the timer would have exited zero, monitoring that reads the file would be satisfied, and every client connecting to the service is still being handed the old certificate. In production that state persists until the process restarts for some unrelated reason, which is why the outage lands on a date that seems to have nothing to do with the certificate.

Task 8 — Reload, and prove the gap closes

$ docker exec rbpki-web14 nginx -s reload

cd "$LAB"
sleep 1
{
  echo "=== stage 3: after the reload ==="
  ./compare.sh
} >> served-vs-disk.txt
tail -8 served-vs-disk.txt

The two serials agree again. Nothing was restarted, no connection was refused, and the only thing that changed is that the master process was asked to read the configuration again. That single command is what the deploy hook exists to run, and its absence is the whole of the failure you just reproduced.

Task 9 — Make it automatic, and prove that too

cd "$LAB"
docker run --rm --network rbpki-net14 \
  -v "$LAB/letsencrypt:/etc/letsencrypt" \
  -v "$LAB/lib:/var/lib/letsencrypt" \
  -v "$LAB/log:/var/log/letsencrypt" \
  -v "$LAB/webroot:/var/www/certbot" \
  -v "$LAB/hooks/record-deploy.sh:/hooks/record-deploy.sh:ro" \
  -v "$LAB/root.crt:/root.crt:ro" \
  -e REQUESTS_CA_BUNDLE=/root.crt \
  certbot/certbot:latest renew --force-renewal --cert-name web.lab.example

cat webroot/deploy-hook.log
docker exec rbpki-web14 nginx -s reload
sleep 1
{
  echo "=== stage 4: renewal with the hook, then reload ==="
  ./compare.sh
} >> served-vs-disk.txt
cp webroot/deploy-hook.log deploy-hook.log

The hook log now carries one line per real renewal, with a timestamp and the lineage it applied to. In this lab the hook cannot reload nginx directly, because certbot runs in a container that has no route into the web server’s process namespace, so you performed the reload yourself. That limitation is worth stating plainly rather than hiding: on an ordinary host, where certbot and the service share a machine, the hook body is the reload command, and the two events become one.

Task 10 — Capture the deliverables

cd "$LAB"
grep -E '^(===|--|serial=)' served-vs-disk.txt > serial-summary.txt
cat serial-summary.txt
ls -l served-vs-disk.txt renew-dry-run.txt archive-listing.txt deploy-hook.log

Close the lab by reading the configuration the process is running rather than the one on disk, which is the same discipline applied to a different file:

$ docker exec rbpki-web14 nginx -T

Validation

• served-vs-disk.txt contains four stages. At stage 1 the disk and wire serials match, at stage 2 they differ, and at stages 3 and 4 they match again. Matching serials at stage 2 mean the service was restarted between the renewal and the reading, which hides the very effect the lab demonstrates.
• archive-listing.txt shows at least two numbered sets under archive/ and live/ symlinks pointing at the highest-numbered one. Only one set means --force-renewal did not take effect.
• renew-dry-run.txt shows the archive unchanged by the dry run. A new numbered file after a dry run means the command was not run with --dry-run.
• deploy-hook.log contains one line per forced renewal and, whichever answer your certbot gives, the number of lines is consistent with what you observed in Task 6.
• docker exec rbpki-web14 nginx -t reports the configuration is valid at the end of the lab.
• All four deliverables exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-14/
├── root.key            root.crt
├── pebble.key          pebble.crt        pebble-config.json
├── conf.d/http.conf    conf.d/tls.conf
├── hooks/record-deploy.sh
├── letsencrypt/
│   ├── live/web.lab.example/     (symlinks, repointed twice)
│   └── archive/web.lab.example/  (cert1.pem, cert2.pem, cert3.pem, ...)
├── served-vs-disk.txt
├── renew-dry-run.txt
├── archive-listing.txt
└── deploy-hook.log

You can now answer, for any service in your estate, the question that renewal monitoring does not: what certificate is the process serving right now, and how long has it been different from the one on disk. You also have the two-command check that answers it, and the reason a dry run passing tells you nothing about whether the reload works.

Troubleshooting

The first issuance fails on the challenge. nginx must already be answering on port 80 for web.lab.example and the location /.well-known/acme-challenge/ block must have root /var/www/certbot. Test it by hand: write a file into $LAB/webroot/.well-known/acme-challenge/ and fetch it with docker exec rbpki-tools14 curl -sS http://web.lab.example/.well-known/acme-challenge/probe.

nginx -t fails after tls.conf is added. The certificate files do not exist yet, or the letsencrypt mount is missing from the nginx container. nginx will not start a TLS listener whose certificate it cannot open.

certbot renew reports that the certificate is not due for renewal. That is the expected behaviour without --force-renewal, and it is why the lab uses that flag. Confirm the flag exists in your build with docker run --rm certbot/certbot:latest --help renew.

The wire serial never changes, even after a reload. The reload went to a different container, or nginx -s reload returned an error that was not read. Run docker exec rbpki-web14 nginx -t first, then reload, and check docker logs rbpki-web14.

s_client prints nothing. The < /dev/null redirection is required, or s_client waits for input and the pipeline never completes. Keep it.

Cleanup

LAB="$HOME/rbpki-lab-14"

# 1. Stop and forget the lab containers.
docker rm -f rbpki-tools14 rbpki-web14 rbpki-pebble14

# 2. Remove the root-owned certbot state from inside a container.
docker run --rm -v "$LAB:/lab" alpine:3.22 \
  rm -rf /lab/letsencrypt /lab/lib /lab/log /lab/webroot

# 3. Remove the lab network.
docker network rm rbpki-net14

# 4. Compare against the Task 1 capture: both diffs must print nothing.
diff <(docker ps -a --format '{{.Names}}' | sort) "$LAB/state/containers.pre-lab"
diff <(docker network ls --format '{{.Name}}' | sort) "$LAB/state/networks.pre-lab"

# 5. Remove the lab directory.
rm -rf "$LAB"

No host service was installed, no host port was bound and no host trust store was edited, so the restoration assertion is the pair of empty diffs in step 4 together with ls "$HOME/rbpki-lab-14" reporting that the directory is gone.

Production notes

• The renewal timer and the reload belong in the same unit of work. On a host where certbot and the service share a machine, put the reload in the deploy hook; where they do not, the renewal job must reach the service some other way, and that path needs its own monitoring because it is now the fragile part.
• Alert on the certificate being served, not on the file. A probe that opens a connection and reads the expiry is the only check that would have caught the failure in Task 7. A check that reads the file on disk reports healthy for the entire window in which the service is heading for an outage.
• Reload every consumer, not just the obvious one. A certificate is often read by a proxy, a sidecar, a metrics exporter and a health checker, and each of them holds its own copy in memory.
• Renewal timing is a policy decision that the authority can now inform. The renewal information extension in RFC 9773 lets a server suggest a renewal window, and renewals coordinated through it are exempt from ordinary rate limits, which matters when a mass reissuance is needed at short notice.

What You Learned

• Renewal and deployment are two events and only one of them is automated by default. The gap between them is invisible to anything that reads the file rather than the connection.
• The serial is the comparison to make. One reading from disk and one from s_client answer the only question that matters after a certificate change.
• A reload re-parses; it does not restart. New workers get the new context and old workers drain, which is why the fix is cheap and why it must be asked for explicitly.
• A dry run proves the authority path, not the deployment path. It writes no certificate, so it exercises nothing downstream of issuance, and a pipeline tested only that way is untested where it actually fails.