Lab 13: Automate issuance with ACME

Objective

By the end of this lab a program will have obtained a certificate without a human filling in a form, and you will have watched every step of that exchange from both ends. You will have the certificate, the account, the challenge, the server-side validation trace, and both of the refusals that ACME clients meet in production.

The certificate you end up holding is the most useful teaching artefact in this course, because it has no subject. Not an empty common name: no subject distinguished name at all. Its identity lives entirely in the subject alternative name extension, and because the subject is an empty sequence that extension is marked critical, exactly as RFC 5280 requires. If you have ever been told that the common name is deprecated but still works, this certificate is the counter-example, issued by a working ACME server on a real exchange.

The discipline being taught is that automated issuance is a protocol, not a convenience wrapper. You will be able to name what failed when it fails.

Architecture

Three containers on one Docker network. Pebble is the ACME server. certbot is the ACME client, and it also becomes the web server that answers the challenge, because --standalone starts a temporary listener for exactly that purpose. A third container holds the tools used to read the results. Docker’s embedded resolver at 127.0.0.11 is what maps the identifier being validated to the certbot container, which is why the whole exchange works without editing a hosts file.

sequenceDiagram
    participant C as certbot
    participant P as Pebble
    C->>P: GET /dir
    P-->>C: directory resource
    C->>P: newAccount, signed by the account key
    C->>P: newOrder for web.lab.example
    P-->>C: authorisation with an http-01 challenge
    C->>C: start a listener on port 80
    C->>P: challenge is ready
    P->>C: GET /.well-known/acme-challenge/token
    P-->>C: authorisation VALID
    C->>P: finalise with a CSR
    P-->>C: certificate

Three keys appear in that exchange and they are not the same key. The account key signs every ACME request, the certificate key is generated locally and never leaves the client, and the certificate signing request is signed by the certificate key to prove possession. Confusing them is the root of a whole class of ACME misconfiguration, so keep them separate in your head from here.

Requirements

• Docker, with permission to create a network and run containers. The images ghcr.io/letsencrypt/pebble:latest, certbot/certbot:latest and alpine:3.22 are pulled on first use.
• OpenSSL 3.5.x on the host, to issue the certificate Pebble presents on its own ACME endpoint.
• Outbound network access to pull the images and to run one apk add.
• About 50 MB of disk under $HOME.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. No port is published to the host: everything happens inside a user-defined Docker network.

Scenario

Your team has agreed to move internal certificate issuance onto ACME. Before anything is pointed at a real authority you have been asked to demonstrate the whole exchange, including its failures, on a machine where a mistake costs nothing. That is precisely what Pebble is for.

Tasks

Task 1 — Record the starting state

LAB="$HOME/rbpki-lab-13"
rm -rf "$LAB"
mkdir -p "$LAB/letsencrypt" "$LAB/lib" "$LAB/log" "$LAB/state"
cd "$LAB"

# Record what Cleanup must restore.
docker ps -a --format '{{.Names}}' | sort > "$LAB/state/containers.pre-lab"
docker network ls --format '{{.Name}}' | sort > "$LAB/state/networks.pre-lab"
docker network create rbpki-net13
wc -l "$LAB/state/containers.pre-lab" "$LAB/state/networks.pre-lab"

Task 2 — Interrogate the image before you assume anything about it

Container images change their entrypoints between releases, and a flag that existed last year may not exist today. Ask the image what it is rather than trusting any document, including this one:

cd "$LAB"
{
  echo "=== image configuration ==="
  docker inspect --format '{{json .Config.Entrypoint}}' ghcr.io/letsencrypt/pebble:latest
  docker inspect --format '{{json .Config.Cmd}}' ghcr.io/letsencrypt/pebble:latest
  echo "=== flags the binary reports ==="
  docker run --rm ghcr.io/letsencrypt/pebble:latest -help 2>&1 || true
} > pebble-surface.txt
cat pebble-surface.txt

Confirm from that file that the entrypoint is the Pebble binary and that a -config flag exists. If the entrypoint is a shell wrapper instead, add --entrypoint to the docker run in Task 4 naming the binary path the inspect output showed. Do the same for any other image a runbook of yours starts: the five seconds this costs is the difference between a clear failure and an inexplicable one.

Task 3 — Issue the certificate Pebble will present, and write its configuration

Pebble serves its own ACME endpoint over HTTPS, so it needs a certificate, and the client needs to trust whoever issued it. Both of those are yours to create.

cd "$LAB"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab ACME Test Root" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out root.crt

cat > pebble.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:pebble.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out pebble.key
openssl req -new -key pebble.key -sha256 -subj "/CN=pebble.lab.example" -out pebble.csr
openssl x509 -req -in pebble.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 90 -extfile pebble.ext -out pebble.crt

# 644 on a private key is wrong everywhere except here. This key belongs to a
# throwaway lab authority, it is read by a container running as a different uid,
# and it is deleted at cleanup. On any host that matters a private key is 600,
# owned by the identity that reads it, and never world-readable.
chmod 644 pebble.crt pebble.key

Now the configuration. The two profiles entries below are reproduced from a running Pebble 2.10.1 instance, and the numbers are worth reading: 7776000 seconds is 90 days and 518400 seconds is 6 days.

cd "$LAB"
cat > pebble-config.json <<'EOF'
{
  "pebble": {
    "listenAddress": "0.0.0.0:14000",
    "managementListenAddress": "0.0.0.0:15000",
    "certificate": "/pebble/pebble.crt",
    "privateKey": "/pebble/pebble.key",
    "httpPort": 80,
    "tlsPort": 443,
    "ocspResponderURL": "",
    "externalAccountBindingRequired": false,
    "profiles": {
      "default": {
        "description": "The profile you know and love",
        "validityPeriod": 7776000
      },
      "shortlived": {
        "description": "A short-lived cert profile, without actual enforcement",
        "validityPeriod": 518400
      }
    }
  }
}
EOF

httpPort is set to 80 rather than the value Pebble ships with, because the challenge in this lab is answered by a container listening on the ordinary HTTP port and the validating server has to knock on the same door. If Pebble refuses to start on your image, compare this file against the one the image ships and adjust the key names to match what your release expects.

Task 4 — Start the ACME server and fetch the directory resource

cd "$LAB"
docker run -d --name rbpki-pebble13 --network rbpki-net13 \
  --network-alias pebble.lab.example \
  -v "$LAB/pebble-config.json:/pebble/config.json:ro" \
  -v "$LAB/pebble.crt:/pebble/pebble.crt:ro" \
  -v "$LAB/pebble.key:/pebble/pebble.key:ro" \
  ghcr.io/letsencrypt/pebble:latest -config /pebble/config.json

docker run -d --name rbpki-tools13 --network rbpki-net13 \
  -v "$LAB:/lab" alpine:3.22 sleep infinity
docker exec rbpki-tools13 apk add --no-cache curl openssl
sleep 2
docker logs rbpki-pebble13 | tail -5

$ docker exec rbpki-tools13 curl -sS --cacert /lab/root.crt https://pebble.lab.example:14000/dir

Save that response to acme-directory.json. You are looking for a JSON object whose members are the URLs for the rest of the protocol: newNonce, newAccount, newOrder, revokeCert and keyChange, plus a meta object. That is the whole of the client’s configuration surface. Everything else the client does is discovered from this document, which is why moving from one ACME authority to another is a one-line change and why an authority can relocate its endpoints without breaking clients.

If the request fails at the TLS layer, the certificate in Task 3 does not cover pebble.lab.example, or root.crt is not the file that signed it.

Task 5 — Issue a certificate

$ docker run --rm --name rbpki-certbot13 --network rbpki-net13 \
--network-alias web.lab.example \
-v "$LAB/letsencrypt:/etc/letsencrypt" \
-v "$LAB/lib:/var/lib/letsencrypt" \
-v "$LAB/log:/var/log/letsencrypt" \
-v "$LAB/root.crt:/root.crt:ro" \
-e REQUESTS_CA_BUNDLE=/root.crt \
certbot/certbot:latest certonly --standalone \
  --server https://pebble.lab.example:14000/dir \
  --agree-tos --register-unsafely-without-email --non-interactive \
  -d web.lab.example

Account registered.
Requesting a certificate for web.lab.example

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/web.lab.example/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/web.lab.example/privkey.pem
This certificate expires on 2026-11-24.

The expiry in that capture is 90 days after the date it was run, which matches the validityPeriod of Pebble’s default profile exactly. Your own run will print a different date, and the arithmetic is the thing to check rather than the number.

REQUESTS_CA_BUNDLE is doing real work here. certbot talks to the ACME server through the Python requests library, and requests reads its own bundle rather than the operating system trust store, so installing the root into the container’s system store would not have been enough. That is the same distinction between an operating system trust store and a runtime trust store that decides whether half a fleet is fixed or not.

Task 6 — Read the certificate you were issued

docker exec rbpki-tools13 sh -c \
  'openssl x509 -in /lab/letsencrypt/live/web.lab.example/cert.pem \
     -noout -subject -issuer -serial -dates' > "$LAB/issued-certificate.txt"
docker exec rbpki-tools13 sh -c \
  'openssl x509 -in /lab/letsencrypt/live/web.lab.example/cert.pem \
     -noout -ext subjectAltName,keyUsage,extendedKeyUsage,basicConstraints' \
  >> "$LAB/issued-certificate.txt"
cat "$LAB/issued-certificate.txt"

$ docker exec rbpki-tools13 openssl x509 -in /lab/letsencrypt/live/web.lab.example/cert.pem -noout -subject -issuer -serial -dates

subject=
issuer=CN=Pebble Intermediate CA 03d090
serial=3DC74918D250C7E9
notBefore=Aug 26 21:31:04 2026 GMT
notAfter=Nov 24 21:31:03 2026 GMT

$ docker exec rbpki-tools13 openssl x509 -in /lab/letsencrypt/live/web.lab.example/cert.pem -noout -ext subjectAltName,keyUsage,extendedKeyUsage,basicConstraints

X509v3 Key Usage: critical
  Digital Signature
X509v3 Extended Key Usage:
  TLS Web Server Authentication
X509v3 Basic Constraints: critical
  CA:FALSE
X509v3 Subject Alternative Name: critical
  DNS:web.lab.example

subject= with nothing after it is not a formatting accident. The certificate contains an empty subject sequence, and RFC 5280 requires that when the subject is empty the subject alternative name must be marked critical, because it has become the only identity the certificate carries. A client that ignored the extension would have nothing left to match a hostname against, and marking it critical means a client that does not understand it must reject the certificate rather than proceed.

The hexadecimal suffix in the issuer name will differ on your run. Pebble generates a fresh intermediate every time it starts, which is a deliberate property of a test authority and the reason nothing issued by it should ever be trusted beyond the lab.

Task 7 — Read the file layout, because deployment depends on it

$ docker exec rbpki-tools13 ls -l /lab/letsencrypt/live/web.lab.example/

cert.pem      -> ../../archive/web.lab.example/cert1.pem
chain.pem     -> ../../archive/web.lab.example/chain1.pem
fullchain.pem -> ../../archive/web.lab.example/fullchain1.pem
privkey.pem   -> ../../archive/web.lab.example/privkey1.pem

docker exec rbpki-tools13 sh -c \
  'ls -l /lab/letsencrypt/live/web.lab.example/ /lab/letsencrypt/archive/web.lab.example/' \
  > "$LAB/certbot-layout.txt"
cat "$LAB/certbot-layout.txt"

That capture is trimmed to the link targets; your own listing also carries permissions, owner and timestamps for each entry.

live/ holds symlinks and archive/ holds the real files, numbered. On every renewal a new numbered set is written into archive/ and the symlinks in live/ are repointed. Configure every service to reference the live/ path, never the archive/ path and never a copy. A configuration that names fullchain1.pem directly, or that copies the file at deployment time, keeps serving the first certificate forever while certbot renew reports success.

Task 8 — Reproduce a policy refusal

An ACME server can decline an identifier before any challenge is attempted. Pebble refuses one name specifically so that clients can be tested against this case:

$ docker run --rm --network rbpki-net13 \
-v "$LAB/letsencrypt:/etc/letsencrypt" \
-v "$LAB/lib:/var/lib/letsencrypt" \
-v "$LAB/log:/var/log/letsencrypt" \
-v "$LAB/root.crt:/root.crt:ro" \
-e REQUESTS_CA_BUNDLE=/root.crt \
certbot/certbot:latest certonly --standalone \
  --server https://pebble.lab.example:14000/dir \
  --agree-tos --register-unsafely-without-email --non-interactive \
  -d blocked-domain.example

Order included an identifier for which issuance is forbidden by policy: "blocked-domain.example"

Note where the refusal happened. No listener was started, no token was written, and nothing was fetched. The order was rejected at creation. A client that retries this in a loop will be refused identically every time, and the fix is never on the client.

Task 9 — Reproduce an unreachable challenge, then read the server’s own trace

Ask for a name that no container answers to:

$ docker run --rm --network rbpki-net13 \
-v "$LAB/letsencrypt:/etc/letsencrypt" \
-v "$LAB/lib:/var/lib/letsencrypt" \
-v "$LAB/log:/var/log/letsencrypt" \
-v "$LAB/root.crt:/root.crt:ro" \
-e REQUESTS_CA_BUNDLE=/root.crt \
certbot/certbot:latest certonly --standalone \
  --server https://pebble.lab.example:14000/dir \
  --agree-tos --register-unsafely-without-email --non-interactive \
  -d unreachable.lab.example

  Type:   connection
Detail: Get "http://unreachable.lab.example:80/.well-known/acme-challenge/XLMIGp112V_...": error
occurred while resolving URL "...": lookup unreachable.lab.example on 127.0.0.11:53: no such host
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone
webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it
can accept inbound connections from the internet.

Two things in that message are worth separating. 127.0.0.11:53 is Docker’s embedded resolver, which tells you the lookup was performed from inside the network the ACME server sits on. The hint about inbound connections from the internet is certbot guessing, and in this case it is guessing wrong: the name simply does not exist. Read the Type and Detail fields, which come from the server, before you read the hint, which does not.

Now look at the same event from the server’s side:

$ docker logs rbpki-pebble13

Pebble ... Added order "YlHSCHXa4vp2hqiiyeKvpGyJ9yFvlqYmEje3KWbeoIA" to the db
Pebble ... POST /authZ/ -> calling handler()
Pebble ... Starting 3 validations.
Pebble ... Sleeping for 4s seconds before validating
Pebble ... Sleeping for 2s seconds before validating
Pebble ... Sleeping for 0s seconds before validating
Pebble ... Attempting to validate w/ HTTP: http://web.lab.example:80/.well-known/acme-challenge/70MIGp-...
Pebble ... authz qsVryDYim4eSCe5gPGowFLG4ADSYMyvQOvuC_srvaFM set VALID by completed challenge tNqFyyGvBRNrkFXiaztgGnjmcmBUUId0eEygW8SRaHI

Three validations, with different delays, for one authorisation. Pebble does that on purpose to model multi-perspective validation, where an authority checks the challenge from several network vantage points and requires them to agree. It is also the reason a challenge that is reachable from one path can still fail: a split-horizon DNS answer, a per-region firewall rule or a load balancer that only serves the challenge on some of its backends will satisfy one attempt and not the others.

Task 10 — Capture the deliverables

cd "$LAB"
docker exec rbpki-tools13 curl -sS --cacert /lab/root.crt \
  https://pebble.lab.example:14000/dir > acme-directory.json
docker logs rbpki-pebble13 > acme-failures.txt 2>&1
ls -l pebble-surface.txt acme-directory.json issued-certificate.txt \
  certbot-layout.txt acme-failures.txt

Append the two certbot refusals from Tasks 8 and 9 to acme-failures.txt so that the client message and the server trace for the same event sit next to each other in one file. That pairing is the artefact worth keeping: it is the template for every ACME incident write-up you will do afterwards.

Validation

• acme-directory.json parses as JSON and contains a newOrder member. If curl wrote an empty file, Pebble is not running; check docker logs rbpki-pebble13.
• issued-certificate.txt begins with a line that is exactly subject= and nothing more, and its subject alternative name extension is marked critical. A populated subject means the certificate did not come from the ACME server.
• The notAfter in issued-certificate.txt is 90 days after its notBefore, matching the validityPeriod of 7776000 seconds in the configuration file.
• certbot-layout.txt shows four entries under live/ that are symlinks into archive/. Real files under live/ mean certbot was run with an option that disabled the symlink layout, and the renewal lab will behave differently.
• acme-failures.txt contains both refusals, and the policy refusal names blocked-domain.example while the connection failure names 127.0.0.11:53.
• All five deliverables exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-13/
├── root.key             root.crt
├── pebble.key           pebble.crt        pebble-config.json
├── letsencrypt/
│   ├── live/web.lab.example/     (four symlinks)
│   └── archive/web.lab.example/  (cert1.pem, chain1.pem, ...)
├── pebble-surface.txt
├── acme-directory.json
├── issued-certificate.txt
├── certbot-layout.txt
└── acme-failures.txt

You can now describe the ACME exchange as a sequence of HTTP requests rather than as a black box, name which of the three keys is involved at each step, and tell from a failure message alone whether the fault is an identifier the authority will not issue for, a challenge it could not reach, or a client that never got as far as ordering.

Troubleshooting

certbot cannot verify the ACME server’s certificate. REQUESTS_CA_BUNDLE must name a file inside the certbot container, and that file must be the root that signed pebble.crt. Check the mount with docker run --rm -v "$LAB/root.crt:/root.crt:ro" alpine:3.22 ls -l /root.crt.

The challenge fails with a connection error for web.lab.example. The --network-alias is what makes the name resolve, and an alias only exists while the container is running. If you removed --network-alias or used a different network, Pebble’s lookup fails in exactly the way Task 9 shows on purpose.

Pebble exits immediately after starting. Read docker logs rbpki-pebble13. The commonest cause is a configuration path that does not exist inside the container, because a bind mount named a host file that had not been created yet and Docker made a directory instead.

The lab directory contains files you cannot delete. certbot runs as root inside its container, so letsencrypt/ and log/ are root-owned on the host. Cleanup removes them from inside a container; do not reach for sudo rm -rf.

docker run reports that the network does not exist. Task 1 creates rbpki-net13. If a previous run of Cleanup removed it, create it again with docker network create rbpki-net13.

Cleanup

LAB="$HOME/rbpki-lab-13"

# 1. Stop and forget the lab containers.
docker rm -f rbpki-tools13 rbpki-pebble13

# 2. Remove the root-owned certbot state from inside a container.
docker run --rm -v "$LAB:/lab" alpine:3.22 \
  rm -rf /lab/letsencrypt /lab/lib /lab/log

# 3. Remove the lab network.
docker network rm rbpki-net13

# 4. Compare against the Task 1 capture: both diffs must print nothing.
diff <(docker ps -a --format '{{.Names}}' | sort) "$LAB/state/containers.pre-lab"
diff <(docker network ls --format '{{.Name}}' | sort) "$LAB/state/networks.pre-lab"

# 5. Remove the lab directory.
rm -rf "$LAB"

Nothing on the host was reconfigured and no port was published, so the restoration assertion is the pair of empty diffs in step 4 plus ls "$HOME/rbpki-lab-13" reporting that the directory does not exist.

Production notes

• The account key is a credential and deserves the treatment of one. Whoever holds it can order and revoke certificates for every identifier the account has authorised. Back it up separately from the certificates and rotate it deliberately rather than by losing it.
• Never hardcode a certificate lifetime into monitoring, into a renewal timer or into a runbook. Read the notAfter of the certificate in front of you. Public authorities publish several profiles with different lifetimes and the maximum permitted validity is itself on a published, shortening schedule.
• Renewal timing should follow the authority’s advice rather than a fixed fraction of the lifetime. The renewal information mechanism defined in RFC 9773 lets the authority tell the client when to renew, and renewals coordinated through it are exempt from the ordinary rate limits.
• Only the DNS-01 challenge can issue a wildcard certificate. If a design needs wildcards, the automation needs credentials for the DNS zone, which is a different and usually more sensitive secret than anything HTTP-01 requires.

What You Learned

• ACME is a protocol with one configured URL. The directory resource supplies every other endpoint, which is what makes changing authority a one-line change and lets an authority move its endpoints without breaking clients.
• An ACME certificate has no subject, and that is correct. Identity lives in the subject alternative name, and an empty subject is exactly why that extension is marked critical.
• live/ is symlinks and archive/ is files. Every deployment must reference the symlink path, or renewal will succeed while the service keeps serving the first certificate it was ever given.
• A refusal and a validation failure are different events. One happens at order creation and no retry will help; the other happens during validation and the server-side trace names the network path that failed.