Lab 15: Run an internal ACME certificate authority

Objective

By the end of this lab you will be running a certificate authority that speaks ACME, and a client on the same network will have obtained a certificate from it without anyone approving anything. That is the destination most internal PKI projects are aiming at, and it is reachable in an afternoon.

The lab has a second purpose, which is a working habit rather than a piece of knowledge. Every flag used here is derived from the tool’s own help output before it is used, and recorded in a deliverable. Certificate authority software changes its command surface between releases, and a runbook that carries flags copied out of a blog post fails in a way that looks like a broken certificate authority rather than a stale document. You will practise the alternative: ask the binary, record the answer, then act.

The third thing you will take away is a number. Read the validity window your authority chose for the certificate it issued, and compare it with the ninety days a public test authority handed out. Internal authorities usually default to hours. Everything about how you operate the estate follows from that.

Architecture

One container is the authority. It holds a root key, an intermediate key encrypted under a password, a configuration file listing its provisioners, and an HTTPS listener that speaks both its own management protocol and ACME. A second container is the workload asking for a certificate; it runs certbot with a standalone listener so the authority can reach the challenge. A third holds the inspection tools.

flowchart LR
    subgraph CA["step-ca container"]
        RK["root_ca.crt\nroot_ca_key"]
        IK["intermediate_ca.crt\nencrypted key"]
        CFG["ca.json\nacme provisioner"]
    end
    CA -- "ACME directory\n:9000" --> CB["certbot\n--standalone"]
    CB -- "http-01 on :80" --> CA
    CA -- "certificate" --> CB
    RK -- "must be distributed\nout of band" --> T["every client"]

The arrow at the bottom is the one that costs money. A public authority is already in every trust store on earth; an internal authority is in none of them, and the work of getting it into all of yours never appears in the tutorial that shows you how to start the daemon. This lab makes that cost visible by proving that even the authority’s own endpoint is unreachable without its root.

Requirements

• Docker, with permission to create a user-defined network, a named volume and containers. The images smallstep/step-ca:latest, certbot/certbot:latest and alpine:3.22 are pulled on first use.
• OpenSSL 3.5.x on the host is useful for reading certificates, although every inspection in this lab can also be run inside the tools container.
• Outbound network access to pull the images and to run one apk add.
• About 80 MB of disk under $HOME, plus a Docker named volume.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. No host port is published.

Scenario

The platform team has decided that internal services will stop using self-signed certificates and long-lived manually issued ones. You have been asked to prove that an internal authority can issue automatically, and to come back with two facts: what the issuance lifetime will be, and what has to happen on every client before any of it works.

Tasks

Task 1 — Record the starting state

LAB="$HOME/rbpki-lab-15"
rm -rf "$LAB"
mkdir -p "$LAB/letsencrypt" "$LAB/lib" "$LAB/log" "$LAB/state"
cd "$LAB"

docker ps -a --format '{{.Names}}' | sort > "$LAB/state/containers.pre-lab"
docker network ls --format '{{.Name}}' | sort > "$LAB/state/networks.pre-lab"
docker volume ls --format '{{.Name}}' | sort > "$LAB/state/volumes.pre-lab"
docker network create rbpki-net15
wc -l "$LAB"/state/*.pre-lab

Three captures this time. The volume list is the one that matters: containers and networks are obviously transient, and a named volume looks permanent because it is.

Task 2 — Ask the image what it can do, before assuming anything

This is the task the rest of the lab depends on. Certificate authority tooling moves quickly, and a flag that a document shows may have been renamed, may have gained a required companion flag, or may never have existed in the release you pulled. Record the surface first.

cd "$LAB"
{
  echo "=== image configuration ==="
  docker inspect --format '{{json .Config.Entrypoint}}' smallstep/step-ca:latest
  docker inspect --format '{{json .Config.Cmd}}' smallstep/step-ca:latest
  docker inspect --format '{{json .Config.Env}}' smallstep/step-ca:latest
  echo "=== binaries the image ships ==="
  docker run --rm --entrypoint sh smallstep/step-ca:latest -c 'ls -1 /usr/local/bin' 2>&1 || true
  echo "=== step-ca ==="
  docker run --rm --entrypoint step-ca smallstep/step-ca:latest --help 2>&1 || true
  echo "=== step ca init ==="
  docker run --rm --entrypoint step smallstep/step-ca:latest ca init --help 2>&1 || true
  echo "=== step ca provisioner add ==="
  docker run --rm --entrypoint step smallstep/step-ca:latest ca provisioner add --help 2>&1 || true
} > stepca-surface.txt
grep -n -- '--acme\|--provisioner\|--password-file\|--dns\|--address\|--deployment-type' \
  stepca-surface.txt

Read that grep output before you continue. Every flag used in Task 3 must appear there. If one does not, use whatever your image reports in its place: that instruction is the procedure, not a disclaimer. If the help output names a required flag this lab does not use, add it.

Task 3 — Initialise the authority with an ACME provisioner

cd "$LAB"
printf 'lab-only-not-real\n' > ca-password.txt
chmod 600 ca-password.txt
docker volume create rbpki-stepca15-data

$ docker run --rm -v rbpki-stepca15-data:/home/step -v "$LAB/ca-password.txt:/tmp/ca-password.txt:ro" --entrypoint step smallstep/step-ca:latest ca init --name "RunBook Lab Internal CA"         --dns ca.lab.example         --address ":9000"         --provisioner lab-admin         --password-file /tmp/ca-password.txt         --acme

Three things were created by that command and they have different security properties. The root key signs one thing, once, and should then be offline. The intermediate key signs every certificate and is encrypted at rest under the password you supplied. The ACME provisioner is a policy object saying that anything which can answer a challenge for a name may have a certificate for that name, with no further approval.

Task 4 — Start the authority and read the configuration it wrote

cd "$LAB"
docker run -d --name rbpki-stepca15 --network rbpki-net15 \
  --network-alias ca.lab.example \
  -v rbpki-stepca15-data:/home/step \
  -v "$LAB/ca-password.txt:/tmp/ca-password.txt:ro" \
  --entrypoint step-ca smallstep/step-ca:latest \
  --password-file /tmp/ca-password.txt /home/step/config/ca.json

sleep 3
docker logs rbpki-stepca15

$ docker exec rbpki-stepca15 ls -l /home/step/certs /home/step/config

The listing is the map of the authority: a root certificate, an intermediate certificate, their keys, and the configuration file. Copy the root out and read the configuration:

cd "$LAB"
docker cp rbpki-stepca15:/home/step/certs/root_ca.crt root_ca.crt
docker cp rbpki-stepca15:/home/step/config/ca.json ca-config.json

openssl x509 -in root_ca.crt -noout -subject -issuer -dates
openssl x509 -in root_ca.crt -noout -ext basicConstraints,keyUsage
grep -n 'acme\|Duration\|type' ca-config.json

The root’s subject and issuer are the same string, which is what self-signed means, and its basic constraints assert that it is a certificate authority. In ca-config.json you are looking for a provisioner entry whose type is ACME and for any key whose name contains Duration. Write down the provisioner’s name, because it is part of the URL in the next task, and write down whatever duration values you find, because Task 7 checks the certificate against them.

Task 5 — Fetch the directory the internal authority serves

docker run -d --name rbpki-tools15 --network rbpki-net15 \
  -v "$LAB:/lab" alpine:3.22 sleep infinity
docker exec rbpki-tools15 apk add --no-cache curl openssl

$ docker exec rbpki-tools15 curl -sS --cacert /lab/root_ca.crt https://ca.lab.example:9000/acme/acme/directory

Save it as internal-directory.json. You are looking for the same members you would find at a public authority: newNonce, newAccount, newOrder and the rest. That is the point of using ACME internally rather than a bespoke issuance API. The client does not have to be told anything about your authority beyond one URL and one root certificate, and any ACME client works against it.

If the URL returns a 404, the provisioner is not named acme. Rebuild the path from the name you found in ca-config.json: the directory sits under the provisioner’s own segment.

Task 6 — Let a client obtain a certificate with nobody in the loop

$ docker run --rm --network rbpki-net15 --network-alias svc.lab.example -v "$LAB/letsencrypt:/etc/letsencrypt" -v "$LAB/lib:/var/lib/letsencrypt" -v "$LAB/log:/var/log/letsencrypt" -v "$LAB/root_ca.crt:/root_ca.crt:ro" -e REQUESTS_CA_BUNDLE=/root_ca.crt certbot/certbot:latest certonly --standalone   --server https://ca.lab.example:9000/acme/acme/directory   --agree-tos --register-unsafely-without-email --non-interactive   -d svc.lab.example

Nothing approved that request. The client asked for a name, the authority asked it to prove control of the name by serving a token on port 80, Docker’s embedded resolver pointed the authority at the certbot container, the token was served and the certificate was issued. REQUESTS_CA_BUNDLE is required for the same reason it was required against the test authority: certbot reaches the directory over HTTPS, and the certificate on that endpoint is signed by a root that no runtime has ever heard of.

Task 7 — Read the certificate, and read the clock

cd "$LAB"
{
  echo "=== issued certificate ==="
  docker exec rbpki-tools15 openssl x509 \
    -in /lab/letsencrypt/live/svc.lab.example/cert.pem \
    -noout -subject -issuer -serial -dates
  echo "=== extensions ==="
  docker exec rbpki-tools15 openssl x509 \
    -in /lab/letsencrypt/live/svc.lab.example/cert.pem \
    -noout -ext subjectAltName,keyUsage,extendedKeyUsage,basicConstraints
  echo "=== chain as delivered ==="
  docker exec rbpki-tools15 openssl storeutl -noout -certs \
    /lab/letsencrypt/live/svc.lab.example/fullchain.pem
} > issued-internal-certificate.txt
cat issued-internal-certificate.txt

$ docker exec rbpki-tools15 openssl verify -CAfile /lab/root_ca.crt -untrusted /lab/letsencrypt/live/svc.lab.example/chain.pem /lab/letsencrypt/live/svc.lab.example/cert.pem

Now do the arithmetic yourself. Subtract the notBefore from the notAfter and write the result at the top of the file. An internal authority typically issues for hours rather than for months, and that number is the single most important output of this lab, because it decides what the estate has to be able to do.

A certificate measured in hours cannot be renewed by a person. It cannot be renewed by a weekly change window. It demands that renewal and reload are one automated event on every host that holds one, which is precisely the discipline the previous lab proved by breaking it. Short lifetimes are not a burden bolted onto automation; they are what automation buys, and they are worth buying because they shrink the window in which a stolen key is useful.

Task 8 — Prove what running your own authority actually costs

The certificate is issued and trusted by nothing. Demonstrate that against the authority’s own endpoint, which is the shortest possible proof:

cd "$LAB"
{
  echo "=== without the authority root ==="
  docker exec rbpki-tools15 curl -sS \
    https://ca.lab.example:9000/acme/acme/directory 2>&1 || true
  echo "=== with the authority root supplied explicitly ==="
  docker exec rbpki-tools15 curl -sS --cacert /lab/root_ca.crt \
    https://ca.lab.example:9000/acme/acme/directory 2>&1 | head -3
  echo "=== fingerprint the estate must pin ==="
  openssl x509 -in root_ca.crt -noout -fingerprint -sha256
} > trust-distribution.txt
cat trust-distribution.txt

The first call fails and the second succeeds, and the only difference is a file. That file has to reach every host, every container image, every language runtime with its own bundle and every keystore in the estate before a single internal certificate is useful. Nothing in the ACME exchange distributes it, and no amount of automation on the issuance side reduces the work on the trust side.

Record the fingerprint. It is what turns “the root is installed” into “the correct root is installed”, and it is the value to pin in configuration management and to check during an audit.

Task 9 — Capture the deliverables

cd "$LAB"
docker exec rbpki-tools15 curl -sS --cacert /lab/root_ca.crt \
  https://ca.lab.example:9000/acme/acme/directory > internal-directory.json
ls -l stepca-surface.txt ca-config.json internal-directory.json \
  issued-internal-certificate.txt trust-distribution.txt

Validation

• stepca-surface.txt contains a help section for step ca init in which every flag used in Task 3 appears. A missing flag means the lab was followed against a release with a different surface and Task 3 needed adjusting.
• ca-config.json contains a provisioner entry whose type is ACME. Without it the --acme flag did not take effect and Task 6 cannot succeed.
• internal-directory.json parses as JSON and contains a newOrder member. An empty file means the provisioner name in the URL is wrong; check ca-config.json.
• issued-internal-certificate.txt shows an issuer naming the intermediate of your own authority, a subject alternative name of svc.lab.example, and a notAfter you have subtracted from notBefore and written down.
• trust-distribution.txt contains one failed call and one successful call to the same URL, plus the SHA-256 fingerprint of the root.
• docker logs rbpki-stepca15 shows the authority still running and reports no error after the issuance.
• All five deliverables exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-15/
├── ca-password.txt          (lab-only, destroyed by Cleanup)
├── root_ca.crt              (the anchor the estate would have to trust)
├── ca-config.json
├── letsencrypt/
│   ├── live/svc.lab.example/
│   └── archive/svc.lab.example/
├── stepca-surface.txt
├── internal-directory.json
├── issued-internal-certificate.txt
└── trust-distribution.txt

docker volume: rbpki-stepca15-data
  /home/step/certs/    root_ca.crt, intermediate_ca.crt and their keys
  /home/step/config/   ca.json

You can now describe an internal ACME authority to a colleague in terms of its three artefacts and its one policy object, state the lifetime it will issue, and answer the question that decides whether the project succeeds: what has to reach every client, and who is responsible for putting it there.

Troubleshooting

step ca init reports an unknown flag. Read stepca-surface.txt. The release you pulled has a different surface, and the fix is to use the flag it reports rather than the one written here. Some releases also require a deployment type to be named explicitly in non-interactive use; the help output lists the accepted values.

--entrypoint step fails because the binary is not found. The binaries section of stepca-surface.txt lists what the image actually ships and where. Use the path from that listing.

The authority exits at start-up complaining about the password. The password file must be readable inside the container at the path passed to --password-file, and it must contain exactly the same bytes used at initialisation. A trailing newline difference between printf and echo is enough to break it, which is why Task 3 uses printf with an explicit newline.

The directory URL returns 404. The provisioner name is part of the path. Take the name from ca-config.json rather than guessing, and rebuild the URL around it.

certbot fails to validate the challenge. The authority resolves svc.lab.example through Docker’s embedded resolver, which only answers for containers on the same user-defined network. Confirm the certbot container was started with both --network rbpki-net15 and --network-alias svc.lab.example.

A second run of the lab behaves strangely. The named volume survived a previous Cleanup. Remove it with docker volume rm rbpki-stepca15-data and start again from Task 1.

Cleanup

LAB="$HOME/rbpki-lab-15"

# 1. Stop and forget the lab containers.
docker rm -f rbpki-tools15 rbpki-stepca15

# 2. Remove the root-owned certbot state from inside a container.
docker run --rm -v "$LAB:/lab" alpine:3.22 rm -rf /lab/letsencrypt /lab/lib /lab/log

# 3. Destroy the authority volume, including both certificate authority keys.
docker volume rm rbpki-stepca15-data

# 4. Remove the lab network.
docker network rm rbpki-net15

# 5. Compare against the Task 1 capture: all three diffs must print nothing.
diff <(docker ps -a --format '{{.Names}}' | sort) "$LAB/state/containers.pre-lab"
diff <(docker network ls --format '{{.Name}}' | sort) "$LAB/state/networks.pre-lab"
diff <(docker volume ls --format '{{.Name}}' | sort) "$LAB/state/volumes.pre-lab"

# 6. Remove the lab directory, including the authority password file.
rm -rf "$LAB"

No host trust store was edited and no host port was bound, so the restoration assertion is the three empty diffs in step 5. The volume diff is the one worth reading carefully: it is the only artefact of this lab that would otherwise outlive it.

Production notes

• Take the root offline as soon as it has signed the intermediate. A root that is online is a root that can be stolen while the estate is trusting it, and recovering from that means visiting every client. The intermediate is the key that does the daily work and the one whose compromise is survivable.
• Plan the trust distribution before the authority, not after. The realistic sequence is: distribute the root everywhere and verify it, then start issuing. Doing it the other way round produces a working authority and a stream of verification failures that look like certificate problems.
• Give the intermediate a renewal plan of its own with a diarised date. An intermediate that expires takes every certificate under it out of service at once, and it is the failure most likely to be discovered on the day rather than in advance, because nothing monitors it by default.
• Decide the issuance lifetime deliberately and monitor against it. If the authority issues for hours, a host that cannot renew for a day is an outage rather than a warning, so the alert threshold and the escalation path have to be scaled to the lifetime rather than copied from a public-certificate runbook.

What You Learned

• Ask the binary, then act. A recorded help surface turns a renamed flag from an incident into a diff, and it costs five minutes once.
• A provisioner is an authorisation policy wearing a configuration key. An ACME provisioner grants a certificate to whoever can answer a challenge, so it delegates naming authority to whoever controls DNS and the ports behind it.
• Internal authorities issue short. Read the window your own authority chose, because it decides whether renewal can involve a person at all.
• Issuance automation does not reduce trust distribution. The root has to reach every store in the estate by some other route, and until it does, an automatically issued certificate is trusted by nothing.