Lab 11: Install a private trust anchor

Objective

By the end of this lab you will have created a private two-tier certificate authority, served a certificate from it over TLS, and made a client trust that certificate by installing the root into an operating system trust store. You will have measured the trust store before and after the install, so that “the client trusts it now” is a number that changed rather than a claim.

You will then break the comfortable assumption that follows. Installing a root into the operating system store fixes every program that asks the operating system. It does nothing at all for a program that carries its own list of trusted certificates, and a great many programs do. That second half is the part that turns into an incident ticket six months later, because the person who installed the anchor tested with one tool and declared the estate fixed.

The discipline being taught here is that a trust store is a property of a process, not of a machine. You will finish the lab able to answer, for any given client, which file that client actually read.

Architecture

One host runs OpenSSL and Docker. A private root signs an issuing CA, the issuing CA signs a server certificate for app.lab.example, and an nginx container serves the leaf and the issuing CA together. A second container acts as the client and is where all the trust decisions happen. Both containers sit on a user-defined Docker network, so Docker’s embedded resolver maps app.lab.example to the nginx container.

flowchart LR
    R["Root CA\nself-signed"] --> I["Issuing CA\npathlen:0"]
    I --> L["app.lab.example\nserverAuth leaf"]
    L --> N["nginx container\nserves leaf + issuing CA"]
    N --> C["client container"]
    R --> S["/usr/local/share/ca-certificates\nupdate-ca-certificates"]
    S --> B["/etc/ssl/certs/ca-certificates.crt"]
    B --> C
    P["pinned bundle\ninside a runtime"] --> C

The root reaches the client by a completely different route from the one the leaf takes. The leaf and the issuing CA arrive over the wire, inside the TLS handshake, because the server sends them. The root arrives out of band, as a file you place on the client. Nothing in the protocol delivers a trust anchor, and that is the entire point of an anchor. The last node in the diagram is the alternative bundle that some runtimes read instead of the system one.

Requirements

• OpenSSL 3.5.x on the host. The -addext and -CAcreateserial forms used here were confirmed against this release.
• Docker, with permission to create a network and run containers. The images nginx:1.29-alpine and alpine:3.22 are pulled on first use.
• Outbound network access from the client container for one apk add, which fetches ca-certificates, curl, openssl and python3.
• About 20 MB of disk under $HOME for keys, certificates and captures.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. Every trust store it edits lives inside a disposable container.

Scenario

Your organisation has stood up an internal certificate authority, and the platform team has asked you to make a fleet of application containers trust it. The instruction you were handed is one line long: copy the root certificate in and run update-ca-certificates. You are going to do exactly that, and then find out how much of the fleet it actually fixed.

Tasks

Task 1 — Record the starting state

LAB="$HOME/rbpki-lab-11"
rm -rf "$LAB"
mkdir -p "$LAB/www" "$LAB/state" "$LAB/conf.d" "$LAB/certs"
cd "$LAB"

# Record what Cleanup must restore.
docker ps -a --format '{{.Names}}' | sort > "$LAB/state/containers.pre-lab"
docker network ls --format '{{.Name}}' | sort > "$LAB/state/networks.pre-lab"
wc -l "$LAB/state/containers.pre-lab" "$LAB/state/networks.pre-lab"

Two captures, not one. The lab creates containers and a network, and a reader who removes the containers but forgets the network leaves a stale bridge behind on every run. Cleanup diffs against both files.

Task 2 — Build the private root and issuing CA

cd "$LAB"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out root.crt

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out srv-ca.key
openssl req -new -key srv-ca.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA" \
  -out srv-ca.csr

cat > srv-ca.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl x509 -req -in srv-ca.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 1825 -extfile srv-ca.ext -out srv-ca.crt

$ openssl x509 -in root.crt -noout -subject -issuer -ext basicConstraints,keyUsage

The root asserts pathlen:1, which permits exactly one CA beneath it, and the issuing CA asserts pathlen:0, which permits none. pathLenConstraint is only legal when cA is TRUE and keyCertSign is asserted, which is why both extensions appear together in each file.

Task 3 — Issue the server certificate and verify it locally

cd "$LAB"

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
openssl req -new -key app.key -sha256 -subj "/CN=app.lab.example" -out app.csr

cat > app.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl x509 -req -in app.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
  -sha256 -days 90 -extfile app.ext -out app.crt

cat app.crt srv-ca.crt > fullchain.pem
chmod 600 root.key srv-ca.key app.key

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt

app.crt: OK

That command proves the chain is arithmetically sound. It proves nothing about any client, because you handed OpenSSL the anchor on the command line. Every remaining task is about getting a client to supply that anchor for itself.

Task 4 — Serve the certificate from nginx

cd "$LAB"
printf 'lab ok\n' > www/index.html
cp fullchain.pem app.key certs/

cat > conf.d/default.conf <<'EOF'
server {
    listen 443 ssl;
    server_name app.lab.example;

    ssl_certificate     /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/app.key;

    root  /usr/share/nginx/html;
    index index.html;
}
EOF

docker network create rbpki-net11
docker run -d --name rbpki-web11 --network rbpki-net11 \
  --network-alias app.lab.example \
  -v "$LAB/conf.d:/etc/nginx/conf.d:ro" \
  -v "$LAB/certs:/etc/nginx/certs:ro" \
  -v "$LAB/www:/usr/share/nginx/html:ro" \
  nginx:1.29-alpine

docker exec rbpki-web11 nginx -t

ssl_certificate is given fullchain.pem, not app.crt. nginx sends every certificate in that file, in order, and the order matters: leaf first, then the issuing CA. Task 8 shows what happens when you get this wrong even with a perfectly installed anchor.

Every mount above is a directory rather than a single file, which is not an aesthetic choice. Editing a bind-mounted file in place replaces its inode, and the container goes on reading the one it was given at start-up. Task 8 rewrites the configuration, so the mount has to be a directory for the change to be visible at all.

Task 5 — Measure the client trust store before you touch it

docker run -d --name rbpki-client11 --network rbpki-net11 \
  -v "$LAB/root.crt:/opt/rbpki/root.crt:ro" \
  alpine:3.22 sleep infinity

docker exec rbpki-client11 apk add --no-cache ca-certificates curl openssl python3
docker exec rbpki-client11 cp /etc/ssl/certs/ca-certificates.crt /tmp/pinned-bundle.crt

$ docker exec rbpki-client11 sh -c 'grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt'

Record it, because it is the only baseline you get:

docker exec rbpki-client11 sh -c 'grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt' \
  > "$LAB/state/anchors.before"
cat "$LAB/state/anchors.before"

The exact value depends on the public root programme shipped in the image and will not match anyone else’s machine, which is precisely why you measure it rather than look it up.

$ docker exec rbpki-client11 curl -sS https://app.lab.example/

curl: (60) SSL certificate problem: unable to get local issuer certificate

curl prints several further lines of guidance after that one, which have been trimmed here. The wording of the first line varies between curl builds: the same failure on the host, against a curl linked to a newer OpenSSL, reads curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20). Both are the same X.509 error 20, reported by two different front ends. Never match on the sentence; match on the number.

Task 6 — Install the anchor and measure again

docker exec rbpki-client11 mkdir -p /usr/local/share/ca-certificates
docker exec rbpki-client11 cp /opt/rbpki/root.crt \
  /usr/local/share/ca-certificates/runbook-lab-root.crt
docker exec rbpki-client11 update-ca-certificates

$ docker exec rbpki-client11 sh -c 'grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt'

The anchor count must now be exactly one higher than the number you recorded in Task 5. If it is unchanged, the file you copied in was not a valid PEM certificate, or it did not end in .crt, which is the extension update-ca-certificates looks for.

$ docker exec rbpki-client11 curl -sS https://app.lab.example/

lab ok

Task 7 — Prove the operating system store did not fix everything

An installed anchor fixes any client that asks OpenSSL for the default paths. It does not fix a client that was handed a specific bundle file. The next three calls run in the same container, against the same server, seconds apart.

# 1. Python's ssl module uses OpenSSL's default paths, so it is already fixed.
docker exec rbpki-client11 python3 -c \
  'import urllib.request as u; print(u.urlopen("https://app.lab.example/").read().decode().strip())'

# 2. The same interpreter, pinned to the bundle captured before the install.
docker exec -e SSL_CERT_FILE=/tmp/pinned-bundle.crt rbpki-client11 python3 -c \
  'import urllib.request as u; print(u.urlopen("https://app.lab.example/").read().decode().strip())'

# 3. The same interpreter again, pointed at the single anchor file directly.
docker exec -e SSL_CERT_FILE=/opt/rbpki/root.crt rbpki-client11 python3 -c \
  'import urllib.request as u; print(u.urlopen("https://app.lab.example/").read().decode().strip())'

Call 1 prints the page body. Call 2 raises an SSL verification error and prints a traceback instead. Call 3 prints the page body again. Record all three outcomes verbatim, including whatever your Python build prints for call 2:

cd "$LAB"
{
  echo "system default paths:"
  docker exec rbpki-client11 python3 -c \
    'import urllib.request as u; print(u.urlopen("https://app.lab.example/").read().decode().strip())'
  echo "pinned to the pre-install bundle:"
  docker exec -e SSL_CERT_FILE=/tmp/pinned-bundle.crt rbpki-client11 python3 -c \
    'import urllib.request as u; print(u.urlopen("https://app.lab.example/").read().decode().strip())' \
    || echo "failed as expected"
  echo "pointed at the anchor file:"
  docker exec -e SSL_CERT_FILE=/opt/rbpki/root.crt rbpki-client11 python3 -c \
    'import urllib.request as u; print(u.urlopen("https://app.lab.example/").read().decode().strip())'
} > runtime-trust-report.txt 2>&1

Call 2 is the important one. The operating system store contains the anchor. The server is unchanged. The process still refuses, because it was told to read a different file, and that file predates the install. That is exactly the shape of the real failure: a library that ships its own bundle is permanently one install behind.

Java is the fourth common case and does not use an environment variable at all: it keeps its own keystore file, and an anchor has to be imported into that keystore before a JVM will accept it.

Task 8 — Prove that the anchor does not replace the chain

The anchor is installed and curl succeeds. Now serve only the leaf, with the issuing CA removed from what nginx sends, and watch it fail again.

cd "$LAB"
cp app.crt certs/leafonly.pem
sed -i 's#certs/fullchain.pem#certs/leafonly.pem#' conf.d/default.conf
docker exec rbpki-web11 nginx -t
docker exec rbpki-web11 nginx -s reload
sleep 1
docker exec rbpki-client11 curl -sS https://app.lab.example/ || true

The call fails again, with the same error 20 you saw in Task 5, on a client whose trust store contains the correct root. A trust anchor is the top of the chain, not the whole of it. The client still has to receive every certificate between the leaf and the anchor, and the only thing that can send them is the server. Restore the working configuration before continuing:

cd "$LAB"
sed -i 's#certs/leafonly.pem#certs/fullchain.pem#' conf.d/default.conf
docker exec rbpki-web11 nginx -s reload
sleep 1
docker exec rbpki-client11 curl -sS https://app.lab.example/

Task 9 — Capture the deliverables

cd "$LAB"

{
  echo "anchors before install: $(cat state/anchors.before)"
  docker exec rbpki-client11 sh -c 'ls /usr/local/share/ca-certificates'
} > trust-store-before.txt

{
  echo "anchors after install:"
  docker exec rbpki-client11 sh -c 'grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt'
  echo "client body:"
  docker exec rbpki-client11 curl -sS https://app.lab.example/
} > trust-store-after.txt

{
  echo "host copy:"
  openssl x509 -in root.crt -noout -fingerprint -sha256
  echo "container copy:"
  docker exec rbpki-client11 openssl x509 \
    -in /usr/local/share/ca-certificates/runbook-lab-root.crt -noout -fingerprint -sha256
} > anchor-fingerprint.txt

ls -l trust-store-before.txt trust-store-after.txt anchor-fingerprint.txt \
  runtime-trust-report.txt

You must create state/anchors.before yourself in Task 5 by redirecting the count into it. The fingerprint file is what lets you answer the question an auditor asks: not “is an anchor installed” but “is the anchor that is installed the one we issued”.

Validation

• openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt prints app.crt: OK and exits 0. A different message means Task 2 or Task 3 produced a certificate the chain cannot use.
• The anchor count in trust-store-after.txt is exactly one greater than the count in trust-store-before.txt. Equal counts mean the copy or the extension was wrong and update-ca-certificates silently skipped the file.
• docker exec rbpki-client11 curl -sS https://app.lab.example/ prints lab ok. Anything on stderr beginning curl: (60) means the install did not take effect for curl.
• The two fingerprints in anchor-fingerprint.txt are identical strings. Different fingerprints mean the container is trusting some other root.
• runtime-trust-report.txt records three outcomes: success, failure, success. Three successes mean call 2 did not receive the SSL_CERT_FILE value.
• The four deliverables exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-11/
├── root.key            root.crt            (the trust anchor)
├── srv-ca.key          srv-ca.crt          (the issuing CA)
├── app.key             app.crt             fullchain.pem
├── certs/              (fullchain.pem, app.key, leafonly.pem)
├── conf.d/default.conf www/index.html
├── state/
│   ├── containers.pre-lab
│   ├── networks.pre-lab
│   └── anchors.before
├── trust-store-before.txt
├── trust-store-after.txt
├── anchor-fingerprint.txt
└── runtime-trust-report.txt

You can now take any client that reports error 20 and decide, in one question, whether the fault is a missing anchor or a missing intermediate: ask whether the server sent a chain. If it did, install the anchor. If it did not, no amount of anchor installation will help. You can also answer, for a given process, which file it read to make that decision.

Troubleshooting

nginx -t reports that it cannot load the certificate key. The bind mount points at a path that does not exist on the host, so Docker created an empty directory there instead. Remove the container, check that $LAB/app.key is a file, and run the docker run command again.

The anchor count does not change after update-ca-certificates. The file must end in .crt and must be PEM. A DER file copied in with a .crt extension is skipped without an error. Convert it first with openssl x509 -inform DER -in in.der -out out.crt.

curl still fails after the anchor is installed. Check whether curl was given an explicit CA file elsewhere. curl --cacert and the CURL_CA_BUNDLE environment variable both override the system store, and either turns this lab into the Task 7 failure.

docker exec reports that the container is not running. The client container is started with sleep infinity. If the image was pulled but the command failed, docker logs rbpki-client11 shows why. Recreate it rather than restarting it, so the trust store starts clean.

Name resolution fails inside the client container. The --network-alias only works for containers on the same user-defined network. Confirm with docker inspect rbpki-client11 that its network is rbpki-net11.

Cleanup

LAB="$HOME/rbpki-lab-11"

# 1. Stop and forget the two lab containers.
docker rm -f rbpki-client11 rbpki-web11

# 2. Remove the lab network.
docker network rm rbpki-net11

# 3. Compare against the Task 1 capture: both diffs must be empty.
diff <(docker ps -a --format '{{.Names}}' | sort) "$LAB/state/containers.pre-lab"
diff <(docker network ls --format '{{.Name}}' | sort) "$LAB/state/networks.pre-lab"

# 4. Remove the lab directory, including every private key it generated.
rm -rf "$LAB"

The host trust store was never modified, so there is nothing to restore there. To confirm that, run grep -c 'BEGIN CERTIFICATE' /etc/ssl/certs/ca-certificates.crt on the host and check that /usr/local/share/ca-certificates contains no file named runbook-lab-root.crt. Both diffs in step 3 printing nothing is the assertion that the container and network inventory is back where it started.

Production notes

• Anchors reach production hosts through configuration management or a base image, never through a person with a shell. The file that installs the anchor should be in version control, and the fingerprint should be pinned there so a swapped file is a diff rather than a surprise.
• Distributing the anchor to the operating system store is the first of several jobs, not the whole job. Inventory the runtimes first, then decide whether the answer is an environment variable in a unit file, a rebuilt base image, or a keystore import step in the application’s own deployment.
• A container image with an anchor baked in inherits that anchor for the life of the image. When the root is rotated, every image that baked it in has to be rebuilt, which is the reason trust distribution and image lifecycle end up on the same project plan.
• Removing an anchor is as operationally significant as adding one, and it fails in the opposite direction: nothing breaks until a client tries to validate something. Schedule anchor removals with the same care as certificate replacements.

What You Learned

• A trust store belongs to a process, not to a host. The same machine can hold a dozen of them, and the operating system bundle is only the one that OpenSSL happens to compile in as a default.
• The anchor arrives out of band; the chain arrives on the wire. Installing a root never removes the server’s obligation to send its intermediates, and both faults report the same X.509 error 20.
• Measure the store, do not trust the command. An anchor count that did not change is the fastest possible detection of a silently skipped file, and it costs one grep -c.
• Environment variables that name a bundle are override switches, not additions, with one exception. SSL_CERT_FILE and REQUESTS_CA_BUNDLE replace the store for that process; NODE_EXTRA_CA_CERTS adds to Node’s built-in list and is read only at process start.