Lab 7: Diagnose a broken certificate chain

Objective

This is the failure that fills incident channels: a service that worked yesterday now fails for some clients and not others, the certificate is not expired, the hostname is right, and nobody can say what changed. The cause is almost always that the server stopped sending its intermediate, and the reason it is hard is that the error message does not say so.

You will build the fault on purpose. A server that presents only its leaf certificate is a two-character edit away from a correct one, and it produces a client error string that is indistinguishable from a completely different fault - a client that does not trust your root. Both report verify result 20. Both say “unable to get local issuer certificate”. One is fixed on the server and one is fixed on the client, and choosing wrongly wastes the outage.

By the end you will have a measurement that separates them in a single command, a proof that transfers the investigation off the certificate and onto the server, a repair applied without restarting the service, and a regression check that will fail the next time somebody edits the configuration back.

Architecture

One nginx container serves app.lab.example on port 443, published to the host on 8447. It is started with ssl_certificate pointing at the leaf alone. Two clients on the host observe it, and a third observer - openssl verify reading files - stands outside the network entirely and is used to establish what the leaf certificate is worth on its own merits.

flowchart TD
    L["app.crt\nleaf only"] --> N["rbpki-web-07\nnginx presents 1 certificate"]
    N --> S["openssl s_client\nverify error num=20 then num=21"]
    N --> C["curl\nexit 60, verify result 20"]
    N --> X["extract served chain\nstoreutl counts them"]
    X --> D{"how many\ncertificates?"}
    D -- "1" --> F1["server fault\nadd the intermediate"]
    D -- "2 or more" --> F2["client fault\nadd the trust anchor"]

The diagram is the diagnostic method. Every path starts at the same client symptom, and the branch that matters is the count of certificates the server put on the wire. One certificate means the server is incomplete and the repair is on the server. Two or more means the server did its job and the client is missing the anchor at the top. Nothing in the client error message tells you which branch you are on, which is why the measurement is not optional.

Requirements

OpenSSL 3.5.x, providing openssl verify, openssl s_client and openssl storeutl.
Docker with permission to run containers and publish a port. The lab pulls nginx:1.29-alpine.
curl built against OpenSSL.
Host TCP port 8447 free.
No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. It does not modify the system trust store; every client command supplies its anchor explicitly.

Labs 4 and 5 built a two-tier certificate authority - a root, an issuing CA beneath it, and a leaf for app.lab.example - and lab 6 deployed that leaf correctly. This lab breaks the deployment on purpose. If you still have those artefacts, copy root.crt, srv-ca.crt, srv-ca.key, app.key and app.crt into $LAB/pki and skip Task 2. Otherwise Task 2 rebuilds an equivalent set.

Scenario

At 09:14 a monitoring probe starts failing against app.lab.example. The web team reports the site loads fine in their browsers. The batch job that calls the same endpoint from a container has been failing since the overnight deploy with a certificate error. The certificate expires in eleven weeks, so expiry is ruled out immediately, and the hostname in the certificate is correct.

You have two hypotheses and no way yet to choose between them. Either the server has stopped sending part of its chain, or the failing clients have lost a trust anchor the working ones still have. This lab is the sequence that decides it, and the sequence is short.

Tasks

Task 1 — Prepare the lab directory and record the starting state

LAB="$HOME/rbpki-lab-07"
rm -rf "$LAB"
mkdir -p "$LAB/pki" "$LAB/site" "$LAB/html"
cd "$LAB"

{
  echo "--- containers before the lab"
  docker ps -a --format '{{.Names}}'
} > "$LAB/state.pre-lab"

docker rm -f rbpki-web-07 2>/dev/null || true

echo 'lab ok' > "$LAB/html/index.html"
cat "$LAB/state.pre-lab"

Recording the inventory first means Cleanup is a comparison rather than a recollection. The page body gives the clients an unambiguous success signal: if lab ok is returned then verification passed and the request was served, which no partially-successful handshake can fake.

Task 2 — Recreate the certificate authority and the leaf

Skip this task if you copied the artefacts from lab 5 or lab 6.

LAB="$HOME/rbpki-lab-07"
cd "$LAB/pki"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out root.crt

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out srv-ca.key
openssl req -new -key srv-ca.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA" -out srv-ca.csr
cat > srv-ca.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF
openssl x509 -req -in srv-ca.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 1825 -extfile srv-ca.ext -out srv-ca.crt

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
openssl req -new -key app.key -sha256 -subj "/CN=app.lab.example" -out app.csr
cat > app.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF
openssl x509 -req -in app.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
  -sha256 -days 90 -extfile app.ext -out app.crt

cat app.crt srv-ca.crt > fullchain.pem
chmod 600 root.key srv-ca.key app.key
openssl x509 -in app.crt -noout -subject -issuer

Note that you have deliberately built both files: app.crt, the leaf on its own, and fullchain.pem, the leaf followed by its issuer. The fault you are about to introduce is choosing the first one. In a real estate the two files sit side by side in the same directory with names that differ by five characters, which is precisely why this mistake is so common.

Task 3 — Introduce the fault and start the service

LAB="$HOME/rbpki-lab-07"

cat > "$LAB/site/default.conf" <<'EOF'
server {
    listen 443 ssl;
    server_name app.lab.example;

    # THE FAULT: the leaf alone, with no issuer beside it.
    ssl_certificate     /etc/nginx/certs/app.crt;
    ssl_certificate_key /etc/nginx/certs/app.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    root /usr/share/nginx/html;
    index index.html;
}
EOF

docker run -d --name rbpki-web-07 -p 8447:443 \
  -v "$LAB/site:/etc/nginx/conf.d:ro" \
  -v "$LAB/pki:/etc/nginx/certs:ro" \
  -v "$LAB/html:/usr/share/nginx/html:ro" \
  nginx:1.29-alpine

sleep 2
docker exec rbpki-web-07 nginx -t
docker ps --filter name=rbpki-web-07 --format '{{.Names}} {{.Status}}'

nginx -t reports the configuration test successful and the container stays up. That is the first lesson of the lab: nothing on the server side considers this an error. The server was asked to present a certificate and it is presenting one. Completeness of the chain is a client-side judgement, and the server has no opinion about it.

Task 4 — Reproduce the failure from two clients

LAB="$HOME/rbpki-lab-07"
cd "$LAB/pki"

# Client one: openssl, with the correct trust anchor supplied.
openssl s_client -connect 127.0.0.1:8447 \
  -servername app.lab.example -CAfile root.crt -showcerts </dev/null 2>&1 | head -12

# Client two: curl, with the same correct trust anchor supplied.
curl -sS --resolve app.lab.example:8447:127.0.0.1 --cacert root.crt \
  https://app.lab.example:8447/ 2>&1 | head -1

Read-only / Safehost - what openssl says about a server that sent one certificate— Two different verify errors from one connection, and only one certificate listed.

$ openssl s_client -connect 127.0.0.1:8447 -servername app.lab.example -CAfile root.crt -showcerts

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

Illustrative output

Read-only / Safehost - what curl says about the same connection, holding the same anchor— A different tool, the same root cause, an error string that names neither the server nor the intermediate.

$ curl -sS --resolve app.lab.example:8447:127.0.0.1 --cacert root.crt https://app.lab.example:8447/

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)

Illustrative output

Read the openssl output carefully, because it contains the whole problem. The chain listing shows one entry. Entry 0 is the leaf, and its issuer line names the issuing CA - so the server has told you what signed it, and has not given you that certificate. The client holds the root, not the intermediate, and has no way to bridge the gap.

Now read the curl output and notice what it does not say. It does not say the intermediate is missing. It does not say how many certificates arrived. It reports verify result 20, which is the same code you would get from a server sending a perfect chain to a client that lacks your root. Two entirely different faults, one string. This is why the next task exists.

Why one connection produced verify error 20 and then verify error 21

Path building starts at the leaf and walks upward. For each certificate the verifier reads the issuer name and the authority key identifier, then looks for a certificate whose subject and subject key identifier match, first among the certificates the server sent, then among the trusted anchors it was configured with.

Error 20, unable to get local issuer certificate, is raised at a specific depth. Here it is raised at depth 0, meaning the verifier was working on the leaf and could not find the certificate that issued it anywhere - not in the presented chain, because the server sent only one, and not in the anchor set, because the anchor is the root and the leaf’s issuer is the intermediate.

Error 21, unable to verify the first certificate, is the summary verdict for the presented chain as a whole. It is emitted once, after path building has given up. Seeing 20 at depth 0 followed by 21 is the signature of a chain that stops one certificate too early. Seeing 20 at a depth greater than zero means the server sent some of its issuers but not all of them, which is the same class of fault further up. Seeing error 2, unable to get issuer certificate, at the top depth means the chain was complete and terminated in something you do not trust - a client-side problem, not a server-side one.

Task 5 — Measure the chain the server actually sent

LAB="$HOME/rbpki-lab-07"
cd "$LAB/pki"

# Pull every certificate the server put on the wire into a file.
openssl s_client -connect 127.0.0.1:8447 -servername app.lab.example \
  -showcerts </dev/null 2>/dev/null \
  | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/' > "$LAB/served-chain-before.pem"

# Count them, and read what each one is.
openssl storeutl -noout -certs "$LAB/served-chain-before.pem"
grep -c 'BEGIN CERTIFICATE' "$LAB/served-chain-before.pem"

This is the measurement the whole lab turns on. The awk range extracts every PEM block from the -showcerts output, and the count that comes back is the number of certificates the server chose to send. One certificate for a leaf issued by an intermediate is always wrong. The rule is simple and mechanical: a server must send its own certificate plus every certificate between it and a trust anchor, and it must not send the anchor.

With that number in hand the branch in the Architecture diagram resolves. A count of one sends you to the server’s configuration. A count of two or more, with the same client error, sends you to the client’s trust store instead, and you would be diagnosing a different lab.

Task 6 — Exonerate the leaf certificate

LAB="$HOME/rbpki-lab-07"
cd "$LAB/pki"

# 1. The leaf with nothing but the anchor: reproduces the client failure.
openssl verify -CAfile root.crt app.crt

# 2. The leaf verified against exactly what the server sent: same failure,
#    which proves the server's chain adds nothing.
openssl verify -CAfile root.crt -untrusted "$LAB/served-chain-before.pem" app.crt

# 3. The leaf with its real issuer supplied: the certificate is fine.
openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt

Read-only / Safehost - the same leaf, judged three ways— The certificate is sound; only the set of intermediates supplied to the verifier changes.

$ openssl verify -CAfile root.crt app.crt; openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt

error 20 at 0 depth lookup: unable to get local issuer certificate
app.crt: OK

Illustrative output

The third call prints app.crt: OK, and that single line moves the investigation. The leaf is correctly signed, in date, and chains to the root through the issuing CA. Nothing needs to be reissued, nobody needs to call the CA, and the change that fixes this touches no certificate at all.

The second call is the one people skip and the one that closes the argument. By feeding the verifier precisely the bytes the server transmitted, you demonstrate that the server’s contribution to path building is empty. It is the difference between suspecting the server and proving it.

Task 7 — Repair the server and reload without restarting

LAB="$HOME/rbpki-lab-07"

# Point ssl_certificate at the leaf-plus-issuer bundle.
cat > "$LAB/site/default.conf" <<'EOF'
server {
    listen 443 ssl;
    server_name app.lab.example;

    ssl_certificate     /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/app.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    root /usr/share/nginx/html;
    index index.html;
}
EOF

docker exec rbpki-web-07 nginx -t
docker exec rbpki-web-07 nginx -s reload
sleep 2

cd "$LAB/pki"
openssl s_client -connect 127.0.0.1:8447 -servername app.lab.example \
  -CAfile root.crt </dev/null 2>/dev/null | \
  grep -E 'Certificate chain|^ [0-9] s:|^   i:|^New,|^Protocol|Verify return code'

curl -sS --resolve app.lab.example:8447:127.0.0.1 --cacert root.crt \
  https://app.lab.example:8447/

Configuration changehost - the repaired server, read from the wire— Two certificates presented, the path terminates in the trusted root, verification returns zero.

$ openssl s_client -connect 127.0.0.1:8447 -servername app.lab.example -CAfile root.crt

Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Illustrative output

Two entries now. Entry 1 is the issuing CA and its issuer line names the root, which the client holds, so path building terminates and the verdict flips to Verify return code: 0 (ok). curl returns lab ok on the same connection. No certificate was reissued and no client was modified.

The reload rather than a restart matters more in production than it does here. nginx -s reload re-reads the configuration and the certificate files in new worker processes while the old ones finish their in-flight requests, so the repair lands without dropping connections. Pair it always with nginx -t first: a reload with a syntax error leaves the old workers running and the fix silently unapplied, which is its own confusing incident.

Task 8 — Capture the deliverables and add a regression check

LAB="$HOME/rbpki-lab-07"
cd "$LAB/pki"

openssl s_client -connect 127.0.0.1:8447 -servername app.lab.example \
  -showcerts </dev/null 2>/dev/null \
  | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/' > "$LAB/served-chain-after.pem"

cat > "$LAB/check-served-chain.sh" <<'EOF'
#!/bin/sh
# Fail if a TLS endpoint presents fewer certificates than it should.
# Usage: check-served-chain.sh HOSTPORT SERVERNAME MINIMUM
set -eu
ENDPOINT="$1"
NAME="$2"
MINIMUM="$3"
COUNT=$(openssl s_client -connect "$ENDPOINT" -servername "$NAME" \
          -showcerts </dev/null 2>/dev/null | grep -c 'BEGIN CERTIFICATE')
echo "$NAME presented $COUNT certificate(s); minimum is $MINIMUM"
[ "$COUNT" -ge "$MINIMUM" ] || exit 1
EOF
chmod +x "$LAB/check-served-chain.sh"
"$LAB/check-served-chain.sh" 127.0.0.1:8447 app.lab.example 2

{
  echo "=== before: certificates presented"
  grep -c 'BEGIN CERTIFICATE' "$LAB/served-chain-before.pem"
  echo "=== after: certificates presented"
  grep -c 'BEGIN CERTIFICATE' "$LAB/served-chain-after.pem"
  echo "=== the leaf, verified against the anchor alone"
  openssl verify -CAfile root.crt app.crt 2>&1 || true
  echo "=== the leaf, verified with its issuer supplied"
  openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt
} > "$LAB/chain-diagnosis.txt"

ls -l "$LAB/served-chain-before.pem" "$LAB/served-chain-after.pem" \
      "$LAB/chain-diagnosis.txt" "$LAB/check-served-chain.sh"

The check takes a minimum rather than an exact count on purpose. A CA that introduces a cross-signed intermediate will legitimately increase the number of certificates presented, and a check that demanded exactly two would fail on a correct change. What must never happen is the count dropping below the number of issuers between your leaf and a trust anchor.

Validation

grep -c 'BEGIN CERTIFICATE' served-chain-before.pem returns 1, and the same count against served-chain-after.pem returns 2. A pass here requires the first number to be wrong.
Against the misconfigured server, openssl s_client prints verify error:num=20:unable to get local issuer certificate followed by verify error:num=21:unable to verify the first certificate, and lists exactly one entry under Certificate chain.
Against the misconfigured server, curl exits 60 and reports verify result 20.
openssl verify -CAfile root.crt app.crt prints error 20 at 0 depth lookup: unable to get local issuer certificate and exits 2.
openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt prints app.crt: OK and exits 0.
After the repair, openssl s_client lists two entries and ends with Verify return code: 0 (ok), and curl returns lab ok.
check-served-chain.sh 127.0.0.1:8447 app.lab.example 2 exits 0 after the repair, and exits 1 if you restore the faulty configuration and reload.
The four deliverables exist and are non-empty.

A failed validation is usually one of two things. If the count before the repair is 2, the configuration written in Task 3 did not take effect - confirm ssl_certificate reads app.crt and that the container was started after the file was written. If openssl verify with -untrusted srv-ca.crt still fails, the leaf and the intermediate do not belong together, which means Task 2 was run partially and app.crt was signed by a different issuing CA than the one now on disk.

Expected Outcome

$HOME/rbpki-lab-07/
├── html/
│   └── index.html
├── pki/
│   ├── app.crt
│   ├── app.key
│   ├── fullchain.pem
│   ├── root.crt
│   ├── root.key
│   ├── srv-ca.crt
│   └── srv-ca.key
├── site/
│   └── default.conf
├── chain-diagnosis.txt
├── check-served-chain.sh
├── served-chain-after.pem
├── served-chain-before.pem
└── state.pre-lab

You can now take a client error that names no component and, in one command, decide whether the repair belongs on the server or on the client. You can prove a certificate innocent before anyone starts a reissue that would not have helped, and you can leave behind a check that fails the next time the chain shortens.

Troubleshooting

openssl s_client prints the chain but no verify errors. The intermediate is already in your default trust store from earlier work, so your client can repair the chain itself. Add -CAfile root.crt -no-CAstore -no-CApath to force the client to use only the anchor you name.

The extracted chain file is empty. s_client was not given -showcerts, or the connection failed before any certificate arrived. Run the command without the awk pipeline and read what it actually printed.

docker exec rbpki-web-07 nginx -s reload reports the container is not running. nginx exited after the configuration was rewritten. docker logs rbpki-web-07 names the file it could not read; the usual cause is a path in default.conf that does not exist under /etc/nginx/certs/.

The repair does not take effect after a reload. The configuration directory is bind-mounted as a directory precisely so this works. If you replaced the mount with a single file, editing that file on the host usually creates a new inode and the container keeps reading the old one. Mount the directory.

check-served-chain.sh reports 0 certificates. The endpoint argument is wrong or the service is down. The script deliberately does not distinguish a broken chain from a dead port, so confirm the service answers at all before reading anything into the count.

Cleanup

LAB="$HOME/rbpki-lab-07"

# 1. Stop and forget the service.
docker rm -f rbpki-web-07 2>/dev/null || true

# 2. Compare against the inventory recorded in Task 1.
cat "$LAB/state.pre-lab"
echo "--- containers now"
docker ps -a --format '{{.Names}}'

# 3. Remove the lab directory, keys included.
rm -rf "$LAB"

# 4. Assert the host is as you found it.
docker ps -a --format '{{.Names}}' | grep -c '^rbpki-web-07$' || echo "container gone"
test -d "$LAB" && echo "LAB DIRECTORY STILL PRESENT" || echo "lab directory gone"

The two container inventories must differ only by the removal of rbpki-web-07. Because this lab never wrote to the system trust store and never edited /etc/hosts, there is nothing else to restore: both client tools were given their anchor on the command line for exactly that reason. The two assertions in step 4 must both report the resource gone.

Production notes

Put the chain-count check in the same monitor that watches expiry. Expiry monitoring alerts weeks ahead; a chain that shortens breaks instantly at the next deploy, so it needs a probe on the service rather than a scan of a file.
The commonest real-world cause is not a hand-edited configuration. It is an automation change that starts writing cert.pem where it used to write fullchain.pem, or a certificate manager whose template selects the wrong output. Check what the deployment tooling wrote before blaming the person who reloaded.
When a CA rotates its intermediate, a server that pins the old file keeps serving a valid but stale chain for a while and then breaks. Renewal must replace the whole bundle, not just the leaf.
Record the certificate count in the change ticket alongside the fingerprint. It costs one command and turns “the chain looked fine” into a number a reviewer can check.

What You Learned

One client error string covers two opposite faults. Verify result 20 means the path could not be completed; it does not say whether the server sent too little or the client trusts too little.
Counting the certificates on the wire is the discriminator. openssl s_client -showcerts piped into a PEM extractor answers in one command what the error message never will.
Error 20 at depth 0 followed by error 21 is the signature of a short chain. Error 2 at the top depth is the signature of a missing anchor, and the two lead to different teams.
openssl verify -untrusted exonerates the certificate. Verifying the leaf against exactly what the server sent, then against the real issuer, proves the certificate is sound and moves the investigation to the configuration.
The repair belongs on the server, once, not on every client for ever. Distributing an intermediate to trust stores works and makes the next rotation a fleet-wide project.
A browser is not evidence. Clients that fetch missing issuers themselves will hide this fault from exactly the people most likely to be asked whether the site is up.

Objectives

Objective

Architecture

Requirements

Scenario

Tasks

Task 1 — Prepare the lab directory and record the starting state

Task 2 — Recreate the certificate authority and the leaf

Task 3 — Introduce the fault and start the service

Task 4 — Reproduce the failure from two clients

Task 5 — Measure the chain the server actually sent

Task 6 — Exonerate the leaf certificate

Task 7 — Repair the server and reload without restarting

Task 8 — Capture the deliverables and add a regression check

Validation

Expected Outcome

Troubleshooting

Cleanup

Production notes

What You Learned

Deliverables

Verification status