Lab 12: Configure and verify mutual TLS

Objective

By the end of this lab you will have a service that refuses to talk to anyone who cannot present a certificate issued by an authority you nominated. You will have proved that it refuses, twice, in the two ways that matter, and you will have read the reason out of the server log rather than guessing it from the client.

Mutual TLS is usually described as “the client authenticates too”. That sentence hides the interesting part. In ordinary TLS the server proves possession of a key by signing the handshake transcript, and the client checks that signature against a chain it can build to an anchor it already trusts. Mutual TLS runs the same machinery in the other direction: the server asks for a certificate, the client signs the transcript with its own key, and the server builds a chain to an anchor it nominated in its own configuration. There is no new cryptography. There is a new trust store, and it is the one almost nobody audits.

The discipline this lab teaches is that a client certificate authorises nothing by virtue of the name inside it. It authorises because of who signed it. You will prove that by presenting two certificates with byte-identical subjects and watching one succeed and one fail.

Architecture

A single root signs two issuing authorities that do different jobs. One issues the server certificate; the other issues client certificates. nginx is configured to trust only the client-issuing branch when it verifies a client, so possession of a server certificate from the same organisation confers no client rights at all. A completely separate root stands in for a stranger.

flowchart TD
    R["Lab Root CA"] --> SCA["Server issuing CA"]
    R --> CCA["Client issuing CA"]
    SCA --> APP["app.lab.example\nserverAuth"]
    CCA --> CLI["deploy-agent\nclientAuth"]
    F["Foreign Root CA"] --> FCLI["deploy-agent\nsame subject, other issuer"]
    APP --> NX["nginx\nssl_verify_client on"]
    CLI -- "accepted" --> NX
    FCLI -- "rejected" --> NX

Two branches hang off one root, and nginx is told about only one of them for the purpose of client verification. The foreign root is not in nginx’s client bundle at all, so the certificate it signed is rejected even though its subject is character-for-character the same as the accepted one. That contrast is the whole lesson: the subject is a label, and the issuer is the authorisation.

Requirements

OpenSSL 3.5.x on the host, for issuing every certificate and for the purpose checks.
Docker, with permission to create a network and run containers. The images nginx:1.29-alpine and alpine:3.22 are pulled on first use.
Outbound network access from the client container for one apk add that fetches curl and openssl.
About 20 MB of disk under $HOME.
No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. Every service it starts lives in a container that Cleanup destroys.

Scenario

An internal API is being moved off a shared network segment where anything that could route to it was implicitly allowed to call it. The replacement control is mutual TLS. Your job is to configure it, and then to answer the question the security reviewer will ask, which is not “is mTLS enabled” but “show me a rejection”.

Tasks

Task 1 — Record the starting state

LAB="$HOME/rbpki-lab-12"
rm -rf "$LAB"
mkdir -p "$LAB/www" "$LAB/state" "$LAB/conf.d" "$LAB/certs" "$LAB/client-kit"
cd "$LAB"

# Record what Cleanup must restore.
docker ps -a --format '{{.Names}}' | sort > "$LAB/state/containers.pre-lab"
docker network ls --format '{{.Name}}' | sort > "$LAB/state/networks.pre-lab"
wc -l "$LAB/state/containers.pre-lab" "$LAB/state/networks.pre-lab"

Both captures are needed because this lab leaves a network behind if you remove only the containers, and a stale bridge is the sort of debris that makes the next run of a lab fail for a reason that has nothing to do with the lab.

Task 2 — Build one root and two issuing authorities

cd "$LAB"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out root.crt

cat > ca.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

for CA in srv cli; do
  openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out "$CA-ca.key"
  openssl req -new -key "$CA-ca.key" -sha256 \
    -subj "/O=RunBook Academy Lab/CN=RunBook Lab $CA Issuing CA" -out "$CA-ca.csr"
  openssl x509 -req -in "$CA-ca.csr" -CA root.crt -CAkey root.key -CAcreateserial \
    -sha256 -days 1825 -extfile ca.ext -out "$CA-ca.crt"
done

ls -1 srv-ca.crt cli-ca.crt

Two issuing authorities under one root is not ceremony. It is what lets the next task hand nginx a client trust bundle that contains cli-ca and does not contain srv-ca, so a stolen server key cannot be replayed as a client identity.

Task 3 — Issue the server certificate and the client certificate

cd "$LAB"

cat > app.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

cat > client.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature
extendedKeyUsage=clientAuth
subjectAltName=DNS:deploy-agent.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
openssl req -new -key app.key -sha256 -subj "/CN=app.lab.example" -out app.csr
openssl x509 -req -in app.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
  -sha256 -days 90 -extfile app.ext -out app.crt

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out client.key
openssl req -new -key client.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=deploy-agent" -out client.csr
openssl x509 -req -in client.csr -CA cli-ca.crt -CAkey cli-ca.key -CAcreateserial \
  -sha256 -days 30 -extfile client.ext -out client.crt

cat app.crt srv-ca.crt > fullchain.pem
cat client.crt cli-ca.crt > client-fullchain.pem
cat root.crt cli-ca.crt > client-ca-bundle.pem
chmod 600 root.key srv-ca.key cli-ca.key app.key client.key

# Stage only what each side needs. The certificate authority keys stay here.
cp fullchain.pem app.key client-ca-bundle.pem certs/
cp root.crt client-fullchain.pem client.key client-kit/

Read-only / Safehost - one usage, in one direction, and no catch-all— Read this before configuring anything: it is the whole authorisation surface of the certificate.

$ openssl x509 -in client.crt -noout -ext basicConstraints,keyUsage,extendedKeyUsage,subjectAltName

The client certificate carries extendedKeyUsage=clientAuth and nothing else. Resist the urge to put both usages in one certificate so that one file works everywhere. The Baseline Requirements permit clientAuth alongside serverAuth in a public TLS certificate, and they forbid anyExtendedKeyUsage outright, but permission is not a recommendation: a certificate that is valid in both directions is a certificate that can be replayed in both directions.

Task 4 — Prove the purposes before you trust the configuration

Read-only / Safehost - the server certificate, checked as a server certificate— The chain is sound and the purpose matches.

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslserver app.crt

app.crt: OK

Read-only / Safehost - the same certificate, checked as a client certificate— Same chain, same signature, different question, different answer.

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslclient app.crt

error 26 at 0 depth lookup: unsuitable certificate purpose

Error 26 is not a chain failure. The signature verified, the dates are fine and the issuer was found. OpenSSL then applied the purpose test at depth 0, which is the leaf, and rejected it because serverAuth does not satisfy a request for sslclient. Run the mirror image of both commands against client.crt with -untrusted cli-ca.crt and record all four results:

cd "$LAB"
{
  echo "app.crt as sslserver:"
  openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslserver app.crt
  echo "app.crt as sslclient:"
  openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslclient app.crt
  echo "client.crt as sslclient:"
  openssl verify -CAfile root.crt -untrusted cli-ca.crt -purpose sslclient client.crt
  echo "client.crt as sslserver:"
  openssl verify -CAfile root.crt -untrusted cli-ca.crt -purpose sslserver client.crt
} > purpose-matrix.txt 2>&1
cat purpose-matrix.txt

Two of the four lines end in OK and two do not. That two-by-two table is the cheapest possible pre-flight check before a mutual TLS rollout, and it catches the single commonest cause of a rollout stalling: someone reused a server certificate as a client identity because it was the certificate they had.

Task 5 — Configure nginx to demand a client certificate

cd "$LAB"
printf 'mtls ok\n' > www/index.html

cat > conf.d/default.conf <<'EOF'
server {
    listen 443 ssl;
    server_name app.lab.example;

    ssl_certificate     /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/app.key;

    ssl_client_certificate /etc/nginx/certs/client-ca-bundle.pem;
    ssl_verify_client on;
    ssl_verify_depth 2;

    add_header X-Client-Verify $ssl_client_verify always;
    add_header X-Client-DN     $ssl_client_s_dn    always;
    add_header X-Client-Issuer $ssl_client_i_dn    always;

    root  /usr/share/nginx/html;
    index index.html;
}
EOF

docker network create rbpki-net12
docker run -d --name rbpki-web12 --network rbpki-net12 \
  --network-alias app.lab.example \
  -v "$LAB/conf.d:/etc/nginx/conf.d:ro" \
  -v "$LAB/certs:/etc/nginx/certs:ro" \
  -v "$LAB/www:/usr/share/nginx/html:ro" \
  nginx:1.29-alpine

docker exec rbpki-web12 nginx -t

Both mounts are directories. A bind mount of a single file breaks the moment that file is edited on the host, because an in-place edit writes a new inode and the container keeps reading the old one. Task 9 changes the configuration, so a directory mount is a correctness requirement rather than a preference.

Read-only / Safehost - exactly who is allowed to be a client— Two certificates, and the server-issuing authority is deliberately not one of them.

$ openssl storeutl -noout -certs client-ca-bundle.pem

ssl_client_certificate is the client trust store, and it does two jobs at once. It is the set of authorities nginx will build a client chain to, and the subject names in it are advertised to every connecting client in the CertificateRequest message so the client can choose which certificate to send. Both jobs argue for keeping it small.

Task 6 — Prove the accepted case

docker run -d --name rbpki-client12 --network rbpki-net12 \
  -v "$LAB/client-kit:/opt/rbpki:ro" alpine:3.22 sleep infinity
docker exec rbpki-client12 apk add --no-cache curl openssl

Read-only / Safeclient container - a request that carries a client certificate— The response headers carry the identity nginx decided on.

$ docker exec rbpki-client12 curl -sS -D - --cacert /opt/rbpki/root.crt --cert /opt/rbpki/client-fullchain.pem --key /opt/rbpki/client.key https://app.lab.example/

You are looking for the body mtls ok, an X-Client-Verify header whose value is SUCCESS, and an X-Client-DN header naming deploy-agent. Those three together are the proof that nginx did not merely accept the connection but resolved an identity from it. An application that wants to make an authorisation decision reads that identity out of a request header the proxy sets, which is why the header exists at all.

Now look at what the server advertises before any of that happens:

docker exec rbpki-client12 sh -c \
  'openssl s_client -connect app.lab.example:443 -servername app.lab.example \
     -CAfile /opt/rbpki/root.crt < /dev/null 2>&1 | head -40'

The output contains a section listing acceptable client certificate CA names. Those names come straight out of ssl_client_certificate. Record what you see: this is why a deployment that dumps two hundred authorities into that file produces a CertificateRequest message large enough to matter, and hands every anonymous client a list of your internal authority names.

What the server actually asks for, and what the client actually proves

In TLS 1.3 the server sends a CertificateRequest message after EncryptedExtensions, so unlike the ClientHello it is not sent in the clear. The client answers with its own Certificate message and then a CertificateVerify containing a signature over the handshake transcript, made with the private key belonging to the certificate it just sent. The server checks that signature and then, separately, builds and validates a chain from the presented certificate to an anchor in ssl_client_certificate.

Those are two independent checks and they fail for different reasons. A signature failure means the client does not hold the key. A chain failure means the client holds the key but the certificate does not come from an authority this server nominated. The server log distinguishes them; the client message usually does not, which is why every diagnosis in this lab starts on the server.

Note also what mutual TLS does not do. It authenticates the peer. It does not say what the peer may do. Mapping CN=deploy-agent to a set of permissions is an application concern, and a deployment that treats “presented a valid client certificate” as “may do anything” has built a single very large role.

Task 7 — Prove the rejection when nothing is presented

docker exec rbpki-client12 curl -sS -D - --cacert /opt/rbpki/root.crt \
  https://app.lab.example/ || true
docker logs --tail 5 rbpki-web12

The TLS handshake itself completes: the client verified the server, the server asked for a certificate and the client declined to send one. nginx then refuses the request at the HTTP layer and returns a client error response whose body states that no required certificate was sent. Record the exact status line and body you receive.

That distinction matters when you are reading a graph. A missing client certificate shows up as an HTTP error from a working TLS listener, not as a handshake failure, so a dashboard that only counts handshake errors will show nothing while every caller is being turned away.

Task 8 — Prove the rejection when the wrong authority signed it

Build a stranger’s authority and have it issue a certificate whose subject is identical to the accepted one:

cd "$LAB"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out foreign-root.key
openssl req -x509 -new -key foreign-root.key -sha256 -days 3650 \
  -subj "/O=Some Other Organisation/CN=Foreign Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out foreign-root.crt

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out foreign-client.key
openssl req -new -key foreign-client.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=deploy-agent" -out foreign-client.csr
openssl x509 -req -in foreign-client.csr -CA foreign-root.crt -CAkey foreign-root.key \
  -CAcreateserial -sha256 -days 30 -extfile client.ext -out foreign-client.crt

cp foreign-client.crt foreign-client.key client-kit/

# The subjects are identical. Confirm it before you test.
openssl x509 -in client.crt -noout -subject
openssl x509 -in foreign-client.crt -noout -subject

docker exec rbpki-client12 curl -sS --cacert /opt/rbpki/root.crt \
  --cert /opt/rbpki/foreign-client.crt --key /opt/rbpki/foreign-client.key \
  https://app.lab.example/ || true
docker logs --tail 5 rbpki-web12 2>&1 | tee -a "$LAB/nginx-client-verify.log"

curl reports a transport failure, because this time the handshake itself is abandoned rather than an HTTP response being returned. The useful information is in the nginx log, which records a client certificate verification error and the X.509 error number behind it. That number is the same family of number you saw from openssl verify in Task 4, and reading it is the difference between a five-minute diagnosis and an afternoon.

Task 9 — The verification depth experiment

ssl_verify_depth defaults to 1, which is enough for a client certificate signed directly by an authority in ssl_client_certificate and not enough for one signed by an intermediate beneath it. Task 5 set it to 2 because this lab uses a two-tier authority. Take it back out and watch:

cd "$LAB"
sed -i 's/^    ssl_verify_depth 2;/    ssl_verify_depth 1;/' conf.d/default.conf
docker exec rbpki-web12 nginx -s reload
sleep 1
docker exec rbpki-client12 curl -sS --cacert /opt/rbpki/root.crt \
  --cert /opt/rbpki/client-fullchain.pem --key /opt/rbpki/client.key \
  https://app.lab.example/ || true
docker logs --tail 5 rbpki-web12 2>&1 | tee -a "$LAB/nginx-client-verify.log"

Record what happens and what the server log says. Then restore the working value and confirm the accepted case works again:

cd "$LAB"
sed -i 's/^    ssl_verify_depth 1;/    ssl_verify_depth 2;/' conf.d/default.conf
docker exec rbpki-web12 nginx -s reload
sleep 1
docker exec rbpki-client12 curl -sS --cacert /opt/rbpki/root.crt \
  --cert /opt/rbpki/client-fullchain.pem --key /opt/rbpki/client.key \
  https://app.lab.example/

This is worth doing by hand once, because the symptom it produces is a certificate that is genuinely valid being rejected by a server that genuinely trusts its authority. Nothing about the certificate is wrong. A number in the server configuration is too small.

Task 10 — Capture the deliverables

cd "$LAB"

{
  echo "=== accepted: client certificate from cli-ca ==="
  docker exec rbpki-client12 curl -sS -D - --cacert /opt/rbpki/root.crt \
    --cert /opt/rbpki/client-fullchain.pem --key /opt/rbpki/client.key \
    https://app.lab.example/
  echo "=== rejected: no client certificate ==="
  docker exec rbpki-client12 curl -sS -D - --cacert /opt/rbpki/root.crt \
    https://app.lab.example/ || true
  echo "=== rejected: identical subject, foreign issuer ==="
  docker exec rbpki-client12 curl -sS --cacert /opt/rbpki/root.crt \
    --cert /opt/rbpki/foreign-client.crt --key /opt/rbpki/foreign-client.key \
    https://app.lab.example/ || true
} > mtls-evidence.txt 2>&1

docker logs rbpki-web12 2>&1 | grep -i 'client' >> nginx-client-verify.log || true
ls -l purpose-matrix.txt mtls-evidence.txt nginx-client-verify.log client-ca-bundle.pem

Validation

purpose-matrix.txt contains exactly two OK lines. If it contains four, one of the certificates carries both extended key usages and Task 3 was edited; if it contains none, the chain arguments are wrong.
The accepted request in mtls-evidence.txt returns the body mtls ok and an X-Client-Verify header whose value is SUCCESS. A value of NONE means curl did not send the certificate at all, usually because --cert named a file the container could not read.
The no-certificate attempt returns an HTTP client error response and no page body. A 200 here means ssl_verify_client is not on in the running configuration; re-check with docker exec rbpki-web12 nginx -T.
The foreign-issuer attempt fails at the transport layer and produces a fresh line in nginx-client-verify.log mentioning a client certificate verification error. An empty log file means you captured before the request.
client-ca-bundle.pem contains exactly two certificates and neither of them is srv-ca.crt. Check with openssl storeutl -noout -certs client-ca-bundle.pem.
All four deliverables exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-12/
├── root.key             root.crt
├── srv-ca.key           srv-ca.crt        (issues servers)
├── cli-ca.key           cli-ca.crt        (issues clients)
├── app.key              app.crt           fullchain.pem
├── client.key           client.crt        client-fullchain.pem
├── foreign-root.crt     foreign-client.crt
├── client-ca-bundle.pem                   (root + cli-ca, nothing else)
├── certs/               (what nginx mounts: server pair + client bundle)
├── client-kit/          (what the client mounts: no CA private keys)
├── conf.d/default.conf  www/index.html
├── purpose-matrix.txt
├── mtls-evidence.txt
└── nginx-client-verify.log

You can now answer the question a reviewer asks about any mutual TLS deployment: which file decides who may connect, what is in it today, and what does the server log look like when it says no. You can also tell, from a single X.509 error number, whether a rejected client holds the wrong key, the wrong purpose, or a certificate from an authority the server was never told about.

Troubleshooting

nginx -t fails on the ssl_client_certificate line. The bundle must be PEM and must contain at least one certificate. If client-ca-bundle.pem is empty, the cat in Task 3 ran before cli-ca.crt existed.

Every request returns X-Client-Verify: NONE. curl only sends a client certificate when both --cert and --key are supplied and the key matches. Confirm the pair with openssl pkey -in client.key -pubout | openssl sha256 and openssl x509 -in client.crt -noout -pubkey | openssl sha256; the two digests must be identical.

The accepted case fails with a server verification error instead. That is the client rejecting the server, not the reverse. --cacert must name root.crt, and fullchain.pem must contain the leaf followed by srv-ca.crt.

The foreign certificate is accepted. client-ca-bundle.pem has picked up the foreign root, or nginx is still running an older configuration. Print the running configuration with docker exec rbpki-web12 nginx -T rather than reading the file on disk.

docker logs shows nothing at all. nginx in the official image writes its error log to stderr, but only above the configured level. Add error_log /dev/stderr info; at the top of the configuration file and reload if you need more detail during the failure tasks.

Cleanup

LAB="$HOME/rbpki-lab-12"

# 1. Stop and forget the two lab containers.
docker rm -f rbpki-client12 rbpki-web12

# 2. Remove the lab network.
docker network rm rbpki-net12

# 3. Compare against the Task 1 capture: both diffs must print nothing.
diff <(docker ps -a --format '{{.Names}}' | sort) "$LAB/state/containers.pre-lab"
diff <(docker network ls --format '{{.Name}}' | sort) "$LAB/state/networks.pre-lab"

# 4. Remove the lab directory, including five private keys and two CA keys.
rm -rf "$LAB"

Nothing on the host was reconfigured: no trust store was edited, no service was installed, and no port outside the Docker network was bound. Confirm the restoration by checking that both diffs in step 3 printed nothing and that docker network ls no longer lists rbpki-net12.

Production notes

Client certificates need a renewal story before they need a rollout plan. A 30-day client certificate that no automation renews becomes a mass outage on a known date, and unlike a server certificate the failure is distributed across every caller rather than concentrated in one endpoint.
Keep the client trust bundle separate from the server trust bundle and keep both under change control. The review question for ssl_client_certificate is not “is it correct” but “who can add an authority to it”.
Terminating mutual TLS at a proxy and forwarding the identity in a header is normal, and it creates an obligation: the backend must be unreachable except through the proxy, and the proxy must strip any inbound copy of that header. A backend that trusts a header it did not set has replaced certificate authentication with a text field.
Revocation is the weak joint in every mutual TLS design. Decide early whether a compromised client certificate is handled by a certificate revocation list the server actually enforces, or by short lifetimes and fast reissuance. The second is easier to operate and is the honest answer for most internal estates.

What You Learned

The issuer authorises, the subject only labels. Two certificates with identical subjects got opposite answers because only one chained to an authority the server nominated.
Extended key usage is a direction, and OpenSSL will tell you which one. -purpose sslclient and -purpose sslserver turn an assumption into a two-line check, and error 26 names a purpose failure rather than a chain failure.
A missing client certificate and a bad one fail at different layers. One produces an HTTP error from a healthy listener; the other abandons the handshake. They need different alerts and different dashboards.
The reason lives on the server. The client is told that something went wrong and deliberately not told what, so every mutual TLS diagnosis begins by correlating the attempt with the server error log.

Objectives