Lab 6: Configure TLS on a real service and verify it from a real client

Objective

By the end of this lab you will have a real TLS service running on a real port, presenting a certificate you issued yourself, and you will have proved it correct from outside the machine that serves it. That last clause is the whole point. A certificate deployment is not finished when the files are in place. It is finished when a client that knows nothing about your filesystem completes a handshake and reports that the identity it was shown is the identity it asked for.

You will assemble the bundle a TLS server actually needs, which is not the same thing as the file the CA sent you. A leaf certificate on its own is an orphan: it names an issuer that the client has almost certainly never heard of. The server has to carry the intermediate along with it. You will see that requirement fail at the file level first, where it is cheap and unambiguous, before you put it on a socket.

You will then verify the running service twice, with two tools that share almost no code path at the policy layer, and you will finish by proving that the certificate on the wire is the one whose private key the server holds. Those three checks - chain, name, key possession - are the entire content of a TLS deployment review.

Architecture

One container terminates TLS for app.lab.example on port 443, published to the host on 8443. The certificate material is mounted read-only from the lab directory, so nothing is baked into an image and nothing outlives the container. Both clients run on the host: openssl s_client speaks directly to the published port, and curl reaches the same port under the correct name using --resolve, so no entry is ever written to /etc/hosts.

flowchart LR
    K["app.key\nprivate key"] --> N["rbpki-web-06\nnginx 1.29-alpine"]
    F["fullchain.pem\nleaf + issuing CA"] --> N
    N -- "TLS on 127.0.0.1:8443" --> S["openssl s_client\nreads the chain"]
    N -- "TLS on 127.0.0.1:8443" --> C["curl --resolve\napplies policy"]
    R["root.crt\ntrust anchor"] --> S
    R --> C

The diagram splits the deployment into the two halves that fail independently. On the left, the server holds a private key and a chain file; if either is wrong the server either refuses to start or presents something the client cannot use. On the right, two clients each hold a trust anchor and a hostname they expect; if either of those does not line up with what the server sent, the handshake is completed at the protocol level and then rejected at the policy level. Reading a failure correctly starts with knowing which half you are looking at.

Requirements

• OpenSSL 3.5.x on the host, providing openssl genpkey, openssl req, openssl x509, openssl verify and openssl s_client.
• Docker with permission to run containers, create a user-defined network and publish a port. The lab pulls nginx:1.29-alpine, roughly 50 MB.
• curl built against OpenSSL, and GNU date for the validity arithmetic in later labs of this group.
• Host TCP port 8443 free. Nothing else in the lab binds a privileged port.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. It creates one Docker network, one container and one directory under your home, all prefixed rbpki-.

Labs 4 and 5 built a two-tier certificate authority and issued a server certificate from it: a root CA, an issuing CA signed by that root, and a leaf for app.lab.example carrying a subject alternative name and the serverAuth extended key usage. This lab consumes those artefacts. If you still have that lab directory, copy root.crt, srv-ca.crt, srv-ca.key, app.key and app.crt into $LAB/pki in Task 1 and skip Task 2. If you do not, Task 2 rebuilds an equivalent set in under a minute.

Scenario

A service that has been running on plain HTTP inside the estate is being moved behind TLS. The internal CA has issued you a certificate for app.lab.example and handed back three files. The change window is short, and the change is being reviewed by someone who will not accept “the files are in the right place” as evidence.

Your job is to produce the evidence instead: a transcript from two clients showing the chain the server presents, the protocol and cipher it negotiated, the verification result, and a digest comparison proving the served certificate belongs to the key the server is holding. That transcript is the deliverable. The running service is merely how you obtain it.

Tasks

Task 1 — Prepare the lab directory and record the starting state

LAB="$HOME/rbpki-lab-06"
rm -rf "$LAB"
mkdir -p "$LAB/pki" "$LAB/site" "$LAB/html"
cd "$LAB"

# Record what Cleanup must restore: the containers and networks that
# existed before this lab created any of its own.
{
  echo "--- containers before the lab"
  docker ps -a --format '{{.Names}}'
  echo "--- networks before the lab"
  docker network ls --format '{{.Name}}'
} > "$LAB/state.pre-lab"

# Remove any container or network left by a previous run of THIS lab.
docker rm -f rbpki-web-06 2>/dev/null || true
docker network rm rbpki-net-06 2>/dev/null || true
docker network create rbpki-net-06

echo 'lab ok' > "$LAB/html/index.html"
cat "$LAB/state.pre-lab"

The state.pre-lab file is the reference Cleanup is measured against. Recording it before you create anything is the difference between a cleanup you can prove and a cleanup you hope worked. The single-line page body gives the clients something unambiguous to fetch: if lab ok comes back, the handshake completed and the request was served, with no room for a cached error page to impersonate success.

Task 2 — Recreate the certificate authority and the leaf certificate

Skip this task if you copied the artefacts from lab 5.

LAB="$HOME/rbpki-lab-06"
cd "$LAB/pki"

# Root CA: a 4096-bit RSA key, self-signed, allowed one level of
# subordinate CA beneath it.
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out root.crt

# Issuing CA: signed by the root, forbidden from creating further CAs.
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out srv-ca.key
openssl req -new -key srv-ca.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA" \
  -out srv-ca.csr
cat > srv-ca.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF
openssl x509 -req -in srv-ca.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 1825 -extfile srv-ca.ext -out srv-ca.crt

# Leaf: an EC P-256 key, a request naming the service, and an extension
# file that carries the identity the CA is willing to assert.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
openssl req -new -key app.key -sha256 -subj "/CN=app.lab.example" -out app.csr
cat > app.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF
openssl x509 -req -in app.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
  -sha256 -days 90 -extfile app.ext -out app.crt

chmod 600 app.key srv-ca.key root.key
openssl x509 -in app.crt -noout -subject -issuer -dates

Three key pairs, two of them belonging to certificate authorities and one to the service. The -extfile on each signing step is where the CA states what it is willing to assert. Nothing in the request compels it: a certificate signing request carries a public key and a proposed identity, and the CA is free to issue something narrower. That is why the identity a certificate ends up with is read from the certificate, never from the request that asked for it.

$ openssl x509 -in app.crt -noout -subject -issuer -serial -dates

subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

The serial and the two dates will differ on your run. What must match is the shape: a subject naming the service, an issuer naming the intermediate rather than the root, and a validity window that has already started.

Task 3 — Build the deployment bundle, and prove why the intermediate has to travel with it

LAB="$HOME/rbpki-lab-06"
cd "$LAB/pki"

# The leaf on its own. The client is asked to find the issuer itself.
openssl verify -CAfile root.crt app.crt
echo "exit code with no intermediate: $?"

# The same leaf, with the intermediate supplied alongside it.
openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt
echo "exit code with the intermediate: $?"

# The bundle a TLS server presents: leaf first, then every issuer above
# it, stopping short of the trust anchor.
cat app.crt srv-ca.crt > fullchain.pem
openssl storeutl -noout -certs fullchain.pem

The two openssl verify calls differ by one flag and describe the entire problem. The trust anchor is the same in both. The leaf is the same in both. What changes is whether the verifier was handed the certificate that sits between them.

$ openssl verify -CAfile root.crt app.crt; openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt

error 20 at 0 depth lookup: unable to get local issuer certificate
app.crt: OK

error 20 says the verifier reached depth 0, the leaf itself, and could not find the certificate that issued it. It is not saying the leaf is malformed, expired or untrusted. It is saying the chain has a hole in it. Adding -untrusted srv-ca.crt fills the hole and the same leaf verifies cleanly, which is the proof that the leaf was never the problem.

A TLS server has no -untrusted flag. Its only way to hand the client the intermediate is to send it during the handshake, and the way you tell nginx to do that is to point ssl_certificate at a file containing the leaf followed by its issuer. The root is deliberately not in that file: the client either already trusts the root or it does not, and sending a copy of it changes nothing.

Task 4 — Configure nginx and start the service

LAB="$HOME/rbpki-lab-06"

cat > "$LAB/site/default.conf" <<'EOF'
server {
    listen 443 ssl;
    server_name app.lab.example;

    ssl_certificate     /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/app.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    root /usr/share/nginx/html;
    index index.html;
}
EOF

docker run -d --name rbpki-web-06 \
  --network rbpki-net-06 --network-alias app.lab.example \
  -p 8443:443 \
  -v "$LAB/site:/etc/nginx/conf.d:ro" \
  -v "$LAB/pki:/etc/nginx/certs:ro" \
  -v "$LAB/html:/usr/share/nginx/html:ro" \
  nginx:1.29-alpine

sleep 2
docker exec rbpki-web-06 nginx -t
docker ps --filter name=rbpki-web-06 --format '{{.Names}} {{.Status}}'

ssl_certificate takes the fullchain and ssl_certificate_key takes the private key; they are two directives because they are two files, and the commonest deployment error in this lab family is pointing the first one at app.crt out of habit. The configuration directory is mounted rather than the single file, which matters later: replacing a bind-mounted file usually replaces the inode and the container keeps reading the old one, whereas replacing a file inside a bind-mounted directory is visible immediately.

nginx -t is a configuration test, not a certificate test. It will pass on a chain in the wrong order, on a certificate that expired last year, and on a certificate for an entirely different hostname. Treating it as deployment verification is how broken certificates reach production during a change window that looked clean.

Task 5 — Verify from openssl s_client

LAB="$HOME/rbpki-lab-06"
cd "$LAB/pki"

openssl s_client -connect 127.0.0.1:8443 \
  -servername app.lab.example \
  -CAfile root.crt </dev/null 2>/dev/null | tee "$LAB/s_client.out" | \
  grep -E 'Certificate chain|^ [0-9] s:|^   i:|^New,|^Protocol|Verify return code'

$ openssl s_client -connect 127.0.0.1:8443 -servername app.lab.example -CAfile root.crt

Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Read the chain block from the bottom up. Entry 1 is the issuing CA, and its issuer line names the root. Entry 0 is the service, and its issuer line names entry 1. The chain therefore terminates in something the -CAfile holds, and Verify return code: 0 (ok) is the client saying so.

-servername sends the Server Name Indication extension. Without it a server hosting several names on one address has no way to know which certificate to present, and you end up debugging a certificate that was never meant for you. Send it every time, and make it match the name you are claiming to be.

Task 6 — Verify from curl, which asks a different question

LAB="$HOME/rbpki-lab-06"
cd "$LAB/pki"

# --resolve maps the name to the published port for this call only.
# Nothing is written to /etc/hosts.
curl -sS --resolve app.lab.example:8443:127.0.0.1 \
     --cacert root.crt \
     https://app.lab.example:8443/

# The same call with the trust anchor withheld, to see the other verdict.
curl -sS --resolve app.lab.example:8443:127.0.0.1 \
     https://app.lab.example:8443/ || true

$ curl -sS --resolve app.lab.example:8443:127.0.0.1 --cacert root.crt https://app.lab.example:8443/

lab ok

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)

The first call returns the page body, which means curl completed the handshake, built a chain to the anchor in root.crt, and matched app.lab.example against the certificate’s subject alternative name. The second call fails with exit code 60 because the private root is not in the system trust store, so the chain the server sent terminates in an issuer curl has no reason to believe.

That second failure is worth dwelling on. The server did nothing different. The certificate did not change. Only the client’s trust set changed, and the whole verdict changed with it. Verification is a property of the conversation between a specific server and a specific client, which is why “it works on my machine” is a meaningless statement about TLS.

Task 7 — Prove the served certificate matches the key the server holds

LAB="$HOME/rbpki-lab-06"
cd "$LAB/pki"

# The public key inside the certificate the server actually sent.
openssl s_client -connect 127.0.0.1:8443 -servername app.lab.example </dev/null 2>/dev/null \
  | openssl x509 -pubkey -noout | openssl sha256

# The public key derived from the private key on disk.
openssl pkey -in app.key -pubout | openssl sha256

$ openssl pkey -in app.key -pubout | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Two identical lines. The digest itself is meaningless as a value and will differ every time you regenerate the key; what carries information is that the two lines agree. If they disagree, the server is presenting a certificate whose private key it does not hold, and the handshake would already have failed - but running the comparison against files, before a restart, catches the same mistake without an outage in between.

This is the check to run after any key rotation, because the failure it catches is silent until the service restarts. A new certificate copied into place beside the old key looks entirely correct in a directory listing.

Task 8 — Capture the deliverables

LAB="$HOME/rbpki-lab-06"
cd "$LAB/pki"

{
  echo "=== openssl s_client, trust anchor supplied"
  openssl s_client -connect 127.0.0.1:8443 -servername app.lab.example \
    -CAfile root.crt </dev/null 2>/dev/null | \
    grep -E 'Certificate chain|^ [0-9] s:|^   i:|^New,|^Protocol|Verify return code'
  echo
  echo "=== curl, trust anchor supplied"
  curl -sS --resolve app.lab.example:8443:127.0.0.1 --cacert root.crt \
    https://app.lab.example:8443/
  echo
  echo "=== curl, trust anchor withheld (expected failure)"
  curl -sS --resolve app.lab.example:8443:127.0.0.1 \
    https://app.lab.example:8443/ 2>&1 | head -1
} > "$LAB/verify-report.txt"

{
  echo "served certificate public key:"
  openssl s_client -connect 127.0.0.1:8443 -servername app.lab.example </dev/null 2>/dev/null \
    | openssl x509 -pubkey -noout | openssl sha256
  echo "private key public half:"
  openssl pkey -in app.key -pubout | openssl sha256
} > "$LAB/served-key-match.txt"

ls -l "$LAB/pki/fullchain.pem" "$LAB/site/default.conf" \
      "$LAB/verify-report.txt" "$LAB/served-key-match.txt"

The report deliberately contains a failure as well as a success. A verification transcript that only shows the happy path proves the reviewer nothing about whether the check has any teeth.

Validation

• openssl storeutl -noout -certs fullchain.pem reports two certificates, and openssl x509 -in fullchain.pem -noout -subject prints subject=CN=app.lab.example.
• openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt prints app.crt: OK and exits 0.
• docker exec rbpki-web-06 nginx -t reports the configuration test successful, and docker ps --filter name=rbpki-web-06 shows the container up rather than restarting.
• openssl s_client -connect 127.0.0.1:8443 -servername app.lab.example -CAfile root.crt lists two entries under Certificate chain and ends with Verify return code: 0 (ok).
• curl -sS --resolve app.lab.example:8443:127.0.0.1 --cacert root.crt https://app.lab.example:8443/ returns lab ok and exits 0.
• The same curl call without --cacert exits 60. A pass here is the failure appearing, not disappearing.
• The two digest lines in served-key-match.txt are identical.
• The deliverables fullchain.pem, default.conf, verify-report.txt and served-key-match.txt exist and are non-empty.

A failed validation looks specific, not vague. Verify return code: 20 with only one entry under Certificate chain means ssl_certificate is pointing at app.crt rather than fullchain.pem. Verify return code: 20 with two entries listed means the client does not trust your root, which is a -CAfile problem on the client side. An empty response with no chain at all usually means nginx is not listening, and docker logs rbpki-web-06 will say why.

Expected Outcome

$HOME/rbpki-lab-06/
├── html/
│   └── index.html
├── pki/
│   ├── app.crt
│   ├── app.csr
│   ├── app.ext
│   ├── app.key
│   ├── fullchain.pem
│   ├── root.crt
│   ├── root.key
│   ├── srv-ca.crt
│   ├── srv-ca.key
│   └── srv-ca.ext
├── site/
│   └── default.conf
├── s_client.out
├── served-key-match.txt
├── state.pre-lab
└── verify-report.txt

You can now answer a question you could not answer before: given a hostname and a port, is the TLS deployment behind it correct, and if not, which of the three independent things - the chain, the name, the key - is wrong. You have the commands that answer each part separately, and a transcript format that a reviewer can read without access to the server.

Troubleshooting

docker: Error response from daemon: driver failed programming external connectivity ... address already in use. Something already holds host port 8443. Find it with ss -ltnp | grep 8443, then either stop it or change both the -p mapping and every 8443 in the client commands to a free port.

nginx exits immediately and docker logs rbpki-web-06 mentions the certificate or key file. The bind mount path and the path in default.conf disagree, or app.key is not readable. The container reads the certificate as root at startup, so permissions are rarely the cause; a typo in /etc/nginx/certs/ almost always is.

openssl s_client prints Verify return code: 21 (unable to verify the first certificate). The server sent a chain that does not reach your anchor. Confirm what it sent with -showcerts and count the certificates before assuming the trust store is at fault.

curl returns curl: (7) Failed to connect. This is not a TLS problem. The container is not running or the port is not published. docker ps --filter name=rbpki-web-06 settles it in one command.

curl returns the page but openssl s_client hangs. s_client keeps the connection open waiting for input. Redirect from /dev/null, as every command in this lab does, or press Ctrl-D.

Cleanup

LAB="$HOME/rbpki-lab-06"

# 1. Stop and forget the service.
docker rm -f rbpki-web-06 2>/dev/null || true
docker network rm rbpki-net-06 2>/dev/null || true

# 2. Compare against the inventory recorded in Task 1.
cat "$LAB/state.pre-lab"
echo "--- containers now"
docker ps -a --format '{{.Names}}'
echo "--- networks now"
docker network ls --format '{{.Name}}'

# 3. Remove the lab directory, keys included.
rm -rf "$LAB"

# 4. Assert the host is as you found it.
docker ps -a --format '{{.Names}}' | grep -c '^rbpki-web-06$' || echo "container gone"
docker network ls --format '{{.Name}}' | grep -c '^rbpki-net-06$' || echo "network gone"
test -d "$LAB" && echo "LAB DIRECTORY STILL PRESENT" || echo "lab directory gone"

The two inventories printed in step 2 must differ only by the removal of rbpki-web-06 and rbpki-net-06. If any other name has appeared or disappeared, something outside the lab changed during the session and should be investigated before you close it out. The three assertions in step 4 are the restoration proof: all three must report the resource gone.

Production notes

• Real deployments almost never concatenate the chain by hand. ACME clients write fullchain.pem for you and certificate management platforms assemble it on delivery, but every one of them can be pointed at the wrong file by a configuration template, so the wire-level check in Task 5 stays relevant no matter how the file arrived.
• Publishing the private root into a container as a bind mount is a lab convenience. In production the root CA private key is offline and never present on a host that terminates TLS; only the issuing CA signs, and only the leaf key lives beside the service.
• nginx -t belongs in a change procedure as a pre-flight, never as the verification step. Pair it with an openssl s_client check against the reloaded service, because the second one is the only one that reads what is actually being served.
• The --resolve technique in Task 6 is the right way to test a certificate before DNS points at the new host. It exercises the real name in SNI and in hostname matching while sending the packets wherever you choose.

What You Learned

• A TLS server must present its intermediates; a leaf on its own is an orphan. The file-level proof is the difference between openssl verify with and without -untrusted, and the same failure on the wire is error 20.
• The trust anchor is not part of the chain the server sends. The client either has the root or it does not, and shipping a copy of it changes no client’s mind.
• Order in the bundle is load-bearing and unchecked. nginx starts happily with the chain reversed; the first certificate in the file is what the server claims to be.
• Verification is a property of a client, not of a certificate. The same server produced Verify return code: 0 (ok) and a curl exit of 60 in this lab, and the only thing that changed was which anchors the client held.
• Key possession is provable without a restart. Comparing the digest of the served public key with the digest derived from the private key catches a mismatched pair while it is still a file problem rather than an outage.
• nginx -t validates syntax, not identity. It cannot tell you the certificate expired, names the wrong host, or is missing its issuer.