Lab 1: Inspect an X.509 certificate from a file and from a live endpoint

Objective

By the end of this lab you will be able to answer one question that comes up in almost every TLS incident, and answer it with a command rather than a belief: which certificate is this service actually presenting right now?

Reading a certificate is two separate skills. The first is decoding a file: turning a base64 blob into a subject, an issuer, a serial number, a validity window and a set of extensions, and knowing which of those fields a client actually uses. The second is capture: getting the certificate out of a running listener rather than out of the directory somebody told you it was deployed from. Most wasted incident time comes from doing the first skill against the wrong file.

You will issue a small certificate, take it apart field by field, put it behind an nginx container, pull it back off the wire, and then quietly change what the service serves so that the file and the listener disagree. The audit you build in the last task is the one you will reach for on a real host.

Architecture

Everything runs on one Linux host. A single-tier issuing CA signs one leaf certificate for app.lab.example. That leaf is mounted into an nginx:1.29-alpine container which listens on 127.0.0.1:18443. You read the certificate twice: once from the file with openssl x509, and once from the socket with openssl s_client.

flowchart LR
    K["EC P-256 key\napp.key"] --> R["CSR\napp.csr"]
    CA["issuing CA\nca.key + ca.crt"] --> L["leaf certificate\napp.crt"]
    R --> L
    L --> N["nginx container\n127.0.0.1:18443"]
    L --> F["file read\nopenssl x509"]
    N --> W["wire read\nopenssl s_client"]

The two reading paths matter more than the issuing path. openssl x509 answers “what does this file say”. openssl s_client answers “what did this socket send me”. A healthy service is one where those two answers are the same object, and the whole of the last task is about noticing when they are not.

Requirements

• OpenSSL 3.5.x on the host. The commands below use the 3.x spellings; a 3.0 or later build behaves the same for everything in this lab.
• Docker, able to pull nginx:1.29-alpine and publish a port on the loopback address. Roughly 100 MB of image download on first run.
• curl on the host, for the optional trust check in Task 6.
• TCP port 18443 free on 127.0.0.1. Nothing binds a routable interface.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. Every listener is bound to loopback and every file lives under one directory in your home.

Scenario

A colleague reports that an internal service is serving “the wrong certificate”, and the ticket contains a screenshot of a browser warning and nothing else. The deployment pipeline says the correct certificate was rolled out. The file on the host has the right name and a sensible modification time. Both of those observations are consistent with the service serving something else entirely, because a process that has already loaded a certificate into memory does not notice that the file underneath it changed, and a process pointed at the wrong path never read your file at all.

You are going to build that exact situation deliberately, in a directory nothing depends on, so that the next time you see it you recognise it in under a minute.

Tasks

Task 1 — Create the workspace and record the starting state

LAB="$HOME/rbpki-lab-01"

# Remove any container or network a previous attempt left behind.
docker rm -f rbpki-lab01-nginx >/dev/null 2>&1 || true
docker network rm rbpki-lab01-net >/dev/null 2>&1 || true

rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

# Record what Cleanup must restore: the containers and networks that existed
# before this lab ran. Cleanup compares against these two files.
docker ps -a --format '{{.Names}}' | sort > "$LAB/state.pre-lab-containers"
docker network ls --format '{{.Name}}' | sort > "$LAB/state.pre-lab-networks"

wc -l "$LAB/state.pre-lab-containers" "$LAB/state.pre-lab-networks"

Neither file should mention rbpki-lab01-nginx or rbpki-lab01-net. If either does, a previous run of this lab did not clean up and the docker rm above has just fixed it. Re-run this task so the recorded baseline is accurate.

Task 2 — Build the certificate you are going to read

This lab is about reading, not about issuing, so the issuing steps arrive here without commentary. Lab 4 builds a proper two-tier authority and explains every extension as it sets it. What matters now is that the leaf has a real issuer other than itself, so that the subject and issuer fields say different things.

cd "$HOME/rbpki-lab-01"

# A single-tier issuing CA. Its private key never leaves this directory.
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out ca.key
openssl req -x509 -new -key ca.key -sha256 -days 1825 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -addext "subjectKeyIdentifier=hash" \
  -out ca.crt

# The subject key for the service, and a request that names it.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
openssl req -new -key app.key -sha256 -subj "/CN=app.lab.example" \
  -addext "subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example" \
  -out app.csr

The extension file below is what the CA grants. It is deliberately written out as a file rather than passed inline, because the set of extensions a CA is willing to issue is a policy document, and policy documents belong in version control.

basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

cd "$HOME/rbpki-lab-01"

cat > app.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
EOF

openssl x509 -req -in app.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -sha256 -days 90 -extfile app.ext -out app.crt

OpenSSL prints two lines here. The first confirms it checked the signature on the request against the public key inside the request, and the second echoes the subject it is about to certify. Read them: a silent success and a success you confirmed are not the same thing.

Task 3 — Read the identity fields

Four fields answer “who is this, who vouched for it, and when does it stop counting”. Ask for them together, because asking for them one at a time is how you end up reading the serial of one certificate and the dates of another.

$ openssl x509 -in app.crt -noout -subject -issuer -serial -dates

subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

Take each line in turn. subject is the distinguished name the CA assigned; the Common Name inside it is a human label and nothing more, because RFC 9525 removed the Common Name fallback for service identity, so a conforming client must ignore it when deciding whether the certificate matches the name it dialled. issuer is the subject of whichever certificate signed this one, which is how a chain is walked. serial is unique only within one issuer, so a serial is meaningless unless you quote the issuer with it. notBefore and notAfter are absolute UTC instants, not durations, and a certificate that is not yet valid fails exactly as hard as one that has expired.

Task 4 — Decode the extensions

The identity fields say who. The extensions say what the certificate is allowed to be used for, and they are where certificates go wrong in production.

$ openssl x509 -in app.crt -noout -ext subjectAltName,keyUsage,extendedKeyUsage,basicConstraints

X509v3 Basic Constraints: critical
  CA:FALSE
X509v3 Key Usage: critical
  Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
  TLS Web Server Authentication
X509v3 Subject Alternative Name:
  DNS:app.lab.example, DNS:www.app.lab.example

• Basic Constraints, critical, CA:FALSE. This certificate may not sign other certificates. Marked critical, so a client that does not understand the extension must reject the certificate rather than ignore the restriction.
• Key Usage, critical, Digital Signature and Key Encipherment. These are the cryptographic operations the key is authorised for. A TLS 1.3 server proves possession of its key by signing the handshake transcript, so digitalSignature is the bit that matters on a modern connection.
• Extended Key Usage, TLS Web Server Authentication. The purpose. A certificate with only serverAuth cannot be presented as a client certificate, and Lab 5 proves that with a verification that fails on purpose.
• Subject Alternative Name. The identity a client actually matches against. Everything the service may legitimately be called has to appear here, or the connection fails on the name even though the chain is perfect.

Task 5 — Fingerprint the certificate and its public key

A fingerprint is a digest over the whole DER encoding of the certificate. Change one byte anywhere in it, including the serial or a single extension, and the fingerprint changes completely. That property is what makes it the right identifier for “is this the same object”, which is the question the rest of the lab is about.

$ openssl x509 -in app.crt -noout -fingerprint -sha256

sha256 Fingerprint=4C:85:DD:7B:E5:73:A6:A7:23:EE:82:D8:0F:05:98:DD:1D:AF:A6:FB:9F:93:C7:58:15:C7:9C:8F:02:D3:CC:AD

There is a second digest worth knowing, and it answers a different question. Hashing the certificate’s public key rather than the certificate gives you a value that survives reissuance: renew the same key into a new certificate and the certificate fingerprint changes while the public key digest does not.

cd "$HOME/rbpki-lab-01"

# Digest of the public key carried inside the certificate.
openssl x509 -in app.crt -noout -pubkey | openssl sha256

# Digest of the public half of the private key file.
openssl pkey -in app.key -pubout | openssl sha256

Those two digests must be identical, because the certificate certifies that key. Lab 2 turns this pair of commands into a deployment gate and shows what a mismatch looks like. For now, note only that you have two different identifiers with two different lifetimes, and that confusing them is how people conclude a rotation did not happen when it did.

Task 6 — Put it behind a listener and read it off the wire

Now stop reading the file and start reading the socket.

LAB="$HOME/rbpki-lab-01"
cd "$LAB"

cat > nginx.conf <<'EOF'
events {}
http {
  server {
    listen 8443 ssl;
    server_name app.lab.example;
    ssl_certificate     /certs/serve.crt;
    ssl_certificate_key /certs/serve.key;
    location / { return 200 "lab ok\n"; }
  }
}
EOF

# serve.crt and serve.key are what the service loads. Right now they are copies
# of the leaf you just issued.
cp app.crt serve.crt
cp app.key serve.key
chmod 644 serve.crt nginx.conf

docker network create rbpki-lab01-net
docker run -d --name rbpki-lab01-nginx --network rbpki-lab01-net \
  --network-alias app.lab.example \
  -p 127.0.0.1:18443:8443 \
  -v "$LAB:/certs:ro" \
  -v "$LAB/nginx.conf:/etc/nginx/nginx.conf:ro" \
  nginx:1.29-alpine

sleep 3
docker ps --filter name=rbpki-lab01-nginx --format '{{.Names}} {{.Status}}'

With the listener up, ask it what it is holding. The -showcerts flag prints every certificate the server sent, in the order it sent them.

$ openssl s_client -connect 127.0.0.1:18443 -servername app.lab.example -showcerts </dev/null

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

Read the s: and i: lines first. Position 0 is the leaf, s: is its subject and i: is its issuer, and here the chain stops at position 0 because the server sent one certificate and nothing else. The two verify errors are the client saying it cannot build a path to a trust anchor, which is correct: your CA is a file in your home directory and no trust store has ever heard of it. Lab 7 makes that failure the subject rather than the background noise.

What you want from this task is the certificate itself, as a file you own:

cd "$HOME/rbpki-lab-01"

openssl s_client -connect 127.0.0.1:18443 -servername app.lab.example \
  </dev/null 2>/dev/null | openssl x509 -out served-first.pem

openssl x509 -in served-first.pem -noout -subject -issuer -serial

# Optional: confirm the service is genuinely serving content over that socket.
curl -sS --cacert ca.crt --resolve app.lab.example:18443:127.0.0.1 \
  https://app.lab.example:18443/

openssl x509 reading from a pipe takes the first certificate of the PEM stream and re-emits it, which is exactly the leaf. You now hold two files that should describe the same object.

Task 7 — Make the file and the listener disagree

This is the production question. Issue a second certificate for the same name, put it where the service reads from, reload, and leave app.crt untouched so that the file an operator would inspect is no longer the file being served.

cd "$HOME/rbpki-lab-01"

# A second leaf: same name, different key, different serial.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app2.key
openssl req -new -key app2.key -sha256 -subj "/CN=app.lab.example" \
  -addext "subjectAltName=DNS:app.lab.example" -out app2.csr

cat > app2.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:app.lab.example
EOF

openssl x509 -req -in app2.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -sha256 -days 30 -extfile app2.ext -out app2.crt

# Deploy the second one, and only the second one.
cp app2.crt serve.crt
cp app2.key serve.key
chmod 644 serve.crt
docker exec rbpki-lab01-nginx nginx -s reload
sleep 2

Now run the audit. The two commands below are the whole answer to “which certificate is this service presenting”, and they are worth committing to memory.

cd "$HOME/rbpki-lab-01"

openssl s_client -connect 127.0.0.1:18443 -servername app.lab.example \
  </dev/null 2>/dev/null | openssl x509 -out served-now.pem

echo "on disk at app.crt:"
openssl x509 -in app.crt -noout -fingerprint -sha256 -serial

echo "served by the listener:"
openssl x509 -in served-now.pem -noout -fingerprint -sha256 -serial

The two fingerprints differ, and so do the two serials. Both certificates are valid, both name app.lab.example, both were issued by the same CA, and the file an operator would open is not the one on the wire. Nothing in the file’s name, size or modification time would have told you that.

Task 8 — Capture the deliverables

cd "$HOME/rbpki-lab-01"

{
  echo "# lab leaf app.crt, captured $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  openssl x509 -in app.crt -noout -subject -issuer -serial -dates
  echo
  openssl x509 -in app.crt -noout -ext subjectAltName,keyUsage,extendedKeyUsage,basicConstraints
} > cert-fields.txt

cp served-now.pem served-leaf.pem

{
  echo "# fingerprint audit, host $(hostname), $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "endpoint: 127.0.0.1:18443 (SNI app.lab.example)"
  echo -n "on-disk app.crt : "
  openssl x509 -in app.crt -noout -fingerprint -sha256
  echo -n "served leaf     : "
  openssl x509 -in served-leaf.pem -noout -fingerprint -sha256
  echo "verdict: the served certificate is NOT the file at app.crt"
} > fingerprint-audit.txt

ls -l cert-fields.txt served-leaf.pem fingerprint-audit.txt

Validation

• openssl x509 -in app.crt -noout -subject -issuer prints a subject of CN=app.lab.example and an issuer naming the lab CA. If subject and issuer are the same string, Task 2 produced a self-signed certificate and the -CA arguments were not applied.
• openssl x509 -in app.crt -noout -ext basicConstraints reports CA:FALSE and the word critical. A missing critical means the extension file was not read.
• openssl x509 -in app.crt -noout -checkend 0 exits 0 and prints that the certificate will not expire. A non-zero exit here means the host clock is wrong, not that the certificate is.
• openssl x509 -in served-leaf.pem -noout -serial and openssl x509 -in app.crt -noout -serial print different serials after Task 7. Identical serials mean the reload in Task 7 did not take effect.
• docker ps --filter name=rbpki-lab01-nginx lists exactly one running container.
• The deliverables cert-fields.txt, served-leaf.pem and fingerprint-audit.txt exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-01/
├── ca.crt                        # the lab issuing CA certificate
├── ca.key                        # its private key, mode 600
├── app.key  app.csr  app.crt     # the first leaf: key, request, certificate
├── app2.key app2.csr app2.crt    # the second leaf, deployed in Task 7
├── app.ext  app2.ext             # the extensions the CA granted
├── serve.crt serve.key           # what the container actually loads
├── nginx.conf                    # the server block
├── served-first.pem              # captured from the socket before the swap
├── served-now.pem                # the post-reload capture
├── served-leaf.pem               # deliverable copy of the served certificate
├── cert-fields.txt               # deliverable
├── fingerprint-audit.txt         # deliverable
├── state.pre-lab-containers      # Cleanup baseline
└── state.pre-lab-networks        # Cleanup baseline

You can now take any TLS endpoint, retrieve the certificate it is presenting, decode every field that governs whether a client will accept it, and state with a digest rather than an assumption whether that certificate is the file you meant to deploy.

Troubleshooting

openssl s_client hangs and never returns to the prompt. s_client keeps the connection open for interactive use. Every invocation in this lab redirects /dev/null into it so it closes the connection after the handshake. If you retyped a command without that redirection, press Ctrl-D.

docker run fails with a port already allocated. Something else holds 127.0.0.1:18443. Find it with ss -ltnp and either stop it or edit both the published port and every -connect 127.0.0.1:18443 in this lab to a free port.

nginx -s reload reports that it cannot open the certificate. The container mounts $LAB read-only at /certs, so a file created after the container started is visible, but a file whose permissions exclude the nginx worker is not. Re-run chmod 644 serve.crt on the host and reload again. The key file needs to stay restrictive; nginx reads it as root before dropping privileges.

The served fingerprint in Task 7 still matches app.crt. The reload did not happen, or it happened before the copy. Confirm the container is running with docker ps, re-run the two cp commands, then reload and re-capture. A reload that targets a stopped container fails loudly, so check the exit status.

openssl x509 -out served-first.pem produces an empty file. The pipeline produced no certificate, which means the handshake failed rather than the parsing. Drop the 2>/dev/null and read what s_client reported.

Cleanup

LAB="$HOME/rbpki-lab-01"

# 1. Stop and forget the container, then remove its network.
docker rm -f rbpki-lab01-nginx
docker network rm rbpki-lab01-net

# 2. Compare against the Task 1 capture: the lab's names must be gone and
#    nothing else may have changed.
docker ps -a --format '{{.Names}}' | sort > "$LAB/state.post-lab-containers"
docker network ls --format '{{.Name}}' | sort > "$LAB/state.post-lab-networks"
diff "$LAB/state.pre-lab-containers" "$LAB/state.post-lab-containers"
diff "$LAB/state.pre-lab-networks" "$LAB/state.post-lab-networks"

# 3. Remove the lab directory, keys included.
rm -rf "$LAB"

Both diff invocations must print nothing and exit 0. That is the restoration assertion: the Docker state after the lab is byte-for-byte the list you recorded before it. Confirm the directory is gone with find "$HOME" -maxdepth 1 -name 'rbpki-lab-01' -print, which should produce no output.

Production notes

• On a real host the certificate you want is rarely reachable on loopback. Add -servername with the name the client would send, because a server with several virtual hosts selects its certificate from SNI, and connecting by IP address without -servername gets you the default virtual host rather than the one you are debugging.
• Run the capture from where the failing client runs. A certificate served correctly to your workstation and incorrectly to a pod in another network segment is a routing or proxy problem wearing a TLS costume, and the only way to see it is to capture from both places and compare the fingerprints.
• Store the public key digest, not the certificate fingerprint, when you want a stable identifier across renewals. Store the certificate fingerprint when you are proving that one specific issuance is deployed.
• Never keep a private key in the same backup, repository or ticket attachment as the certificate. The certificate is public by design and the key never is.

What You Learned

• A certificate has two readings, and they can disagree. The file says one thing, the socket says another, and only the socket describes what clients experience.
• Subject and issuer are distinguished names, not hostnames. The identity a client matches is the Subject Alternative Name, and RFC 9525 removed the Common Name fallback outright rather than merely deprecating it.
• Extensions are the authorisation layer. basicConstraints decides whether the certificate may sign, keyUsage decides which cryptographic operations are permitted, and extendedKeyUsage decides what role it may play in a protocol.
• A fingerprint identifies one issuance; a public key digest identifies one key. Choosing the wrong one turns every routine renewal into an alert, or hides a key change you needed to see.
• A filesystem check cannot see a stale process. The certificate on disk and the certificate in a running process are independent facts, and every certificate monitoring system that reads only files inherits that blind spot.