Lab 2: Generate keys safely and prove key to certificate correspondence

Objective

A private key has exactly one job: to be the only copy. Everything that goes wrong with key material goes wrong at one of two moments, and both of them are in this lab.

The first moment is generation and handling. A key created with the right command can still end up readable by every account on the host, not because anybody changed its permissions but because somebody copied it with a shell redirection instead of an option. You will produce that leak deliberately and learn to see it.

The second moment is deployment, where a certificate and a key arrive from different places and nobody checks that they belong together. You will build the check: two commands whose output must be byte-identical, wrapped into a script that a deployment pipeline can run before it restarts anything. Then you will break the pair on purpose so that you recognise the failure when a service refuses to start at three in the morning.

Architecture

There is no network topology here. The whole lab is one directory holding key files, one self-signed certificate that binds one of those keys, and a throwaway container used once at the end to show how a real service reacts to a mismatched pair.

flowchart TD
    G["openssl genpkey"] --> R["rsa-4096.key"]
    G --> E["app.key\nEC P-256"]
    G --> D["decoy.key\nEC P-256"]
    E --> C["app.crt\nself-signed over app.key"]
    E --> P1["pkey -pubout | sha256"]
    C --> P2["x509 -pubkey | sha256"]
    P1 --> V{"digests equal?"}
    P2 --> V
    V -- "yes" --> OK["safe to deploy"]
    V -- "no" --> NO["refuse to deploy"]

The certificate is self-signed on purpose. Correspondence between a key and a certificate is a property of the public key inside each of them, so it is exactly the same test whether the certificate came from a public authority, a private CA or a self-signature. Removing the issuer from the picture keeps the lab on the key.

Requirements

• OpenSSL 3.5.x on the host. genpkey is the current generator; the older genrsa and ecparam spellings still exist but do not take a uniform option set.
• Docker, able to run nginx:1.29-alpine. It is used once, for about ten seconds, in Task 7, and it is removed by the same command that starts it.
• A shell you can run several commands in, so that variables set in Task 1 survive.
• Roughly 5 MB of disk. A 4096-bit RSA key takes a few seconds of CPU to generate.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. It binds no ports and starts no long-lived process.

Scenario

A deployment fails at 02:40. The service will not start, the log says something about a private key, and the change that went out was described in the ticket as “certificate renewal, no config change”. Two teams are now awake: one convinced the certificate is bad, one convinced the key is bad.

Both are wrong in a useful way. Neither file is bad. They simply do not belong to each other, because the renewal produced a fresh key and the deployment shipped the new certificate alongside the old key. The check that would have caught it before anything restarted takes under a second and fits on two lines. You are going to write it.

Tasks

Task 1 — Create the workspace and record the starting state

LAB="$HOME/rbpki-lab-02"

docker rm -f rbpki-lab02-check >/dev/null 2>&1 || true

rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

# Record what Cleanup must restore. The umask matters to Task 3, and the
# container list proves at the end that the lab left nothing running.
{
  echo "umask: $(umask)"
  echo "user:  $(id -un)"
  echo "date:  $(date -u +%Y-%m-%dT%H:%M:%SZ)"
} > "$LAB/state.pre-lab"

docker ps -a --format '{{.Names}}' | sort > "$LAB/state.pre-lab-containers"
cat "$LAB/state.pre-lab"

Note the umask value. On most Linux distributions it is either 0022 or 0002. Task 3 turns that number into a permission bug, and knowing which one you started from is the difference between reading the result and guessing at it.

Task 2 — Generate one RSA key and two EC keys

openssl genpkey is the modern generator. It takes the algorithm as an argument and the algorithm’s parameters as -pkeyopt pairs, which means one command shape covers every algorithm rather than one command per algorithm.

cd "$HOME/rbpki-lab-02"

# RSA. The pkeyopt controls the modulus size in bits.
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out rsa-4096.key

# Elliptic curve. The pkeyopt names the curve, not a bit count.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key

# A second EC key with no relationship to anything. This is the decoy that
# Task 7 uses to break the pair.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out decoy.key

stat -c '%a %U %n' rsa-4096.key app.key decoy.key

Two things are worth noticing in that stat output. First, the mode: on the OpenSSL 3.5 build used to prepare this course, writing a private key with -out produced mode 600, and the key was never briefly world-readable. Do not take that on trust on your own build; the point of running stat is that you looked. Second, the sizes: the RSA key file is an order of magnitude larger than the EC key file, and the P-256 key is generally taken to offer security comparable to a 3072-bit RSA modulus. That ratio is why EC keys dominate new TLS deployments.

Task 3 — Watch a private key lose its permissions

This is the part nobody teaches, and it is the reason keys leak on hosts where every individual step looked correct. -out is an OpenSSL option and OpenSSL controls the mode it creates the file with. A shell redirection is not an OpenSSL option, and the shell creates the file with your umask.

cd "$HOME/rbpki-lab-02"

# Path A: OpenSSL creates the file.
openssl pkey -in app.key -out copy-via-out.key

# Path B: the shell creates the file, then OpenSSL writes into it.
openssl pkey -in app.key > copy-via-redirect.key

# Path C: the same mistake, wearing a different hat.
cat app.key > copy-via-cat.key

# Path D: cp, which reproduces the source file's mode for a new file.
cp app.key copy-via-cp.key

stat -c '%a %n' app.key copy-via-out.key copy-via-redirect.key \
  copy-via-cat.key copy-via-cp.key

Read the four modes against the umask you recorded in Task 1. Paths A and D come out restrictive. Paths B and C come out at whatever your umask permits, which is 644 on a host with umask 0022 and 664 on a host with umask 0002. In both cases the private key is now readable by an account that is not yours, and nothing warned you.

The fix is to create the file with the mode you want before anything writes to it:

cd "$HOME/rbpki-lab-02"

install -m 0600 /dev/null safe-copy.key
openssl pkey -in app.key > safe-copy.key

stat -c '%a %n' safe-copy.key

# Repair the two leaky copies so the rest of the lab is tidy.
chmod 600 copy-via-redirect.key copy-via-cat.key
stat -c '%a %n' copy-via-redirect.key copy-via-cat.key

install -m 0600 /dev/null target creates an empty file with exactly the mode you asked for, regardless of umask. It is the idiom to reach for in any script that writes key material through a redirection, and it belongs in the same script as the redirection rather than in a chmod on the following line, because the gap between the two lines is a window in which the key is readable.

Task 4 — Encrypt a key at rest, and understand the limit of what that buys

OpenSSL can wrap a private key in a passphrase-derived cipher. It is worth knowing how, and it is more important to know what it does not protect.

cd "$HOME/rbpki-lab-02"

# The passphrase below is deliberately self-labelling. Never use a real one on
# a command line, where it is visible in the process table and the shell history.
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
  -aes256 -pass pass:lab-only-not-real -out encrypted.key

head -1 encrypted.key

# Reading it requires the passphrase.
openssl pkey -in encrypted.key -passin pass:lab-only-not-real -pubout \
  | openssl sha256

# The wrong passphrase produces an error, not a wrong answer. Pass it on the
# command line here so the command cannot stop to prompt you.
openssl pkey -in encrypted.key -passin pass:wrong-passphrase -pubout

The first line of the file reads BEGIN ENCRYPTED PRIVATE KEY rather than BEGIN PRIVATE KEY, which is how you tell the two apart at a glance. The wrong passphrase fails rather than producing garbage, because the wrapper is authenticated and OpenSSL can tell that the decryption did not work. Omit -passin altogether and OpenSSL prompts on the terminal instead, which is convenient by hand and a hang in a script.

Now the limit. An encrypted key protects material at rest. A service that must start unattended has to be given the passphrase, which means the passphrase lives somewhere on the same host, in a systemd unit, an environment file or a prompt that somebody automated away. At that point you have moved the secret rather than removed it. Encryption at rest is worth having for a key that a human carries, such as a CA key kept offline. For a web server’s key on the server that uses it, the control that matters is file ownership and mode, plus not putting the key on that host at all if a hardware-backed store can hold it instead.

Task 5 — Bind one key into a certificate

You now need a certificate to test against. Sign one directly over app.key so that the only interesting relationship in the file is the one between the certificate and that key.

cd "$HOME/rbpki-lab-02"

openssl req -x509 -new -key app.key -sha256 -days 90 \
  -subj "/CN=app.lab.example" \
  -addext "basicConstraints=critical,CA:FALSE" \
  -addext "keyUsage=critical,digitalSignature,keyEncipherment" \
  -addext "extendedKeyUsage=serverAuth" \
  -addext "subjectAltName=DNS:app.lab.example" \
  -out app.crt

openssl x509 -in app.crt -noout -subject -issuer

Subject and issuer are the same string, which is the definition of self-signed. That tells you nothing about whether the key matches, and it is worth being explicit about why: a self-signature proves the signer held the private key at signing time, but you are reading a file somebody handed you, and the question in front of you is whether the key file next to it is the same key. Only a comparison of the two files answers that.

$ openssl x509 -in app.crt -noout -checkend 0

Certificate will not expire

-checkend 0 asks whether the certificate expires within zero seconds from now, so it answers “is it valid at this instant” and exits 0 when it is. Give it a number of seconds instead and it becomes the monitoring primitive the whole course returns to.

Task 6 — Prove correspondence with two digests

Here is the test. Extract the public key from the private key file, extract the public key from the certificate, and hash both. A certificate certifies a public key, so if the key file is the counterpart of that certificate the two digests are the same value.

$ openssl pkey -in app.key -pubout | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

$ openssl x509 -in app.crt -noout -pubkey | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Both commands emit a PEM SUBJECT PUBLIC KEY INFO block on standard output and the digest is taken over that text. Because both sides serialise the same structure the same way, an equal digest is a reliable statement that the two files describe one key pair. An unequal digest is an equally reliable statement that they do not.

Wrap it into something a pipeline can run:

cd "$HOME/rbpki-lab-02"

cat > pair-check.sh <<'EOF'
#!/usr/bin/env bash
# Exit 0 when the key in $1 corresponds to the certificate in $2.
set -euo pipefail

key="$1"
cert="$2"

key_digest=$(openssl pkey -in "$key" -pubout | openssl sha256)
cert_digest=$(openssl x509 -in "$cert" -noout -pubkey | openssl sha256)

if [ "$key_digest" = "$cert_digest" ]; then
  echo "MATCH   $key corresponds to $cert"
  exit 0
fi

echo "MISMATCH $key does not correspond to $cert" >&2
echo "  key  digest: $key_digest" >&2
echo "  cert digest: $cert_digest" >&2
exit 1
EOF

chmod 755 pair-check.sh
./pair-check.sh app.key app.crt
echo "exit status: $?"

set -euo pipefail matters more than it looks. Without pipefail, a failure in openssl pkey would be hidden by the successful openssl sha256 at the end of the pipe, and the script would compare a digest of nothing against a digest of nothing and report a match. A gate that fails open is worse than no gate.

Task 7 — Break the pair on purpose

Now run the same script against the decoy key, and then let a real service tell you the same thing in its own words.

cd "$HOME/rbpki-lab-02"

# The script exits non-zero, so do not let set -e in your own shell stop here.
./pair-check.sh decoy.key app.crt || echo "gate correctly refused the pair"

The two digest lines it prints differ from each other. That difference is the entire test: you do not need to know what either value means, only that they are not the same.

Give the mismatched pair to nginx and watch it refuse:

cd "$HOME/rbpki-lab-02"

cat > nginx.conf <<'EOF'
events {}
http {
  server {
    listen 8443 ssl;
    server_name app.lab.example;
    ssl_certificate     /certs/app.crt;
    ssl_certificate_key /certs/decoy.key;
    location / { return 200 "lab ok\n"; }
  }
}
EOF

chmod 644 app.crt nginx.conf decoy.key

docker run --rm --name rbpki-lab02-check \
  -v "$HOME/rbpki-lab-02:/certs:ro" \
  -v "$HOME/rbpki-lab-02/nginx.conf:/etc/nginx/nginx.conf:ro" \
  nginx:1.29-alpine nginx -t

# Restore the restrictive mode on the key immediately afterwards.
chmod 600 decoy.key

The configuration test fails and nginx exits non-zero. Read the message: it names the private key file it was given and reports that the key values do not match the certificate it was paired with. That is the same fact your script reported, arriving several minutes later in a deployment window, from a process that has now not started.

Task 8 — Capture the deliverables

cd "$HOME/rbpki-lab-02"

{
  echo "# key inventory, $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  for k in rsa-4096.key app.key decoy.key encrypted.key; do
    mode=$(stat -c '%a' "$k")
    kind=$(head -1 "$k")
    echo "file: $k  mode: $mode  header: $kind"
  done
  echo
  echo "public-key digest of app.key:"
  openssl pkey -in app.key -pubout | openssl sha256
  echo "public-key digest of decoy.key:"
  openssl pkey -in decoy.key -pubout | openssl sha256
} > key-inventory.txt

{
  echo "# mismatch evidence, $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "certificate under test: app.crt"
  ./pair-check.sh decoy.key app.crt 2>&1 || true
  echo "verdict: decoy.key must never be deployed alongside app.crt"
} > mismatch-evidence.txt

ls -l key-inventory.txt pair-check.sh mismatch-evidence.txt

Validation

• stat -c '%a' app.key reports 600. Anything wider means the key was regenerated by a command other than the one in Task 2, or copied over by a later step.
• stat -c '%a' copy-via-redirect.key reported a mode wider than 600 before you repaired it in Task 3. If it did not, your umask is already 0077, which is a good habit and hides the lesson.
• ./pair-check.sh app.key app.crt prints a line beginning MATCH and exits 0.
• ./pair-check.sh decoy.key app.crt prints two different digest values and exits 1. Identical digests here would mean decoy.key is not actually a second key.
• head -1 encrypted.key names an encrypted private key, and openssl pkey -in encrypted.key -pubout -passin pass:wrong-passphrase fails instead of printing a public key. Omitting -passin entirely makes OpenSSL prompt on the terminal and wait, which is correct behaviour and not a hang.
• docker ps -a --filter name=rbpki-lab02-check lists nothing, because the container ran with --rm.
• The deliverables key-inventory.txt, pair-check.sh and mismatch-evidence.txt exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-02/
├── rsa-4096.key             # RSA 4096, mode 600
├── app.key                  # EC P-256, the key under test
├── decoy.key                # EC P-256, unrelated to any certificate here
├── encrypted.key            # RSA 2048 wrapped in AES-256
├── app.crt                  # self-signed over app.key
├── copy-via-out.key         # created by OpenSSL, restrictive
├── copy-via-redirect.key    # created by the shell, leaked then repaired
├── copy-via-cat.key         # the same leak by another route
├── copy-via-cp.key          # cp reproduced the source mode
├── safe-copy.key            # install -m 0600 first, then redirect
├── nginx.conf               # the deliberately mismatched server block
├── pair-check.sh            # deliverable, executable
├── key-inventory.txt        # deliverable
├── mismatch-evidence.txt    # deliverable
├── state.pre-lab            # Cleanup baseline
└── state.pre-lab-containers # Cleanup baseline

You can now generate a key of either family with one command shape, state where in a deployment script a key would silently become readable, and answer the question “does this key belong to this certificate” in one second with a result you can put in a change record.

Troubleshooting

openssl genpkey with -pkeyopt ec_paramgen_curve:P-256 is rejected. The curve name is case-sensitive and the short OpenSSL alias is prime256v1. Both name the same curve. If neither is accepted, the build is older than the 3.x series and genpkey is taking a different option set; check openssl version first.

The RSA generation appears to hang. A 4096-bit modulus takes noticeably longer than a 2048-bit one, and on a virtual machine with a thin entropy pool it can take several seconds with no output at all. It is not stuck. If it truly never returns, generate a 2048-bit key instead and continue; nothing else in the lab depends on the size.

stat -c is rejected as an invalid option. You are on a BSD-derived stat, which spells the format option -f and uses different placeholders. ls -l shows the same information in a form you can read directly.

pair-check.sh reports a match for two keys you know are different. The most likely cause is a copy and paste that lost the set -euo pipefail line, letting a failed openssl invocation produce two empty digests that compare equal. Run the two digest commands by hand and confirm each prints a value.

The docker run in Task 7 fails to read the certificate. The bind mount is read-only, and the container’s nginx runs its configuration test as root, so the usual cause is the host path being wrong rather than the mode. Confirm ls -l "$HOME/rbpki-lab-02/app.crt" from the host before rerunning.

Cleanup

LAB="$HOME/rbpki-lab-02"

# 1. Keep the one artefact worth keeping.
cp "$LAB/pair-check.sh" "$HOME/pair-check.sh"

# 2. The container ran with --rm, so it should already be gone. Prove it, and
#    force the removal if a previous attempt left one behind.
docker rm -f rbpki-lab02-check >/dev/null 2>&1 || true
docker ps -a --format '{{.Names}}' | sort > "$LAB/state.post-lab-containers"
diff "$LAB/state.pre-lab-containers" "$LAB/state.post-lab-containers"

# 3. Confirm the umask is the one Task 1 recorded, then remove everything.
cat "$LAB/state.pre-lab"
umask
rm -rf "$LAB"

The diff must print nothing and exit 0, and the umask output must equal the value recorded in state.pre-lab. Confirm the directory is gone with find "$HOME" -maxdepth 1 -name 'rbpki-lab-02' -print, which should produce no output. The only thing this lab leaves behind is $HOME/pair-check.sh, which you copied out deliberately.

Production notes

• Generate the key where it will be used. Every copy of a private key is another place it can leak from, and a key that has travelled through a laptop, a ticket attachment or a chat message is a key that must be treated as rotated regardless of what the policy says.
• Run the correspondence check in the deployment pipeline, before the reload, not after the failure. It costs one second and it converts a class of outage into a failed deployment that never touched the running service.
• Prefer a fresh key at every renewal where the automation supports it. Reusing a key across renewals means a compromise of that key spans every certificate ever issued over it, and a certificate rotation does not end it.
• Keep the key out of the artefact that carries the certificate. Chain files, container images and configuration repositories are all places the certificate legitimately belongs and the key never does.

What You Learned

• openssl genpkey is one command shape for every algorithm. The algorithm is an argument and its parameters are -pkeyopt pairs, which is why the RSA and EC invocations differ only in those pairs.
• Permissions are lost at creation, not at rest. A shell redirection creates the file with your umask before OpenSSL writes a byte into it, and a later chmod cannot close the window it opened.
• Encryption at rest moves a secret rather than removing it. For an unattended service the passphrase has to live on the same host, which is why ownership and mode do more work than a passphrase does.
• Correspondence is a property of the public key, and it is one comparison. Two digests, taken the same way, over the same serialisation, that must be equal.
• A mismatched pair is invisible to any single-file check. Both files are valid; the defect is in the relationship, so only a comparison finds it before a service does.