Lab 4: Build a disposable two-tier certificate authority

Objective

Almost every organisation that runs an internal certificate authority runs a bad one, and it is nearly always the same bad one: a single self-signed certificate, generated once, with a ten-year life, no constraints, and its private key on the machine that issues from it. It works, right up until the key is exposed or the certificate expires, at which point every trust store in the estate has to be edited at the same time.

The alternative is two tiers. A root that signs exactly one thing, an intermediate that does the daily work, and constraints on each that limit the damage the other can do. By the end of this lab you will have built that, and, more importantly, you will be able to say what each extension on each certificate is for, because you will have deleted one and watched validation refuse the result.

The two failure experiments at the end are the part that sticks. Constraints you have only read about feel like paperwork; constraints you have watched stop a certificate you built yourself feel like a mechanism.

Architecture

Two authority certificates and nothing else. The root signs the intermediate. The intermediate is the one that will sign server certificates in Lab 5. The root’s key is used exactly once in this lab and should be used approximately never afterwards.

flowchart TD
    RK["root.key\nRSA 4096, mode 600"] --> RC["root.crt\nself-signed, 10 years\nCA:TRUE pathlen:1"]
    RC -- "signs" --> SC["srv-ca.crt\n5 years\nCA:TRUE pathlen:0"]
    IK["srv-ca.key\nRSA 3072"] --> CSR["srv-ca.csr"]
    CSR --> SC
    SC -- "will sign in Lab 5" --> LEAF["leaf certificates\nCA:FALSE"]
    SC --> CH["ca-chain.pem\nintermediate then root"]
    RC --> CH

The arrow from root to intermediate is the only time the root key is used. Everything after that point runs on the intermediate key, which is the property that makes the root protectable: a key that is used once a year can live somewhere a key used daily cannot.

Requirements

• OpenSSL 3.5.x. Every extension name and every option below is 3.x syntax.
• Roughly 10 MB of disk and a couple of minutes of CPU. A 4096-bit RSA key takes the longest single step in the lab.
• A shell session you can keep, so that LAB set in Task 1 survives to Cleanup.
• No network, no containers, no ports, no privileged operations.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. It installs nothing into any trust store; Lab 11 covers that separately and reversibly.

Scenario

You are standing up an internal PKI for a platform that currently uses a mixture of self-signed certificates and one expired wildcard nobody can find the key for. The design review asked two questions you could not answer: what stops this authority issuing a certificate for a name outside our estate, and what happens the day the issuing key is compromised.

Neither question is answerable with a single self-signed certificate, because a single certificate has nothing below it to constrain and nothing above it to replace it with. Both questions have straightforward answers once there are two tiers, and the answers are written into the extensions rather than into a policy document.

Tasks

Task 1 — Create the workspace and record the starting state

LAB="$HOME/rbpki-lab-04"

rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

# Two private keys are about to be written here. Nothing this shell creates
# should be group or world readable.
umask 077

# Record what Cleanup must restore.
{
  echo "umask now:  $(umask)"
  echo "openssl:    $(openssl version)"
  echo "started at: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
} > "$LAB/state.pre-lab"

ls -A "$HOME" | sort > "$LAB/state.pre-lab-home"
cat "$LAB/state.pre-lab"

Setting umask 077 here is belt and braces rather than a substitute for anything. OpenSSL writes key files restrictively when it creates them with -out, but this lab also builds chain files and extension files with shell redirection, and a tight umask means a mistake in one of those cannot produce a readable key.

Task 2 — Create the root key and the self-signed root certificate

The root is created in one step, because a self-signed certificate needs no request: the key that signs it is the key it certifies.

$ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root.key

cd "$HOME/rbpki-lab-04"

chmod 600 root.key
stat -c '%a %n' root.key

openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -addext "subjectKeyIdentifier=hash" \
  -out root.crt

Take the three -addext values in turn, because each one is a decision.

• basicConstraints=critical,CA:TRUE,pathlen:1. CA:TRUE is what makes this a certificate authority certificate rather than an end-entity certificate; without it no conforming validator will accept a signature made by this key on another certificate. pathlen:1 says at most one non-self-issued CA certificate may appear between this one and an end entity, which is precisely “one intermediate, then leaves”. critical means a validator that does not understand the extension must reject the certificate rather than proceed without the constraint, which is the only sane behaviour for a restriction.
• keyUsage=critical,keyCertSign,cRLSign. These are the only two operations this key is authorised for: signing certificates and signing certificate revocation lists. A root that also asserts digitalSignature is a root someone can use to terminate TLS. Marking it critical is what turns the restriction into an enforced one, and RFC 5280 requires that pathLenConstraint appear only where cA is TRUE and keyCertSign is asserted, so the two extensions have to agree.
• subjectKeyIdentifier=hash. A short identifier derived from the public key. It is not a security control; it is what lets a validator find the issuing certificate quickly in a store holding thousands, by matching it against the authority key identifier of the certificate below.

Task 3 — Read the root back before you build anything on it

An authority certificate with a wrong extension is worth catching now, while the only thing that depends on it is a directory you are about to delete.

cd "$HOME/rbpki-lab-04"

openssl x509 -in root.crt -noout -subject -issuer -dates
openssl x509 -in root.crt -noout -ext basicConstraints,keyUsage,subjectKeyIdentifier

Four things must be true in that output. Subject and issuer are the same string, which is what self-signed means. Basic Constraints reads CA:TRUE, pathlen:1 and is marked critical. Key Usage lists Certificate Sign and CRL Sign, is marked critical, and lists nothing else. Subject Key Identifier is present and holds a colon-separated hex value. If any of those is missing, the -addext argument was mis-quoted and the certificate has to be rebuilt now rather than after it has signed something.

Task 4 — Create the intermediate key and its request

The intermediate is a normal issuance: a key, a request, and a signature from a different key. The only unusual thing about it is what the extension file will say.

cd "$HOME/rbpki-lab-04"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out srv-ca.key
chmod 600 srv-ca.key

openssl req -new -key srv-ca.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA" \
  -out srv-ca.csr

openssl req -in srv-ca.csr -noout -subject -verify

The intermediate key is 3072 bits rather than 4096. That is a deliberate asymmetry: the root has to outlive several intermediates, so it gets the longer key and the longer validity, while the intermediate is meant to be replaced. There is nothing sacred about these two numbers; what matters is that the root is at least as strong as anything below it and lives at least as long.

Task 5 — Write the intermediate policy and sign it

basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always

Three of those four lines are the same decisions as on the root, made again for a different certificate. Two differences carry the design.

• pathlen:0 means no further CA certificate may appear below this one. The intermediate may sign end-entity certificates and nothing else. A compromise of srv-ca.key is therefore bounded: the attacker can mint leaf certificates, which is bad, but cannot mint another authority to hide behind, and cannot extend the tree.
• authorityKeyIdentifier=keyid:always copies the root’s subject key identifier into this certificate. always makes OpenSSL fail rather than silently omit the extension if the issuer has no subject key identifier to copy, which is why Task 2 set one on the root.

cd "$HOME/rbpki-lab-04"

cat > srv-ca.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl x509 -req -in srv-ca.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 1825 -extfile srv-ca.ext -out srv-ca.crt

$ openssl x509 -req -in srv-ca.csr -CA root.crt -CAkey root.key -CAcreateserial -sha256 -days 1825 -extfile srv-ca.ext -out srv-ca.crt

Certificate request self-signature ok
subject=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

-CAcreateserial creates root.srl, a file holding the next serial number this issuer will use. Serial numbers must be unique per issuer, and that file is the whole of OpenSSL’s memory of which ones it has handed out. Delete it and the next issuance starts again from a random value, which is survivable; copy the CA to a second host without it and the two hosts will issue colliding serials, which is not.

Task 6 — Verify the intermediate and match the key identifiers

cd "$HOME/rbpki-lab-04"

# Does the root actually vouch for this certificate?
openssl verify -CAfile root.crt srv-ca.crt

# The link the validator follows, read from both ends.
echo "root subject key identifier:"
openssl x509 -in root.crt -noout -ext subjectKeyIdentifier
echo "intermediate authority key identifier:"
openssl x509 -in srv-ca.crt -noout -ext authorityKeyIdentifier

openssl verify prints the filename followed by OK and exits 0. Then compare the two hex strings: the intermediate’s authority key identifier is the root’s subject key identifier, byte for byte. That correspondence is how a validator picks the right issuer out of a store without trying every certificate in it, and a mismatch here is the signature of a chain that was assembled from the wrong files.

$ openssl x509 -in srv-ca.crt -noout -ext basicConstraints,keyUsage,subjectKeyIdentifier,authorityKeyIdentifier

Task 7 — Assemble the chain file

A chain file is a plain concatenation of PEM certificates, ordered from the certificate nearest the leaf towards the anchor. Nothing parses the order for you, and getting it wrong is one of the commoner causes of a client that half works.

cd "$HOME/rbpki-lab-04"

cat srv-ca.crt root.crt > ca-chain.pem
chmod 644 ca-chain.pem

$ openssl storeutl -noout -certs ca-chain.pem

0: Certificate
1: Certificate
Total found: 2

storeutl is the quickest way to answer “how many certificates are actually in this file”, which matters because a truncated cat, a missing trailing newline, or an editor that stripped the last line all produce a file that still looks like PEM. Read each one in turn with a loop over openssl x509 if you need the subjects as well.

Task 8 — Prove that pathlen is enforced rather than decorative

Build a third tier and watch validation reject it. The intermediate will sign the third CA quite happily, because signing is a local operation and OpenSSL does not stop you issuing a certificate that will not validate.

cd "$HOME/rbpki-lab-04"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out third.key
openssl req -new -key third.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Third Tier CA" -out third.csr

cat > third.ext <<'EOF'
basicConstraints=critical,CA:TRUE
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl x509 -req -in third.csr -CA srv-ca.crt -CAkey srv-ca.key -CAcreateserial \
  -sha256 -days 365 -extfile third.ext -out third.crt

# Now issue a leaf from the third tier, so that the third tier is an
# intermediate in the path rather than the target of the verification.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out rogue.key
openssl req -new -key rogue.key -sha256 -subj "/CN=rogue.lab.example" -out rogue.csr

cat > rogue.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:rogue.lab.example
EOF

openssl x509 -req -in rogue.csr -CA third.crt -CAkey third.key -CAcreateserial \
  -sha256 -days 30 -extfile rogue.ext -out rogue.crt

cat srv-ca.crt third.crt > untrusted-chain.pem

$ openssl verify -CAfile root.crt -untrusted untrusted-chain.pem rogue.crt

Every signature in that path is cryptographically valid. Every certificate is inside its validity window. The chain is refused anyway, because pathlen:0 on the intermediate means no CA certificate may follow it, and pathlen:1 on the root means at most one may follow the root. The certificate that “works” locally and fails everywhere else is almost always this: a signature nobody disputes, under a constraint somebody forgot.

Task 9 — Prove that keyUsage is enforced too

Repeat the experiment with a different defect. Build an intermediate that is a CA by basic constraints but whose key usage does not permit certificate signing.

cd "$HOME/rbpki-lab-04"

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out broken-ca.key
openssl req -new -key broken-ca.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Broken Issuing CA" -out broken-ca.csr

cat > broken-ca.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,digitalSignature
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl x509 -req -in broken-ca.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 365 -extfile broken-ca.ext -out broken-ca.crt

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out broken-leaf.key
openssl req -new -key broken-leaf.key -sha256 \
  -subj "/CN=broken.lab.example" -out broken-leaf.csr

cat > broken-leaf.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:broken.lab.example
EOF

openssl x509 -req -in broken-leaf.csr -CA broken-ca.crt -CAkey broken-ca.key \
  -CAcreateserial -sha256 -days 30 -extfile broken-leaf.ext -out broken-leaf.crt

$ openssl verify -CAfile root.crt -untrusted broken-ca.crt broken-leaf.crt

Notice what this proves about your own root. The constraints you set in Task 2 are not documentation; a validator reads them on every connection and refuses paths that violate them. That is why an internal CA built without them is not merely untidy. It is a CA whose behaviour is bounded by nothing except the intentions of whoever holds the key.

Task 10 — Capture the deliverables

cd "$HOME/rbpki-lab-04"

{
  echo "# authority extension audit, $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo
  echo "== root.crt =="
  openssl x509 -in root.crt -noout -subject -issuer -dates
  openssl x509 -in root.crt -noout -ext basicConstraints,keyUsage,subjectKeyIdentifier
  echo "reason: CA:TRUE makes it an authority; pathlen:1 permits exactly one"
  echo "        intermediate below it; keyCertSign and cRLSign are the only"
  echo "        operations the key is authorised for."
  echo
  echo "== srv-ca.crt =="
  openssl x509 -in srv-ca.crt -noout -subject -issuer -dates
  openssl x509 -in srv-ca.crt -noout -ext basicConstraints,keyUsage,authorityKeyIdentifier
  echo "reason: pathlen:0 bounds a compromise of this key to leaf certificates;"
  echo "        the authority key identifier points back at the root's subject"
  echo "        key identifier so validators can find the issuer."
} > ca-extensions.txt

chmod 644 root.crt srv-ca.crt ca-chain.pem ca-extensions.txt
ls -l root.crt srv-ca.crt ca-chain.pem ca-extensions.txt

Validation

• openssl verify -CAfile root.crt srv-ca.crt prints srv-ca.crt: OK and exits 0.
• openssl x509 -in root.crt -noout -ext basicConstraints reports CA:TRUE, pathlen:1 and the word critical. openssl x509 -in srv-ca.crt -noout -ext basicConstraints reports CA:TRUE, pathlen:0 and critical.
• The hex value from openssl x509 -in root.crt -noout -ext subjectKeyIdentifier appears inside openssl x509 -in srv-ca.crt -noout -ext authorityKeyIdentifier.
• openssl storeutl -noout -certs ca-chain.pem reports two certificates. One means the concatenation in Task 7 lost a file.
• openssl verify -CAfile root.crt -untrusted untrusted-chain.pem rogue.crt exits 2 and refuses the path. An exit of 0 here means pathlen did not reach the certificates, which would mean Task 5 or Task 8 dropped its -extfile.
• openssl verify -CAfile root.crt -untrusted broken-ca.crt broken-leaf.crt also exits 2. Check with echo $? immediately after the command.
• stat -c '%a' root.key srv-ca.key reports 600 for both.
• The deliverables root.crt, srv-ca.crt, ca-chain.pem and ca-extensions.txt exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-04/
├── root.key  root.crt  root.srl        # the anchor and its serial counter
├── srv-ca.key  srv-ca.csr  srv-ca.crt  # the issuing intermediate
├── srv-ca.ext                          # the policy the root applied
├── srv-ca.srl                          # the intermediate's serial counter
├── ca-chain.pem                        # deliverable, intermediate then root
├── ca-extensions.txt                   # deliverable, decoded with reasoning
├── third.key third.csr third.crt       # Task 8: the tier that must not exist
├── third.ext  rogue.*                  # Task 8: and the leaf beneath it
├── untrusted-chain.pem                 # Task 8: the four-certificate path
├── broken-ca.*  broken-leaf.*          # Task 9: the CA that may not sign
├── state.pre-lab                       # Cleanup baseline
└── state.pre-lab-home                  # Cleanup baseline

You now hold a two-tier authority whose constraints you can defend line by line, and you have watched a validator refuse two certificates you signed yourself. Lab 5 issues the first real server certificate from srv-ca.crt and runs the full verification matrix against it.

Troubleshooting

openssl req -x509 rejects the -addext value. The whole name=value pair must arrive as one argument, so it has to be quoted. An unquoted basicConstraints=critical,CA:TRUE,pathlen:1 is one word to the shell and usually works by accident, but an unquoted key usage list containing a space does not.

authorityKeyIdentifier=keyid:always fails when signing the intermediate. The issuer has no subject key identifier for OpenSSL to copy. Task 2 sets one on the root; if the root was built without it, rebuild the root and re-sign the intermediate. This is exactly what always is for: failing rather than silently omitting the link.

openssl verify reports an unknown issuer for a certificate you just signed. Check that -CAfile names the root and not the intermediate. Verifying a leaf against the intermediate alone produces an error about being unable to get the issuer certificate, because the intermediate itself is then unverified.

Task 8 verification succeeds instead of failing. The most likely cause is verifying third.crt rather than rogue.crt. A path length constraint bounds the number of intermediate CA certificates in a path, so a CA certificate presented as the target of the verification is treated as the end entity and does not trip it. The leaf beneath it does.

Both keys are readable by your group. The umask 077 from Task 1 applies to the shell that ran it. If you opened a second terminal partway through, the files it created used that terminal’s umask. Run chmod 600 on both keys and set the umask before continuing.

Cleanup

LAB="$HOME/rbpki-lab-04"

# 1. Nothing was started, so there is nothing to stop. Confirm the certificates
#    are not referenced anywhere outside the lab directory.
grep -rl 'RunBook Lab Root CA' /etc/ssl/certs 2>/dev/null || \
  echo "root not present in the system trust store, as expected"

# 2. Compare the home directory against the Task 1 capture and keep a copy of
#    the baseline that will survive the deletion.
ls -A "$HOME" | sort > "$LAB/state.post-lab-home"
diff "$LAB/state.pre-lab-home" "$LAB/state.post-lab-home"
cp "$LAB/state.pre-lab-home" "$HOME/rbpki-lab-04.baseline"

# 3. Remove the directory, both authority keys included.
rm -rf "$LAB"

# 4. Confirm the only remaining difference is the baseline copy, then remove it.
ls -A "$HOME" | sort | diff "$HOME/rbpki-lab-04.baseline" - || true
rm -f "$HOME/rbpki-lab-04.baseline"

find "$HOME" -maxdepth 1 -name 'rbpki-lab-04' -print

Step 1 must report that the root is absent from the system trust store, step 2 must show no difference, and the find must produce no output. Together those three are the restoration assertion: no authority material remains and no trust decision was made on this host. If you also ran Lab 11, run its Cleanup as well, because that lab does modify the trust store.

Production notes

• Keep the root key offline. In practice that means generated on a machine with no network, stored on removable media or in a hardware security module, and brought out only to sign a new intermediate. A root used weekly is a root with the exposure of an issuing CA and the blast radius of a root.
• Give the intermediate a validity that fits your replacement plan, and start replacing it at half its life. An intermediate that expires with no successor already trusted is the outage that takes an entire estate down at once, and it is always known about years in advance.
• The 200-day cap that public authorities are now held to applies to TLS subscriber certificates, not to CA certificates, and does not apply to a private PKI at all. Choose your own intermediate and root lifetimes deliberately rather than copying a number from the public web.
• Record the fingerprint of root.crt somewhere outside this system the moment it is created. Distribution of a trust anchor is only as good as the recipient’s ability to check they received the right one.

What You Learned

• Two tiers exist to bound a compromise and to make replacement possible. The root signs once and can be kept somewhere the issuing key cannot; the intermediate does the work and can be revoked and replaced without touching a single trust store.
• basicConstraints decides what a certificate is. CA:TRUE makes it an authority, critical makes the restriction binding, and pathlen bounds how deep the tree below it may go.
• keyUsage decides what its key may do. A CA asserts keyCertSign and cRLSign and nothing else, and RFC 5280 ties pathLenConstraint to keyCertSign being present.
• The key identifier extensions are the index, not the security. They let a validator find the issuer quickly; the signature is what proves anything.
• Constraints are enforced by validators, not by issuers. OpenSSL will sign a certificate that can never validate, which is why both failure experiments had to build the certificate first and only then discover it was refused.