Lab 26: Restore a PKI from backup

Objective

Most people who have backed up a certificate authority have backed up a key and a certificate. That is enough to prove you still control the authority, and it is not enough to keep operating it. The state that decides whether the restored CA behaves correctly is the issuance state: which serial number comes next, and which certificates have already been signed.

You will build a two-tier authority, take a backup, keep issuing after the backup as any real CA would, then delete the issuing CA completely. You will restore it, prove it is genuinely the same authority, and then watch it commit the most quietly damaging error a CA can make: signing a second certificate with a serial number it has already used. Nothing will warn you. Both certificates will validate.

The lab ends with the repair, and with the reason some organisations choose keys that cannot be backed up at all.

Architecture

An offline root signs one issuing CA. The issuing CA signs leaf certificates and maintains two pieces of state alongside its key: the serial counter that OpenSSL keeps in a .srl file, and a ledger of what it has signed.

flowchart TD
    A["root CA\nroot.key and root.crt"] -- "signs once" --> B["issuing CA\nsrv-ca.key and srv-ca.crt"]
    B -- "signs each leaf" --> C["issued archive\napp, api, web, shop"]
    B -- "reads, then advances" --> D["srv-ca.srl\nthe next serial number"]
    B -- "appends one line per signature" --> E["issued.ledger\nserial, subject, timestamp"]

The key and the certificate prove identity. The .srl file and the ledger carry behaviour. A restore that recovers the first pair and loses the second produces an authority that is unmistakably yours and quietly wrong, which is the worst of the available outcomes.

Requirements

• OpenSSL 3.5.x on the path. Earlier 3.x releases accept every command used here, but the lab is written against 3.5.
• GNU coreutils, for tar, sha256sum, sort, cut and uniq.
• A Linux host with a writable home directory. Everything the lab creates lives under a single directory prefixed rbpki-.
• No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. No trust store is modified, so nothing you build here is trusted by anything else on the host.

Scenario

Your organisation runs an internal issuing CA on a single virtual machine. It has signed roughly four hundred certificates for internal services over two years. There is a backup, taken monthly, and somebody once confirmed it contained the private key.

This morning the volume that holds the CA is gone. You have the backup from three weeks ago, and in those three weeks the CA carried on issuing. You are about to find out what that gap costs.

Tasks

Task 1 — Record the starting state

LAB="$HOME/rbpki-lab-26"
PRE=/tmp/rbpki-26-state.pre-lab

# Record what Cleanup must restore, before creating anything.
{
  openssl version
  printf 'lab directory existed before start: '
  if [ -e "$LAB" ]; then echo yes; else echo no; fi
} > "$PRE"

rm -rf "$LAB"
mkdir -p "$LAB/root" "$LAB/ca" "$LAB/issued" "$LAB/backup"
mv "$PRE" "$LAB/state.pre-lab"
cd "$LAB"
cat state.pre-lab

Task 2 — Build the two-tier authority

The root is generated in its own directory and is not part of the backup you will take later. That reflects how a root should be held: offline, separate, and not restored as a side effect of restoring the issuing CA.

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 \
  -out root/root.key
chmod 600 root/root.key
openssl req -x509 -new -key root/root.key -sha256 -days 3650 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out root/root.crt

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 \
  -out ca/srv-ca.key
chmod 600 ca/srv-ca.key
openssl req -new -key ca/srv-ca.key -sha256 \
  -subj "/O=RunBook Academy Lab/CN=RunBook Lab Server Issuing CA" \
  -out ca/srv-ca.csr

cat > ca/ca.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

openssl x509 -req -in ca/srv-ca.csr -CA root/root.crt \
  -CAkey root/root.key -CAcreateserial -sha256 -days 1825 \
  -extfile ca/ca.ext -out ca/srv-ca.crt
: > ca/issued.ledger
ls root ca

Generating two 4096-bit RSA keys takes a few seconds and prints a progress trace to standard error. That trace is noise, not output you need to keep.

pathlen:1 on the root and pathlen:0 on the issuing CA say that the root may sign one intermediate and that the intermediate may sign no further CAs. -CAcreateserial creates the root’s own .srl file, so you have already met the mechanism this lab is about, one tier up.

Task 3 — Issue the first certificate through a repeatable helper

Real authorities issue through a program, not by hand, because the ledger has to be written by the same thing that writes the certificate. Build the smallest honest version of that.

cat > "$LAB/issue.sh" <<'EOF'
#!/bin/bash
# issue.sh NAME - mint one leaf certificate and record it in the ledger.
set -eu
CN="$1"
LAB="$HOME/rbpki-lab-26"
CA="$LAB/ca"
OUT="$LAB/issued"
cat > "$OUT/$CN.ext" <<EXT
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:$CN.lab.example
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EXT
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
  -out "$OUT/$CN.key"
openssl req -new -key "$OUT/$CN.key" -sha256 \
  -subj "/CN=$CN.lab.example" -out "$OUT/$CN.csr"
openssl x509 -req -in "$OUT/$CN.csr" -CA "$CA/srv-ca.crt" \
  -CAkey "$CA/srv-ca.key" -CAcreateserial -sha256 -days 90 \
  -extfile "$OUT/$CN.ext" -out "$OUT/$CN.crt"
SERIAL=$(openssl x509 -in "$OUT/$CN.crt" -noout -serial | cut -d= -f2)
printf '%s\t%s\t%s\n' "$SERIAL" "$CN.lab.example" \
  "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$CA/issued.ledger"
echo "issued $CN.lab.example serial $SERIAL"
EOF
chmod +x "$LAB/issue.sh"
"$LAB/issue.sh" app
cat "$LAB/ca/srv-ca.srl"

The first issuance creates srv-ca.srl, and the value inside it is the serial that was just used. Every later issuance reads that file, adds one, and writes the result back. That is the entire counter, and it is a single line of text on one disk.

$ cd "$HOME/rbpki-lab-26/issued" && openssl verify -CAfile ../root/root.crt -untrusted ../ca/srv-ca.crt app.crt

app.crt: OK

$ openssl x509 -in "$HOME/rbpki-lab-26/issued/app.crt" -noout -subject -issuer -serial -dates

subject=CN=app.lab.example
issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

Task 4 — Take a backup you can verify

Back up the CA directory, and take a digest of every component before you archive it. A backup you cannot verify is a story about a backup.

cd "$LAB/ca"
sha256sum srv-ca.key srv-ca.crt srv-ca.srl issued.ledger ca.ext \
  > "$LAB/backup/backup-manifest.txt"
tar -czf "$LAB/backup/ca-backup.tar.gz" \
  srv-ca.key srv-ca.crt srv-ca.srl issued.ledger ca.ext
cat "$LAB/backup/backup-manifest.txt"
tar -tzf "$LAB/backup/ca-backup.tar.gz"

Task 5 — Keep issuing, because a real CA does not pause

Three weeks pass in the scenario. Two more services are onboarded.

$ "$HOME/rbpki-lab-26/issue.sh" api && "$HOME/rbpki-lab-26/issue.sh" web

cat "$LAB/ca/issued.ledger"
cat "$LAB/ca/srv-ca.srl"

The ledger now has three lines and the counter has advanced twice. The backup you took in Task 4 has one line and the original counter. That divergence is not a mistake; it is what a backup is. The mistake comes next, and it is in how the restore is performed.

Task 6 — Lose the issuing CA

rm -rf "$LAB/ca"
ls "$LAB"

$ cd "$HOME/rbpki-lab-26/issued" && openssl verify -CAfile ../root/root.crt app.crt

error 20 at 0 depth lookup: unable to get local issuer certificate

Nothing is wrong with app.crt. It is the same bytes it was in Task 3. Error 20 is a statement about what the verifier could reach, not about the certificate in front of it, and confusing those two is the single most common misdiagnosis in chain troubleshooting.

Task 7 — Restore, and prove it is the same authority

mkdir -p "$LAB/ca"
cd "$LAB/ca"
tar -xzf "$LAB/backup/ca-backup.tar.gz"
sha256sum -c "$LAB/backup/backup-manifest.txt"

Every line must report OK. That proves the bytes survived the round trip; it does not yet prove the key belongs to the certificate. Prove that separately, because a restore that mixes generations of a key and a certificate produces failures that look like anything but a restore problem.

openssl pkey -in "$LAB/ca/srv-ca.key" -pubout | openssl sha256
openssl x509 -in "$LAB/ca/srv-ca.crt" -noout -pubkey | openssl sha256

Both commands print a SHA-256 digest of the same public key in two different encodings of the same object. The reference capture looks like this, and the only thing that matters is that your two lines are identical to each other:

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

$ cd "$HOME/rbpki-lab-26/issued" && openssl verify -CAfile ../root/root.crt -untrusted ../ca/srv-ca.crt app.crt

app.crt: OK

{
  date -u +'%Y-%m-%dT%H:%M:%SZ'
  openssl pkey -in "$LAB/ca/srv-ca.key" -pubout | openssl sha256
  openssl x509 -in "$LAB/ca/srv-ca.crt" -noout -pubkey | openssl sha256
  cd "$LAB/issued" && openssl verify -CAfile ../root/root.crt \
    -untrusted ../ca/srv-ca.crt app.crt api.crt web.crt
} > "$LAB/restore-proof.txt"
cat "$LAB/restore-proof.txt"

All three certificates validate, including the two issued after the backup was taken. At this point the restore looks complete, and every runbook that stops here would call it done.

Task 8 — Issue one certificate, and create a collision

cat "$LAB/ca/srv-ca.srl"
cat "$LAB/ca/issued.ledger"
"$LAB/issue.sh" shop

The counter came back from a three-week-old backup, so it is pointing at a serial that the CA consumed two issuances ago. The ledger that came back with it lists one certificate, not three. The new certificate is therefore signed with a serial number that is already in use, and nothing in the process objects.

$ cd "$HOME/rbpki-lab-26/issued" && for f in *.crt; do openssl x509 -in "$f" -noout -serial | cut -d= -f2; done | sort | uniq -d

Build the full inventory and look at the two certificates that share the serial.

cd "$LAB/issued"
for f in *.crt; do
  printf '%s\t%s\n' \
    "$(openssl x509 -in "$f" -noout -serial | cut -d= -f2)" "$f"
done | sort > "$LAB/issued-serials.txt"
cat "$LAB/issued-serials.txt"
openssl verify -CAfile ../root/root.crt -untrusted ../ca/srv-ca.crt \
  api.crt shop.crt
openssl x509 -in api.crt -noout -serial -subject -fingerprint -sha256
openssl x509 -in shop.crt -noout -serial -subject -fingerprint -sha256

The two serial= values are identical. The two subject= values are different, and so are the two fingerprints: these are unambiguously two different certificates. Both are reported OK. No tool involved in issuing, validating or serving them will ever mention it.

Task 9 — Reconcile the counter from what was actually issued

The ledger is not the authority on what was issued, because it came back from the same stale backup. The certificate archive is. Take the highest serial that exists on any certificate and move the counter past it.

MAX=$(cut -f1 "$LAB/issued-serials.txt" | sort | tail -n 1)
echo "highest serial present in the archive: $MAX"
printf '%s\n' "$MAX" > "$LAB/ca/srv-ca.srl"
cp "$LAB/issued-serials.txt" "$LAB/ca/issued.ledger.rebuilt"

Sorting the serials as text is correct here only because every serial this CA has produced is the same width in uppercase hexadecimal. A CA whose serials vary in width needs a numeric comparison, and a CA of any real size needs an issuance database rather than a file.

$ "$HOME/rbpki-lab-26/issue.sh" pay

cd "$LAB/issued"
for f in *.crt; do
  printf '%s\t%s\n' \
    "$(openssl x509 -in "$f" -noout -serial | cut -d= -f2)" "$f"
done | sort > "$LAB/issued-serials.txt"
cut -f1 "$LAB/issued-serials.txt" | uniq -d > "$LAB/remaining-duplicates.txt"
cat "$LAB/issued-serials.txt"
cat "$LAB/remaining-duplicates.txt"

One duplicated serial remains, and it always will. Reconciliation stops the authority from making the mistake again; it cannot unmake a certificate that has already been signed. The pair from Task 8 has to be revoked and reissued, and that is a change with a service impact, which is the real cost of the missing counter.

Task 10 — Capture the deliverables

cd "$LAB"
{
  echo "issued serial inventory after reconciliation:"
  cat "$LAB/issued-serials.txt"
  echo "repaired counter:"
  cat "$LAB/ca/srv-ca.srl"
  echo "duplicated serials remaining:"
  cat "$LAB/remaining-duplicates.txt"
} > serial-reconciliation.txt

cat > collision-report.md <<'EOF'
# Serial collision after a stale restore

Cause: the CA was restored from a backup taken before two further
issuances. The restore returned the serial counter and the ledger to
their state at backup time, so the next issuance reused a serial that
was already in use.

Effect: two certificates with different subjects and different
fingerprints carry one serial. Both validate. A CRL entry for that
serial revokes both, so neither can be revoked independently.

Repair: the counter was reconciled from the certificate archive and
the ledger rebuilt from the same source. Both certificates in the
colliding pair must now be revoked and reissued.

Prevention: back up the counter and the issuance record with the key,
verify a restore by issuing into a test chain, and reconcile the
counter against the archive as the first step of every restore.
EOF
ls -l restore-proof.txt collision-report.md serial-reconciliation.txt
ls -l backup/ca-backup.tar.gz backup/backup-manifest.txt

Validation

• The sha256sum -c in Task 7 reports OK for all five components at the moment of extraction. Running it again after Task 9 correctly reports two mismatches, because issuing advances both the counter and the ledger; a manifest is a statement about one instant.
• The two digests printed in Task 7 are identical to each other. If they differ, the restored key does not belong to the restored certificate and the CA cannot sign anything usable.
• restore-proof.txt shows OK for app.crt, api.crt and web.crt, which proves the restored authority still validates material it signed before the loss.
• remaining-duplicates.txt contains exactly one serial, and the certificate issued in Task 9 does not carry it.
• The deliverables ca-backup.tar.gz, backup-manifest.txt, restore-proof.txt, collision-report.md and serial-reconciliation.txt exist and are non-empty.

Expected Outcome

$HOME/rbpki-lab-26/
├── backup/
│   ├── backup-manifest.txt
│   └── ca-backup.tar.gz
├── ca/
│   ├── issued.ledger
│   ├── issued.ledger.rebuilt
│   ├── srv-ca.crt
│   ├── srv-ca.key
│   └── srv-ca.srl
├── issued/
│   ├── api.crt
│   ├── app.crt
│   ├── pay.crt
│   ├── shop.crt
│   └── web.crt
├── root/
│   ├── root.crt
│   ├── root.key
│   └── root.srl
├── collision-report.md
├── issue.sh
├── issued-serials.txt
├── remaining-duplicates.txt
├── restore-proof.txt
├── serial-reconciliation.txt
└── state.pre-lab

Only the certificates are listed under issued/; each name there also has a .key, a .csr and a .ext file alongside it, written by the issuance helper.

You can now say precisely what a certificate authority backup has to contain, demonstrate the failure that follows from getting it wrong, and show the reconciliation step that belongs at the top of every CA restore procedure.

Troubleshooting

issue.sh reports that the extension file cannot be opened. The script writes the extension file into $LAB/issued, which Task 1 created. If you started at Task 3 without Task 1, create the directory and run the helper again.

The two digests in Task 7 do not match. The archive was created from a directory where the key and the certificate came from different generations of the CA. Rebuild from Task 2; there is no repair for this once the original key is gone.

sha256sum -c reports that a file could not be opened. Run it from $LAB/ca after extracting. The manifest stores bare filenames, so the working directory has to be the one the archive unpacked into.

Task 8 produces no duplicate. The counter file was not part of the restore, so OpenSSL generated a fresh random serial instead of continuing an old sequence. Confirm srv-ca.srl appears in tar -tzf "$LAB/backup/ca-backup.tar.gz", restore it, and re-run the issuance.

Cleanup

LAB="$HOME/rbpki-lab-26"

# 1. Nothing to stop: this lab started no service and no container.
ls "$LAB"

# 2. Confirm against the Task 1 capture before removing anything.
cat "$LAB/state.pre-lab"

# 3. Remove the lab directory, including both private keys.
rm -rf "$LAB"
test ! -e "$LAB" && echo "lab directory removed"

Cleanup is complete when step 3 prints lab directory removed and openssl version still reports the version recorded in state.pre-lab. No trust store was modified at any point, so there is nothing else on the host to restore.

Production notes

• Rehearse the restore, not the backup. A restore rehearsal that ends by issuing a certificate into a test chain would have caught the collision in this lab within a minute of the archive being unpacked.
• Reconcile the counter as step one of every restore, before the first issuance. On an authority with a real issuance database this is a query; with a .srl file it is the scan you ran in Task 9.
• Keep every certificate you issue. The archive was the only trustworthy source of truth in this lab, because the ledger had been rolled back with everything else. A CA that discards its issued certificates cannot reconcile anything after a loss.
• Treat the extension profile as CA state. Certificates issued after a restore that lost the profile differ from the ones before it in key usage and constraints, and those differences reach clients as intermittent, service-specific failures months later.

What You Learned

• CA state is more than a key and a certificate. The serial counter, the issuance record and the extension profile decide whether a restored authority behaves like the one you lost.
• A stale counter produces duplicate serials in silence. No issuance tool, no verifier and no server reports it, because uniqueness is the CA’s obligation and nobody else’s check.
• Duplicate serials destroy precise revocation. A CRL entry names a serial, so one entry covers every certificate carrying it, and the choice of which to revoke no longer exists.
• Some keys are worth not being able to restore. A key that can be backed up can be stolen from the backup. Where that risk dominates, continuity comes from a second device holding the key from the moment it was generated, and from a plan to replace the authority rather than recover it.