Capstone — build and defend a production secrets and PKI architecture

Every control in this course has so far been examined on its own. The capstone puts them in one estate and asks whether they hold together. You will build a trust hierarchy, an automated issuance path, a secret manager and the credentials it hands out, all on containers you can destroy, and then you will attack your own build and prove that each control does what you claim it does.

What you are building, and the diagram you owe

The estate is deliberately small: one root, one issuing authority, one ACME server, one TLS service, one SSH certificate authority, one secret manager, one database, one workload and one monitor. Nothing here needs a cloud account or a spare machine. Every component runs as a container on the same host, using the same fixtures the labs use, so that stage thirteen can destroy half of it without consequence.

Before you build anything, redraw the diagram below as your own. Use your own host names, your own validity windows and your own arrows, and add the two flows this version deliberately leaves incomplete: how a failure reaches a human, and how each authority is itself replaced. Your diagram is a graded deliverable, and it is the artefact a reviewer will read first.

flowchart TD
    ROOT["Offline root CA\npowered off between ceremonies"]
    ISS["Issuing CA\nonline, constrained by the root"]
    ACME["Internal ACME server"]
    TRUST["Trust anchor distribution\nOS store and runtime store"]
    SVC["TLS service\nrenewed and reloaded"]
    SSHCA["SSH certificate authority\nbounded principals and validity"]
    ADMIN["Operator session\ncertificate, not a key file"]
    BAO["Secret manager\nsealed until quorum"]
    WL["Workload\nno credential at rest"]
    DB["Database\nshort-lived dynamic roles"]
    MON["Probes\nexpiry and availability"]
    ROOT --> ISS
    ROOT --> TRUST
    ISS --> ACME
    ACME --> SVC
    TRUST --> SVC
    SSHCA --> ADMIN
    WL --> BAO
    BAO --> DB
    SVC --> MON
    BAO --> MON

Read the diagram as two independent trust systems that meet only at the host. On the left, an offline root vouches for an online issuing authority, which sits behind an enrolment service, and a separate distribution path carries the anchor to every client that must recognise it. On the right, a secret manager holds a barrier that stays closed until a quorum opens it, and hands out credentials that expire on their own. The SSH authority is a third, smaller system with no intermediates at all. A weakness in any one of the three is contained by the other two only if you can say, out loud, where the boundary between them runs.

Stages 1 to 3 — the hierarchy and the certificates it issues

The first three stages produce a chain that validates, a leaf whose identity is unambiguous, and an automated renewal that actually reaches the running process. Nothing later in the capstone works if these are wrong, and the third one is where most real estates fail.

Stage 1 repeats secrets-pki-lab-04-build-two-tier-certificate-authority and stage 2 repeats secrets-pki-lab-05-issue-server-certificate-san-eku; revisit either if a constraint does not come out as you expected. The path length on the root is not decoration. A pathLenConstraint may only appear where the certificate is a certificate authority and asserts keyCertSign, and setting it to zero on the issuing authority is what stops that authority minting a third tier you never approved.

PKI="$HOME/rbpki-capstone/pki"
cd "$PKI" || exit 1

# Identity is whatever the SAN says, and nothing else.
openssl verify -CAfile root.crt -untrusted srv-ca.crt \
  -verify_hostname app.lab.example app.crt
openssl verify -CAfile root.crt -untrusted srv-ca.crt \
  -verify_hostname wrong.lab.example app.crt
openssl verify -CAfile root.crt -untrusted srv-ca.crt \
  -purpose sslclient app.crt

Those three calls are your stage 2 evidence, and they must produce three different results:

app.crt: OK
error 62 at 0 depth lookup: hostname mismatch
error 26 at 0 depth lookup: unsuitable certificate purpose

Stage 3 is the one to take seriously. Renewal writes a file; serving reads memory. Prove the gap exists before you prove you have closed it, by renewing once with the reload hook removed and capturing both serials.

HOST=app.lab.example
PORT=8443
LIVE=/etc/ssl/certs/app.lab.example.pem

# What the process is serving, read over the network.
openssl s_client -connect "$HOST:$PORT" -servername "$HOST" </dev/null 2>/dev/null |
  openssl x509 -noout -serial

# What the renewal wrote to disk.
openssl x509 -in "$LIVE" -noout -serial

Two identical serials after a renewal and a reload is a pass. Two different serials, with the older one on the wire, is the failure this stage exists to make visible. Labs secrets-pki-lab-13-automate-issuance-with-acme, secrets-pki-lab-14-prove-renewal-and-reload and secrets-pki-lab-15-run-internal-acme-ca cover the mechanics.

Stages 4 to 6 — distribution, SSH identity and the barrier

Stage 4 is the stage engineers skip because it works on their laptop. It works on their laptop because their laptop has the anchor. The acceptance criterion demands two separate failures before the two successes, because the operating system store and a language runtime bundle are different stores with different refresh mechanisms, and a fleet that has one but not the other produces the report that a service works from the shell and fails from the application on the same host. Revisit secrets-pki-lab-11-install-private-trust-anchor.

Stage 5 has two traps worth naming before you meet them. Signing a key without a validity flag produces a certificate valid from the Unix epoch to the distant future, so pair the signing flag with an explicit validity interval every single time. And from OpenSSH 10.3, an empty valid-principals list matches nothing rather than matching everything, which reverses the advice in a great deal of published material. Labs secrets-pki-lab-17-build-ssh-certificate-authority and secrets-pki-lab-18-issue-and-revoke-ssh-certificates cover issuance and the key revocation list.

Stage 6 is a ceremony, not a command, and the wording the tool gives you states the obligation plainly:

Vault initialized with 3 key shares and a key threshold of 2. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 2 of these keys to unseal it
before it can start servicing requests.

That text still says Vault, because OpenBao inherited the strings; quote it as it is rather than tidying it. Your deliverable is not the transcript but the written answer to the question it raises: who holds share one, who holds share two, who holds share three, and which two of those people can be in the same room within your recovery objective. Rehearse it in secrets-pki-lab-19-deploy-secret-manager-and-unseal.

Stages 7 to 9 — policy, dynamic credentials and workload identity

Stage 7 must produce a deny as carefully as it produces an allow. A policy that has only ever been tested against the path it permits has not been tested. The list denial is the instructive one: a policy granting read on the data path of a versioned key-value store does not grant list, because listing reads the metadata path, and the refusal names a path the policy author never wrote down. Work it through in secrets-pki-lab-20-write-least-privilege-secret-policy.

Stage 8 is where the phrase short-lived stops being a slogan. The credential must stop working at the database, and the way you demonstrate that is by connecting as the dynamic user after revocation and capturing the database’s own refusal. Anything observed only at the secret manager proves that the manager forgot the credential, not that the database did. Lab secrets-pki-lab-21-issue-dynamic-database-credentials walks the lease lifecycle.

Stage 9 will not eliminate the bootstrap problem, and your writeup should say so. An application role identifier plus a secret identifier is still a credential the workload must obtain from somewhere, so the honest acceptance criterion is not that no secret exists but that nothing at rest remains useful: the identifier is single-use or short-lived, the token it buys has a lease, and a copy taken from disk after the window is worthless. State plainly which component you have chosen to trust as the first link, because that component is now the whole chain. Lab secrets-pki-lab-22-authenticate-workload-with-approle is the starting point.

Stages 10 to 13 — operate it, rotate it, break it, restore it

For stage 10, a three-state probe is the smallest honest one:

CERT=/etc/ssl/certs/app.lab.example.pem
WINDOW=$((30 * 24 * 3600))

if openssl x509 -in "$CERT" -noout -checkend "$WINDOW"; then
  echo "ok"
elif [ -r "$CERT" ]; then
  echo "expiring"
else
  echo "probe broken"
fi

Two of those three states must reach a human, and the third state is the one every estate forgets. A monitor that cannot read its input is not reporting good news, and a check whose failures are swallowed reports good news for exactly as long as it takes for the certificate behind it to expire. Build the probe in secrets-pki-lab-24-monitor-certificate-expiry, then break it on purpose and confirm you hear about it.

Stage 11 wants overlap rather than replacement: add the new credential, move consumers, remove the old one, in that order, which is the same shape for a database password and for a certificate. Rehearse in secrets-pki-lab-23-rotate-database-password-zero-downtime. Stage 12 gives you three choices, and they are not equivalent: a leaked interface token is contained by revoking it at the issuer, a compromised server key requires reissue with a fresh key rather than reissue alone, and a compromised issuing authority requires you to decide what every certificate it ever signed is now worth. Whichever you pick, the deliverable that carries the marks is the sentence describing what you did not contain. Stage 13 closes the loop with secrets-pki-lab-26-restore-pki-from-backup and secrets-pki-lab-25-recover-from-secret-manager-outage; a backup whose restore has never been executed is an untested assumption with a filename.

How the build is assessed

Score each row on evidence, not on intent. A stage with a working build and no captured proof scores zero on that row, because the proof is the deliverable.

Seventy is a pass. Three rows are gating regardless of the total: stage 3, stage 7 and stage 13. A renewal you cannot prove reached the process, a policy that has never denied anything, and a backup that has never been restored are the three failures that turn a well-built estate into an outage on a date nobody has written down.

The six questions, answered three times

Close the capstone by returning to the analysis the course opened with, in secrets-pki-i-02-six-questions-that-govern-every-credential. Answer all six, in order, for three credentials from your own build: one certificate-based credential, one bearer credential and one dynamic credential. The questions are:

1. Which principal is being trusted?
2. Which credential proves that identity?
3. Who issued it, and who checks the issuer?
4. Where does the private material live?
5. How long does it live?
6. What revokes it, and what enforces that?

Two rules make this exercise worth doing rather than pleasant to write. First, questions four and six are never answered by the artefact itself, so write them from your build rather than from the credential, and enumerate the copies of the private material by hand until the count is exact. Second, question six is answered by naming a component and saying what that component does when the revocation record is unavailable. If no component consults it, write that down, and then justify the lifetime you chose in question five as the only containment control you actually have.

Production discipline

1. Capture the evidence at the moment it exists. A serial, a log line or an exit status recorded during the stage is worth more than a reconstruction afterwards, and reconstruction is how honest people end up documenting a state that never was.
2. Test the deny before you trust the allow. Every policy, principal list and trust store in this build has a negative case, and the negative case is the one that carries the security property.
3. Write down what you did not contain. The scope of an incident response is a claim, and the unqualified version of that claim is the one that gets repeated in a review six months later by somebody who was not there.
4. Time every restore. A recovery objective is a number somebody promised; your measured restore is the number that is true, and the gap between them belongs in the design document.
5. Let lifetime absorb whatever enforcement cannot. Where you cannot name the component that reads a revocation record, shorten the credential and automate its reissuance instead of documenting an intention.

Cross-course references

• Kubernetes for Production Sysadmins - Part LXXVI (Certificates) covers a cluster whose authorities and leaves renew by entirely different mechanisms, which is the same two-deadline problem stage 13 asks you to rehearse.
• Observability for Production Sysadmins - Part LXIV (TLS Monitoring) covers probing from the consumer’s network position and alerting on an absent series, which is stage 10 implemented the way a real estate implements it.
• Linux for Production Sysadmins - Part XXXI (Systemd) covers the unit reload that carries a renewed certificate into the running process, the mechanism stage 3 exists to prove.

Quiz