Skip to main content
RunBook Academy

Secrets, PKI & CertificatesXIX · Production ArchitectureArchitecture

Trust boundaries: drawing the diagram that matters

Advanced⏱ ~22 mincoreutils

What you'll learn

  • Distinguish a trust boundary from an organisational or network boundary
  • Map the seven actors that recur in almost every secrets architecture
  • State, for each boundary crossing, which credential is presented and what the receiver independently verifies
  • Convert a boundary drawing into filesystem and policy checks a reviewer can run

Prerequisites

None — start here.

Verified against OpenSSL 3.5.x teaching target; 3.0+ minimum · OpenSSH 10.x teaching target; 8.2+ minimum for certificate workflows · OpenBao 2.6.x · Smallstep step-ca 0.30.x · Certbot / Pebble Certbot current release; Pebble 2.10.x ACME test server · Kubernetes (cross-course target) 1.36.x · PostgreSQL 17.x · 2026-08-26

Not yet marked complete on this device.

A trust boundary is the point at which a request stops being authorised by what the caller claims and starts being authorised by something the receiver can check on its own. Every credential in an estate exists to cross one. Most secrets incidents are not cryptographic failures. They are a boundary that was drawn on a whiteboard, agreed by everyone in the room, and then never enforced by any configuration file, file mode or policy document.

Why the boundary, not the secret, is the unit of design

Secret inventories are useful, and this course spends a whole part on building one. But an inventory answers the wrong question first. It tells you what exists; it does not tell you what any single item buys an attacker who obtains it. The boundary diagram does, because it records what each credential is accepted for and by whom.

Consider two estates with identical inventories: forty API tokens, twelve TLS keys, one CA key. In the first estate every token is accepted by one service for one operation and expires within the hour. In the second, three of the tokens are accepted by everything and never expire. The inventories match. The blast radii differ by orders of magnitude, and only a boundary map shows it.

Four properties decide whether a line on your drawing is really a boundary:

  • The receiver verifies independently. It reaches a verdict using material it already holds, such as a trust anchor, an issuer’s published signing keys, or a locally configured public key.
  • The check does not consult the caller. Asking the caller to confirm its own identity is not verification. That is why bearer tokens presented to a service that never validates the issuer are so much weaker than they look.
  • Crossing produces a record. A boundary you cannot observe being crossed is a boundary you cannot investigate after an incident.
  • A symmetric shared secret is not a boundary. When both sides hold the same bytes, possession proves nothing about which side is speaking, and the blast radius becomes the union of both sides.

The seven actors you will always find

Estates vary enormously in scale and almost not at all in cast. Whether you run six hosts or six thousand, the same seven roles appear, and every credential you own connects two of them.

  • The administrator. A human with a hardware-backed second factor and, ideally, no standing authority at all. Their crossing should be rare, time-boxed and loudly audited.
  • The delivery pipeline. A CI job that runs code contributed by people who are not administrators. It is the most privileged untrusted actor in most estates.
  • The secret manager. The custodian of record. It authenticates other actors, applies policy, issues short-lived credentials and writes the audit trail.
  • The KMS or HSM. The custodian of the keys that protect the custodian. It performs operations and does not release key material.
  • The application. A workload with an identity granted by the platform it runs on, not by a file someone copied onto its host.
  • The certificate authority. The actor that converts a proof of control into a signed statement other parties will believe.
  • The client. Anything that validates a certificate against a trust store, including browsers, service meshes, package managers and your own internal callers.

Drawing the crossings

flowchart TD
    ADMIN["Administrator\nhardware second factor"]
    CI["Delivery pipeline\nissuer-signed job token"]
    APP["Application\nplatform workload identity"]
    SM["Secret manager"]
    KMS["KMS or HSM\nkeys never released"]
    CA["Issuing CA"]
    CLIENT["Client\ntrust store"]
    ADMIN -->|"break-glass, time-boxed"| SM
    CI -->|"short-lived credential"| SM
    APP -->|"workload identity, verified upstream"| SM
    SM -->|"wrap and unwrap requests"| KMS
    CA -->|"sign this digest"| KMS
    CA -->|"leaf certificate"| APP
    APP -->|"possession proved in the handshake"| CLIENT
    CLIENT -->|"path built to a local anchor"| CA

Read the arrows as questions rather than as data flows. The arrow from the application to the secret manager asks how the manager knows which application is calling, and the honest answer is either the name of a platform that vouches for it or the name of a file that anyone on the host can read. The arrow from the certificate authority to the KMS asks whether the signing key can leave. The arrow from the client back to the certificate authority is drawn deliberately in the opposite direction from the certificate itself, because the client never talks to the CA at issuance time; it trusts an anchor it was configured with, possibly years earlier, by a mechanism that is itself a boundary.

What crosses, and what the receiver checks

CrossingWhat is presentedWhat the receiver verifies alone
Administrator to secret managerAn authenticated session bound to a second factorThe factor, the policy attached to the identity, and the time window
Pipeline to secret manager or cloudA short-lived token minted by the forge for one jobThe issuer’s signature, the audience, and the exact subject including branch and environment
Application to secret managerA platform-issued workload identityThe platform’s published signing keys, the token lifetime, and the bound workload
Secret manager to KMSA request to unwrap or signThe caller’s authorisation, not the plaintext, which never leaves the KMS
CA to applicationA signed leaf certificateNothing at this point; verification happens later, at the client
Application to clientThe certificate plus a signature over the handshake transcriptPath to a local trust anchor, the identity in the SAN, validity dates and key usage

The last two rows carry the idea most teams get wrong. Issuance is not verification. A certificate is a statement that becomes meaningful only when some other party, at some later moment, builds a path from it to an anchor it already trusted. Everything you do at issuance time is an investment in a check performed elsewhere, by software you do not control, possibly after your CA has stopped existing.

Turning the drawing into something enforceable

A boundary that exists only in the diagram is a plan, not a control. Each crossing should reduce to a check that a reviewer can run without the author present. Start with the crossings that involve raw key material on a host, because those are the ones a filesystem can answer directly.

# Who can read the private key material on this host, and who runs
# the process that needs it? Compare the two answers.
KEYDIR=/etc/ssl/private
stat -c '%U %G %A %n' "$KEYDIR" "$KEYDIR"/*.key
getent group ssl-cert
ps -o user=,comm= -C nginx

The output is only interesting as a comparison. If the group that can read the directory contains accounts that no process in the third command runs as, the boundary between the operator plane and the serving plane is not enforced on this host. That is a finding with an owner and a date, not a note for later.

Record the crossings themselves as data, so that a review is a diff rather than a conversation.

crossings:
  - id: pipeline-to-cloud
    from: delivery-pipeline
    to: cloud-account
    credential: issuer-signed job token
    verified_by: issuer published keys, audience, subject
    max_lifetime: one job
    evidence: trust policy pins audience and subject
  - id: app-to-secret-manager
    from: application
    to: secret-manager
    credential: platform workload identity
    verified_by: platform signing keys, bound workload
    max_lifetime: one hour
    evidence: no static credential present in the unit file

Production discipline

  1. Draw the boundaries before you choose the tooling. The diagram determines which products can even satisfy the design. Choosing the product first produces an architecture shaped like a feature list.
  2. Name the enforcement point for every crossing. A file mode, a policy path, a trust policy condition or a trust store entry. If there is none, the crossing is decorative.
  3. Treat symmetric shared secrets as merged blast radii. Where you cannot avoid one, write down both sides as a single unit and rotate them as a single unit.
  4. Walk the diagram from the attacker’s position. Put a marker on each node in turn and list what the holder can reach with one more step. The nodes that reach everything are your real priorities.
  5. Review the drawing when the estate changes, not annually. A new runner, a new cluster or a new managed service adds actors, and an unreviewed actor is an unbounded one.

Cross-course references

  • Linux for Production Sysadmins - Part LXXII (Secrets) covers where credentials actually end up on a host, which is the evidence layer beneath every boundary claim in this lesson.
  • Kubernetes for Production Sysadmins - Part LVI (Kubernetes Security Foundations) covers identities, authentication and admission, which is the same seven-actor map expressed in cluster terms.
  • Git, CI/CD & GitOps for Infrastructure Engineers - Part XXXVIII (CI Architecture) covers the control plane, runners and credentials that make the delivery pipeline the most privileged untrusted actor in the diagram.

Quiz

Knowledge check · 4 questions

  1. Q1. An architecture review finds that a deployment host and an application host authenticate to each other using the same HMAC key, held in a configuration file on both. Why does this arrangement fail to create a trust boundary?

  2. Q2. A trust boundary exists where the receiving side reaches its verdict using material it already holds, without asking the caller to confirm anything beyond the assertion being checked.

  3. Q3. Name four of the seven recurring actors in a secrets architecture and, for each, state the credential it presents when it crosses a boundary.

  4. Q4. Work out which boundaries this estate has actually enforced, and which it has only drawn.

    An estate runs one internal issuing CA on ca-1, a secret manager on bao-1 and twelve application hosts. The architecture document shows four boundaries. On investigation, the CA signing key is a file readable by the ca-operators group, which has nine members; every application host holds the same secret manager token in /etc/default/app with mode 0644; the pipeline holds a cloud access key stored as a masked CI variable that was created in 2024; and the trust anchor was distributed by a configuration management run in 2025 whose logs have since rotated away.

Passing score: 75%. Answers are checked in this browser.