Skip to main content
RunBook Academy

Secrets, PKI & CertificatesXIX · Production ArchitectureArchitecture

Least privilege in two dimensions: retrieving a secret and using a key

Advanced⏱ ~22 minssh-keygen

What you'll learn

  • Separate the grant to retrieve a value from the grant to invoke an operation, and compare their duration, detectability and reversibility
  • Read the PKCS#11 attributes that distinguish a key that cannot be exported from one that merely is not exported today
  • Decide from the verification model whether use-without-release is achievable for a given credential
  • Bound the residual risk of an operation oracle with scope, rate and per-use authentication

Prerequisites

Verified against OpenSSL 3.5.x teaching target; 3.0+ minimum · OpenSSH 10.x teaching target; 8.2+ minimum for certificate workflows · OpenBao 2.6.x · Smallstep step-ca 0.30.x · Certbot / Pebble Certbot current release; Pebble 2.10.x ACME test server · Kubernetes (cross-course target) 1.36.x · PostgreSQL 17.x · 2026-08-26

Not yet marked complete on this device.

Two authorisations get written in the same policy grammar and are routinely treated as one. The first says an identity may obtain the bytes of a secret. The second says an identity may cause a key to be used and will never see it. The gap between them decides whether a compromise is a bounded event or a permanent one, and the second grant is available in far more places than most designs exploit.

Two verbs that look like one

In almost every policy engine both grants appear as a capability attached to a path. A capability on a storage path returns a stored value. A capability on an endpoint that performs a computation returns the result of that computation. The grammar is identical. The consequences diverge on three axes, and none of them is cosmetic.

  • Duration. A retrieval grant is bounded by policy, but the value it produced is not bounded by anything. Once the bytes exist in a process heap, a core dump, a log line or a nightly backup, the grant that produced them has no further bearing on who can use them. A use grant leaves nothing behind, so it stops exactly when the caller’s authorisation stops.
  • Detectability. Copying a value you already hold creates no event on any system. Every invocation of a use grant creates a record on the holding side, carrying a caller, a timestamp and the operation requested. Exfiltration is silent; abuse of an oracle is noisy.
  • Reversibility. Withdrawing a retrieval grant does not withdraw the copies, so remediation means rotating whatever the value protected. Withdrawing a use grant stops the next call outright, with nothing to rotate.

This is why “that credential is rotated quarterly” and “that key has never been outside the module” are not comparable claims about strength. The first describes how often a bounded window restarts. The second describes a window that was never opened.

What a non-exportable key actually is

Hardware-backed key storage is usually described in a single word, and the word hides a distinction that matters at audit time. In PKCS#11 the property is carried by two independent booleans. If CKA_SENSITIVE is CK_TRUE, or if CKA_EXTRACTABLE is CK_FALSE, then certain attributes of the key cannot be revealed in plaintext outside the token. Both are one-way: once set they become read only, so an operator cannot quietly walk them back.

The trap is that those two describe the key’s state now, not its history, and that a sensitive key which is still extractable can be wrapped out of the token under another key.

AttributeValue worth havingWhat it actually tells you
CKA_SENSITIVECK_TRUEPlaintext readout is suppressed at present
CKA_EXTRACTABLECK_FALSEThe key cannot be wrapped out at all
CKA_ALWAYS_SENSITIVECK_TRUEIt has never been readable
CKA_NEVER_EXTRACTABLECK_TRUEIt has never been wrappable
CKA_WRAP_WITH_TRUSTEDCK_TRUEOnly a trusted wrapping key may wrap it; the default is CK_FALSE
CKA_ALWAYS_AUTHENTICATECK_TRUEA PIN is required for each individual sign or decrypt

An auditor who reads only the first two rows can be told a true thing that means nothing. The rows that answer the question actually being asked are the third and fourth, because they are the only ones that speak about every moment since the key was created. When extraction is genuinely impossible the token refuses with a specific error rather than a generic one:

CKR_KEY_UNEXTRACTABLE: The specified private or secret key can't be
wrapped because its CKA_EXTRACTABLE attribute is set to CK_FALSE.

Where the second dimension is available

The dividing line is not the product. It is what the verifying party checks.

flowchart TD
    Q{"How does the far side\ncheck this credential?"}
    Q -- "compares a value it also holds" --> V["The bytes must travel\nretrieval is unavoidable"]
    Q -- "verifies a signature\nor decrypts" --> S["Only the result must travel\nuse-without-release is possible"]
    V --> V2["Database passwords, API keys,\nshared HMAC secrets"]
    S --> S2["CA signing, TLS server auth,\nSSH, token issuance, unwrapping"]
    S2 --> H["Bind the key to a module,\nexpose only the operation"]

If the verifier compares a value it also stores, that value has to reach it, and no amount of hardware changes the requirement. Every database password lives here, which is why the honest control for a database credential is a short lease and a distinct principal per consumer rather than a promise about storage. If the verifier instead checks a signature or decrypts under a public key, the private half has no reason to move, and any design that moves it anyway has chosen convenience.

Certificate authorities are the clearest case, because the whole protocol is a request for one operation. An SSH certificate authority can sign with a key that lives in a token and is identified only by its public half:

# The CA private key stays in the token. ssh-keygen sends the data to
# be signed and receives the signature; -s takes the public half.
CA_PUB=/etc/ssh/user_ca.pub
PKCS11_LIB=libpkcs11.so
ssh-keygen -s "$CA_PUB" -D "$PKCS11_LIB" -I "alice@lab.example" \
  -n deploy -V -5m:+1h alice.pub

The same invocation with a plain private key file and no -D produces an identical certificate and a completely different architecture: in the second case the CA key is a file that a backup agent, a snapshot, and anyone with read access to that path now shares. When the token is not available, the CA key may also be held by an agent and identified by its public half in the same way, which is weaker than a token but still keeps the private value out of the signing tool’s address space.

The oracle you built instead

Use-without-release does not remove risk. It converts an unbounded, silent risk into a bounded, observable one, and the bounding is your job rather than the module’s.

  • Scope the operation, not just the key. An authority that will sign arbitrary bytes with a CA key is an authority that will sign a certificate. Constrain which key, which mechanism, and what shape of input the caller may submit.
  • Rate limit and alert on volume. Because every call is recorded, a signing rate that departs from its baseline is a genuine detection opportunity. This is the compensating control that retrieval grants can never offer.
  • Require a human for the rare and catastrophic. Per-use authentication turns a root CA signature into an event somebody has to attend, which is exactly right for an operation that happens twice a decade.
  • Know what rotation does not repair. Old key material is retained so that previously protected data still decrypts, and a new wrapping key does nothing about a data key that has already leaked. Re-encryption is a separate job you have to build and schedule.

Production discipline

  1. Write the two grants as separate rows in the design. For every key, state whether any identity may obtain it and, separately, which identities may invoke which operations with it. A design that only has one column has silently chosen retrieval.
  2. Generate hardware-backed keys inside the hardware. Import is the step that makes the never-extractable attributes untrue, and it cannot be undone by settings applied afterwards.
  3. Treat every use grant as a rate-limited endpoint. Give it a baseline, an alert threshold and an owner, on the grounds that the observability is the entire reason you chose it.
  4. Say out loud which credentials cannot have the second dimension. Shared-value credentials are not a failure of ambition; they are a property of the protocol. Spend the effort on lease length and per-consumer identity instead of on storage theatre.

Cross-course references

  • Linux for Production Sysadmins - Part LXXI (TLS) covers the serving process that must open a private key file, which is the canonical case where release cannot be avoided and file ownership becomes the only remaining control.
  • Kubernetes for Production Sysadmins - Part LXV (Secrets Security) covers delegating envelope encryption of cluster data to an external key service, which is this lesson’s wrapping-key grant applied to etcd.
  • Docker & Containers for Production Sysadmins - Part XXXVIII (Secrets) covers delivering a value into a running process, the point at which a retrieval grant becomes unavoidable and the question turns into how long the value survives.

Quiz

Knowledge check · 4 questions

  1. Q1. An auditor is told a signing key is non-exportable and is shown that CKA_SENSITIVE is CK_TRUE and CKA_EXTRACTABLE is CK_FALSE. Which attributes should they ask for next?

  2. Q2. A hardware security module can allow an application to authenticate to a database without the password ever reaching the application.

  3. Q3. Give the three axes on which a grant to retrieve a value differs from a grant to invoke an operation with a key.

  4. Q4. Decide which of the two dimensions this design is actually using, and what the rotation claim is worth.

    A platform team encrypts application data with data keys wrapped by a managed key in a cloud key service. The managed key is rotated automatically every 365 days. During an incident review it emerges that a plaintext data key was written to an application log in March and the log was retained. The team's position is that the automatic rotation in June has already remediated the exposure.

Passing score: 75%. Answers are checked in this browser.