Skip to main content
RunBook Academy

Secrets, PKI & CertificatesII · The Secret LifecycleLifecycle

Ownership, expiry and destruction - closing the lifecycle

Intermediate⏱ ~23 minopenbao

What you'll learn

  • Test a claimed ownership record rather than accepting the name written against it
  • Distinguish expiry enforced by the relying party from expiry recorded only by the issuer
  • List the storage layers a delete operation does not reach and explain why
  • State the preconditions that make cryptographic erasure a real destruction rather than a claim

Prerequisites

Verified against OpenSSL 3.5.x teaching target; 3.0+ minimum · OpenSSH 10.x teaching target; 8.2+ minimum for certificate workflows · OpenBao 2.6.x · Smallstep step-ca 0.30.x · Certbot / Pebble Certbot current release; Pebble 2.10.x ACME test server · Kubernetes (cross-course target) 1.36.x · PostgreSQL 17.x · 2026-08-26

Not yet marked complete on this device.

The last three stages are the ones that decide whether an estate degrades gracefully or accumulates liability. They are also the three that pass every audit on paper, because each of them is easy to assert and hard to test. A name in an owner field, a documented expiry, and a closed deletion ticket all look like controls until somebody checks whether any of them does anything.

Ownership is a claim until you test it

An owner field containing a team name proves that somebody typed a team name. Three tests convert the claim into a fact, and all three can be run without touching production.

Send a routine change request to the recorded owner and measure whether a decision comes back within the paging commitment. Ask the recorded owner to name the consumers, and compare the answer to the inventory. Then look at the last dated artefact any stage produced and ask whether the recorded owner produced it. A team that cannot decide, cannot enumerate and has produced nothing is listed as the owner rather than acting as one.

The distinction that matters most in practice is between an owner and an approver. An approver says yes or no to a proposal that somebody else brought. An owner brings the proposal. Most orphaned credentials in a mature estate have an approver recorded in the owner field, which is why nothing ever changes: no proposal arrives, so nothing is ever declined, and the record stays green.

Reorganisation is the event that creates orphans, and it does so silently because it changes nothing about the credential. The practical countermeasure is to re-attest ownership on a schedule and on every team boundary change, and to treat an unattested credential as a finding rather than as a formality.

Expiry is the control that works unattended

Every other lifecycle control requires somebody to act. Expiry acts by itself, which makes it the only stage that continues to function during the months when nobody is paying attention. That property is worth a great deal, and it is bought with a cost teams find hard to accept: expiring credentials cause outages.

Accept the cost deliberately, because it purchases something specific. A credential that expires converts a silent unbounded liability into a loud bounded one, and the noise is what builds the rotation capability. An estate whose credentials never expire has never rehearsed replacing one, so its first forced rotation happens under incident conditions.

The engineering question is who enforces the expiry. An expiry recorded only in the issuing system is a reminder. An expiry enforced by the relying party is a control. The difference is visible in a dynamically issued database credential, where the database itself holds the validity bound.

                     rolname                      |     rolvaliduntil
--------------------------------------------------+------------------------
 v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 | 2026-08-26 21:25:48+00

The role in that output stops working at the stated time because the database refuses it, not because a scheduler remembered to clean up. Revoking the same credential early goes further and removes the role outright, so the next attempt fails at the authentication layer with an unambiguous message.

$ psql -U v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423 -d appdb -c "SELECT 1;"
psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  role "v-token-app-read-ghRGRAxnCRE9Q8zLVsIw-1787779423" does not exist

Contrast that with a platform where expiry is the only bound available. Kubernetes documents that there is no API for revoking a service account token, and its guidance for an exposed token is to delete the Pod. Where revocation does not exist, the credential lifetime is the entire blast radius control, and shortening it is the only lever available.

What deletion does not reach

Deleting the value from its store is the smallest part of destruction, and the layers it does not reach are the ones with the longest retention.

flowchart TD
    D["Delete from the store"] --> A["Authoritative copy gone"]
    D -.-> S["Volume snapshot\nretained by policy"]
    D -.-> B["Backup set\nwith its own retention"]
    D -.-> V["Object version\nkept by versioning"]
    D -.-> W["Immutable tier\nlocked until expiry"]
    D -.-> R["Read replica\nor archived log"]

The solid arrow is what the delete achieved. The dotted arrows are the copies that survive it, and each survives for a structural reason rather than through negligence. A snapshot exists precisely so that it is unaffected by later changes. Object versioning exists so that a deletion is recoverable. An immutable retention tier exists so that deletion is refused until the lock expires, and that refusal is the feature somebody paid for. A backup set is a promise that the estate can be reconstituted, and reconstituting it restores the credential in working order.

The storage layer beneath adds its own reach limits. Modern filesystems and flash devices do not overwrite in place, so tools that write over a file address a logical location while the previous content remains on physical media that the tool cannot reach. This is why media sanitisation guidance treats cryptographic erase as a first-class method rather than as a shortcut.

The operational rule that follows is simple and unpopular. A copy you cannot prove destroyed must be treated as live, and the only control that reliably applies to a live copy you cannot reach is invalidating what it authenticates to. Destruction is cleanup. Invalidation is the control.

Write that reasoning down at the time, because the person who needs it will not be the person who did the work.

credential_id: reporting-db-readonly
retired_on: "2026-08-26"
retired_by: platform-data-oncall
invalidated_at_relying_party: true
copies_removed:
  - openbao kv path
  - retained object version in the state bucket
residual_copies:
  - description: nightly backup set
    safe_after: "2026-09-30"
  - description: immutable monthly archive
    safe_after: "2027-07-31"
note: "residual copies are inert because the credential was rotated"

Cryptographic erasure and what it requires

Where a copy cannot be deleted, it can sometimes be made useless. Encrypt the data, destroy the key, and the ciphertext that remains on the immutable tier or in the backup set is no longer recoverable. This is the only technique that reaches media you do not control, and it is genuinely powerful.

It is also conditional, and every condition is a place where real deployments fail.

  • The key must never have left a boundary you can account for. An escrowed or exported copy makes the erasure reversible by whoever holds it.
  • Every wrapping layer must go too. If the destroyed key was itself protected by a master key held in a key backup, the hierarchy can rederive it and nothing was destroyed.
  • Destruction must be attestable. A managed key service that schedules deletion after a mandatory waiting period gives you a date and a record; a key that was merely removed from a configuration file gives you neither.
  • The data must have been encrypted before it was stored. Erasure cannot retroactively cover a copy that was written in clear, which is the case for most leaked credentials.

That last condition is why cryptographic erasure is a data protection technique rather than a credential response. A leaked password was transmitted and stored as plaintext; there is no key to destroy. For credentials, the equivalent move is to invalidate the credential at every relying party, which is exactly what revocation does.

Production discipline

  1. Re-attest ownership on every reorganisation. The team boundary change is the orphaning event, and it produces no signal of its own in any system that holds credentials.
  2. Prefer expiry the relying party enforces. A validity bound held by the database, the certificate verifier or the token audience keeps working when the issuing system is unavailable or forgotten.
  3. Write the destruction record with its residue list. Name the copies that remain, the retention that will remove each one, and the date each becomes safe, so a future responder does not have to reconstruct it.
  4. Tombstone the inventory row rather than deleting it. An incident six months later needs to establish that this credential existed and was retired on a date, which a deleted row cannot answer.
  5. Rotate whenever the copy list is incomplete. Uncertainty about the copies is functionally identical to knowing a copy survives, and the response is the same.

Cross-course references

  • Linux for Production Sysadmins - Part XLVII (Backup) covers retention and restore paths, which is precisely where a destroyed credential returns in working order.
  • Observability for Production Sysadmins - Part XL (LogRetention) covers retention per stream, which determines how long the record of a credential outlives the credential itself.
  • Kubernetes for Production Sysadmins - Part LX (ServiceAccounts) covers token lifetimes on a platform with no revocation API, where expiry carries the entire blast radius bound.

Quiz

Knowledge check · 4 questions

  1. Q1. An exposed credential has been deleted from the secret manager and the change ticket is closed. Which statement best describes the remaining risk?

  2. Q2. Cryptographic erasure is an appropriate response to a password that was leaked in a pipeline log, because destroying the encryption key renders the leaked copy unusable.

  3. Q3. Explain the difference between an expiry recorded by the issuer and an expiry enforced by the relying party, and say why the distinction changes an incident response.

  4. Q4. Assess whether the destruction is complete and specify what must happen before the incident can be closed.

    A shared reporting credential at example.com was removed from the secret manager at 11:00 UTC after appearing in an exported spreadsheet. The owner field names a team that was merged into another group in March. Object versioning is enabled on the state bucket that also referenced the credential. The reporting database is backed up nightly to a tier with 35 day retention, and one monthly backup is written to an immutable archive locked for one year. The credential still authenticates successfully when tested at 11:40 UTC.

Passing score: 75%. Answers are checked in this browser.