Monitoring a secret manager: availability, denials and leases

A secret manager sits on the critical path of every workload that reads a credential from it, which makes its failure modes worth knowing individually rather than as one availability number. Five classes matter, they have different blast radii, and three of them produce no error on the manager’s own health endpoint at all.

The five things that can go wrong

flowchart TD
    A["Workload request"] --> B{"Reachable and unsealed?"}
    B -- "no" --> C["Every consumer fails at once"]
    B -- "yes" --> D{"Caller credential still valid?"}
    D -- "no" --> E["Authentication failure for that caller"]
    D -- "yes" --> F{"Policy permits this path?"}
    F -- "no" --> G["Denied, and recorded as not allowed"]
    F -- "yes" --> H{"Backend able to issue?"}
    H -- "no" --> I["Issuance failure: only new consumers affected"]
    H -- "yes" --> J["Lease granted, with a TTL somebody must renew"]

Follow one request down the diagram and each branch is a different page. Unreachable or sealed takes out everything simultaneously. An expired caller credential takes out one workload, or one whole role if the credential was shared. A policy denial takes out one code path, typically the one added in this morning’s deployment. An issuance failure spares everything already running and breaks everything that starts next. A lease that nobody renews expires on a timer, which means the failure arrives late and all at once.

Alert on rates per class, not on a combined error count. The authentication failure rate for one role, the denial rate for one policy path, and the issuance failure rate for one backend are three different questions with three different owners, and summing them produces a number that is never actionable.

Sealed is an outage with a human in the recovery path

Sealing is not a fault state, it is the normal condition of an instance whose barrier key is not in memory. The data is intact on disk and completely unreadable. Every request gets a distinct answer:

$ bao kv get kv/app/config
Code: 503. Errors:

* Vault is sealed

Two details deserve attention. The status is 503, which places this firmly in the availability class rather than the authorisation class, and a monitoring check that only opens a TCP connection or requests an unauthenticated endpoint will see a healthy listener throughout. Seal state has to be read explicitly, per node, and alerted on directly.

The second detail is what unsealing requires. Initialising with split key shares produces a threshold scheme, and the manager keeps none of the material needed to reconstruct the key:

Vault initialized with 3 key shares and a key threshold of 2. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 2 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 2 keys to
reconstruct the root key, Vault will remain permanently sealed!

That text is quoted exactly as the tool prints it, including its habit of still saying Vault. Read it as an operational statement: a restart puts the service into a state that no automation you own can leave, because the recovery input is held by people. Unseal share holders are therefore part of the on-call rota design, their availability is part of your recovery time, and the number of reachable holders is itself worth monitoring against the threshold.

A denial is a record, and the record has structure

The authorisation signal is the most useful one in the system, because a denial is simultaneously a security event and a deployment defect and the record tells you which. Here is one, as the audit device wrote it:

{"time":"2026-08-26T21:26:20.555636512Z","type":"response",
 "auth":{"client_token":"hmac-sha256:da33377eba1c...","accessor":"hmac-sha256:5eb6ce9e...",
         "policies":["app-read","default"],"policy_results":{"allowed":false}, ...},
 "request":{"operation":"read","mount_point":"kv/","mount_type":"kv",
            "path":"kv/data/app/other","remote_address":"127.0.0.1", ...},
 "response":{"data":{"error":"hmac-sha256:b0a0f532..."}},
 "error":"1 error occurred:\n\t* permission denied\n\n"}

Everything an operator needs to triage is in that one object. The policy set the caller was carrying, the exact operation, the exact path, the caller address, and an explicit allowed result of false. An alert built on this can carry all of it, which is the difference between a page that says permission denied and a page that says the token holding app-read was refused a read of kv/data/app/other from that address.

Reading the path prefix is the skill worth teaching. A key-value store of this generation splits its data paths from its metadata paths, so a policy granting read on kv/data/app/config permits exactly that and nothing else: a read of a sibling data path is denied, a write to the same path is denied, and a list is denied because listing is an operation against the metadata path rather than the data path. A denial whose path is under the metadata prefix with a list operation is almost always a policy that was written against the human-facing path rather than the API path. That is a one-line fix, not an intrusion, and an operator who knows the distinction does not escalate it at three in the morning.

Issuance and leases fail on a delay

Dynamic credentials are created in the target system on demand and destroyed when their lease ends. A grant looks like this:

lease_id        database/creds/app-readonly/xoHI541EXoFgKn1OTiusOdd9
expire_time     2026-08-26T21:25:43.891514518Z
issue_time      2026-08-26T21:23:43.891514368Z
renewable       true
ttl             1m59s

Two independent timers are now running. The manager holds the lease and will revoke at expire_time, and the account it created in the target system carries its own validity set at creation. They are meant to agree. When they do not, the credential either dies early, which the application sees as an authentication failure, or outlives its lease, which nothing sees at all.

The failure shapes follow from the mechanics and are worth predicting before you meet them. An issuance failure, where the manager cannot reach the backing system to create an account, does not disturb a single running process, because their credentials already exist. It breaks every instance that starts next, which means it presents as a failed deployment or a failed scale-up and gets diagnosed as an application problem for the first twenty minutes. A renewal failure is even quieter: a fleet that started together holds leases that expire together, so nothing at all happens until the TTL elapses and then every instance loses its credential within the same minute, which looks exactly like the database falling over.

Revocation deserves its own signal. When a lease ends the account is removed, and the application’s next connection attempt reports that the role does not exist rather than that a password is wrong. Recognising that message as lease expiry rather than as database corruption saves an incident from starting in the wrong place. Track the count of active leases as a series: a sharp fall means mass revocation, and a sharp rise usually means renewal is failing and consumers are papering over it by requesting fresh credentials.

Production discipline

1. Read seal state per node, not service health. A sealed instance answers on its port and fails every request, so any check that stops at connectivity reports a healthy outage.
2. Count unseal share holders you can actually reach. Recovery time is bounded by human availability, so the threshold and the rota need to be reviewed together like any other dependency.
3. Route denials by policy, not by severity. The team whose deployment created the denial can fix it in minutes; the platform rota can only forward it.
4. Alert on issuance failure separately from availability. It spares everything already running, so it never shows up in request success rates until a deployment goes out.
5. Watch lease counts as a series. Absolute numbers say little, while a step change identifies mass revocation or a renewal path that has quietly stopped working.
6. Monitor audit output freshness. The record that reconstructs an incident is worthless if it stopped being written before the incident began.

Cross-course references

• Observability for Production Sysadmins - Part XVIII (AlertingRules) covers expressing rate and absence conditions as rules, which is how the five signal classes here become five distinct alerts rather than one error count.
• Kubernetes for Production Sysadmins - Part XCIV (AuditLogs) covers reading an authorisation audit stream as an operational signal, the same pattern applied to a different policy engine.
• Linux for Production Sysadmins - Part XXXI (Audit) covers host-level audit records and their retention, the layer beneath the application audit device described here.

Quiz