← All runbooks in Secrets, PKI & Certificates

medium riskservice affecting~35 min

Runbook: Troubleshoot Secret Manager Authentication Failure

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-lab-19-deploy-secret-manager-and-unseal
secrets-pki-lab-22-authenticate-workload-with-approle
Shell access to the host or container that is failing to authenticate, not only to your own workstation
Read access to the audit log written by the active node, or to the pipeline that receives it
Authority to read the auth method role configuration at auth/approle/role/app-role, which is a configuration read and not a secret read
A named owner for the affected workload who is reachable for the duration of the incident
Written agreement that no root token will be minted during this procedure without the security owner present

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Classify the refusal by status code before you touch anything. Re-run the failing call yourself and copy the whole error, not a summary of it. Code: 503 with * Vault is sealed is a service outage and belongs to the recovery runbook. Code: 403 is a refusal aimed at one caller. A connection refused or a TLS error is neither, and means you are diagnosing the network.
· Confirm the node is reachable and unsealed from the failing host. Run bao status from inside the same network namespace as the workload. The transcripts recorded exit status 0 when unsealed and exit status 2 for both the sealed and the uninitialised states, so the exit code alone is not enough. Read the Sealed and Initialized lines.
· Write down the time of the last known success and the time of the first failure. Authentication failures that arrive on a clean boundary, an hour or twenty minutes after a deployment, are almost always lifetime arithmetic rather than a broken auth method.
· Read the role configuration before you change it. bao read auth/approle/role/app-role returns the token_ttl and token_max_ttl that govern how long any token issued to this workload can live. Record the values so you can restore them.
· Confirm an audit device is enabled and writing. bao audit list -detailed must show a device. OpenBao is fail-closed on auditing: it will not answer requests that no enabled audit device can record, and a device whose destination is blocked makes requests hang rather than fail.
· Establish the blast radius before you act. Check whether one workload is failing or every workload on the mount is failing. One is a credential problem, all of them is an auth method or service problem, and the two have different owners.
· Agree the stop line out loud. No root token, no widening of a policy, and no removal of a SecretID constraint will be used to end this incident. Those are the actions that turn a fifteen minute authentication failure into a quarterly audit finding.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Reproduce the failure from the workload, using the workload credential. Reproducing from your own session with your own token proves nothing, because your token has different policies and a different lifetime. Enter the container or the host, use the credential file the process uses, and capture the full error text.
2Route on the status code. A Code: 503 response with * Vault is sealed means every caller is failing; stop here and switch to the recovery runbook. Only continue if the refusal is a 403 or a login that returns no token.
3Separate the two questions the 403 leaves open. A refusal at this point can mean the token is no longer valid, or that the token is valid but its policy does not grant the path. Do not guess. The next step settles it in one command.
4Attempt a fresh login through the same auth method, with the same credentials the workload holds. bao write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID" is the decisive test. If the login succeeds and the operation then works, the old token had expired or been revoked. If the login succeeds and the operation still fails, this is an authorisation problem and belongs to the authorisation runbook.
5If the login itself fails, check the SecretID constraints before anything else. A role may set secret_id_ttl, secret_id_num_uses and secret_id_bound_cidrs. An exhausted use count and a SecretID that has aged out both look identical from the client. A CIDR bound to an egress range that changed during a network migration is the failure that survives every restart.
6Check the token lifetime arithmetic against the incident timeline. A role configured with token_ttl=20m and token_max_ttl=1h issues a token that must be renewed within twenty minutes and cannot live beyond one hour however often it is renewed. A workload that reads its secret once at start-up and caches it forever will fail on the first restart after any of those boundaries.
7Confirm the delivery path, not just the credential. Read how the RoleID and SecretID reach the process: a file, an environment variable, an agent, or an init container. A correct credential that the process cannot read is indistinguishable from a wrong credential unless you look.
8Re-deliver the SecretID through the intended pull path. OpenBao documents pull mode, where the platform generates the SecretID and the client fetches it, precisely so response wrapping can keep it confidential in transit. Do not paste a SecretID into a chat window to end an incident faster.
9Restart or reload only the consumer, never the secret manager. Restarting OpenBao to clear an authentication failure reseals a Shamir-sealed node and converts one broken workload into a total outage. The consumer is the component that holds the stale token.
10Watch the audit log while the workload reconnects. A successful login writes a record naming the role and the policies attached to the issued token. That record, not the application log, is the proof that authentication now works.
11Confirm the application is serving, not merely starting. Read the endpoint the application exposes to its own callers. A process that has obtained a token and then failed on the next dependency is still an incident.
12Record the values you changed and where they came from. If a role configuration was corrected, state which version-controlled file is now authoritative, so the next deployment does not quietly restore the broken value.
13Close by revoking anything issued out of band. Any credential created to shorten the incident is now a standing grant. Revoke it in the same session that created it, and record the accessor in the incident notes.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The workload reads its own secret using its own credential, run from inside the workload, and the command exits 0.
✓The audit log contains a login record for the expected role, timestamped after the fix, listing the policies the workload is supposed to hold and no others.
✓The application health endpoint reports ready from an external caller that has never held an OpenBao token.
✓The database or downstream dependency shows a connection from the application, proving the secret was not only read but used.
✓bao status on the node still reports Sealed false, confirming nothing in the procedure resealed it.
✓bao audit list -detailed still lists the audit device, confirming the audit trail was not disabled to make the errors go away.
✓A second workload on the same auth mount, chosen because it was never touched, still authenticates successfully.
✓The role configuration in version control matches the running configuration returned by bao read auth/approle/role/app-role.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a role configuration was edited and the failure changed shape, restore the recorded original values with a single write and re-test before editing anything else.
↶If a SecretID was issued during the incident and the workload did not need it, revoke the tokens it produced rather than leaving them to expire.
↶If an emergency token was minted despite the stop line, revoke it immediately and treat its existence as a reportable event, not as a footnote.
↶bao lease revoke -prefix auth/approle/ revokes every lease created through that mount. It ends the incident and every other workload authenticating through the mount at the same time, so it is a deliberate choice made with the service owner, never a reflex.
↶If the consumer was reconfigured, redeploy the previous manifest from version control rather than editing the running container, so the rollback survives the next scheduler decision.
↶If nothing helps and the failure is now estate-wide, stop rolling back and escalate: a spreading authentication failure is a service incident, not a credential incident.

6 · Escalation

When the runbook isn't enough, contact:

· The refusal is a 503 with * Vault is sealed, or the node reports Initialized false when you know it was initialised: this is a service recovery, and the platform on-call owns it.
· Requests hang instead of failing: suspect a blocked audit destination, because auditing is fail-closed. Escalate to whoever owns the log sink and the disk it writes to.
· A login succeeds but returns policies the workload should not have: stop and page the security owner. An over-broad token is a finding whether or not it fixed the symptom.
· The SecretID delivery path cannot be reconstructed from the deployment manifests: escalate to the platform team, because an unowned credential path will fail again and nobody will know why.
· More than one auth mount is failing at once: treat it as a service incident, page the platform on-call, and hand over the timeline you have already written.
· The incident has run past the agreed time box and the only remaining option is a root token: that decision belongs to the security owner, in writing, with an expiry.

An application that cannot read its secrets produces one alert and three different incidents. The token it was carrying may have expired. The authentication method that issues tokens may have stopped issuing them. Or the secret manager itself may be sealed, in which case nothing is authenticating and the application is merely the first thing to notice.

The distinction this page exists to protect is the one between authentication and authorisation. Authentication answers “which identity is calling”. Authorisation answers “may that identity do this”. They fail at different layers, they are fixed by different people, and the single most expensive mistake in this area is widening a policy to fix an expired token. Once that has been done, the symptom disappears and the finding stays.

Work the status code first, the clock second, and the credential third. Every step below is either read-only or reversible until the last two, and the two irreversible ones are named where they occur.

When this runbook applies, and when it does not

It applies when a workload that previously authenticated to OpenBao now cannot, and the failure is visible as a refusal from OpenBao rather than as a network error.

It does not apply when:

The refusal is a 503 naming the seal state. Every caller is failing, not just yours. Running credential diagnostics against a sealed node wastes the first twenty minutes of an outage, and re-delivering credentials will not help because nothing can decrypt them.
The login succeeds and the subsequent read is refused. That is an authorisation failure. The path, the capability or the KV version prefix is wrong, and the authorisation runbook has the evidence you need.
The workload has never authenticated successfully. A first deployment that has never worked is a configuration task with a design review attached, not an incident. Treat it as one.
The failure is a TLS or DNS error against the listener. The secret manager is not refusing you; you are not reaching it. Diagnose the network path before anything else.

Blast radius

Action	Reversible?	What it costs if wrong
Reading `bao status` and the role configuration	Yes	Nothing. These are reads.
Attempting a login with the workload credential	Yes	Consumes one use if the role sets a use count. Note the count first.
Re-delivering a SecretID	Yes	A leaked SecretID is a standing grant until revoked. Use the wrapped pull path.
Restarting the consumer	Yes	A brief service gap, bounded by the readiness probe.
Revoking one lease	Only if you reissue	The workload holding it loses access immediately.
`bao lease revoke -prefix auth/approle/`	No	Every workload authenticating through that mount loses its token at once.
Minting a root token	No	The grant can be revoked; the fact that it existed cannot be undone.

Step 1 - Read the whole refusal, not the summary

The first artefact is the error text exactly as the workload sees it. A sealed service and a refused caller are different words in the same shaped message, and paraphrasing loses the difference.

Read-only / Safeon bao-1, reproducing the failure the workload reported

$ bao kv get -mount=kv app/config

Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/kv/app/config
Code: 503. Errors:

* Vault is sealed

Illustrative output

That is a service outage wearing an application costume. Note that the OpenBao command line still prints the word “Vault” in several of its strings; this is a fork artefact, and quoting it accurately matters more than tidying it. A Code: 503 here ends this runbook and starts the recovery one.

Step 2 - Prove the node is up and unsealed from the failing host

Run this from inside the same network namespace as the workload, not from your laptop. A node that answers you and refuses the application is a network answer, not a credential answer.

Read-only / Safethe healthy shape: unsealed, initialised, answering

$ bao status

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         2.6.2
Commit Date     2026-08-18T15:48:19Z
Storage Type    inmem
HA Enabled      false

Illustrative output

Read Initialized and Sealed as a pair. The transcripts behind this course recorded exit status 0 for an unsealed node and exit status 2 for both the sealed state and the uninitialised state, so scripting on the exit code alone conflates two very different situations.

Step 3 - Do the arithmetic before you blame the auth method

Token lifetimes are configuration, and configuration is arithmetic. A role carries both a renewal interval and a hard ceiling.

# Read the role, do not write it. Record what you find.
bao read auth/approle/role/app-role

The role used throughout this course was created with token_ttl=20m and token_max_ttl=1h. That means a token must be renewed inside twenty minutes, and no amount of renewal keeps it alive past one hour from issue. A workload that reads its secret once at start-up and never renews will therefore work perfectly until its first restart after the ceiling, which is why these failures so often arrive during an unrelated deployment.

The login response states the lifetime explicitly, so there is no need to infer it:

"auth": {
    "client_token": "s.<REDACTED>",
    "accessor": "n5GSr33ga2aoj00X03Z3y4Qh",
    "policies": ["app-read", "default"],
    "token_policies": ["app-read", "default"],
    "metadata": {"role_name": "app-role"},
    "orphan": true,
    "lease_duration": 1200,
    "renewable": true
}

lease_duration of 1200 seconds is the twenty minutes the role asked for. renewable being true is a capability, not a promise: something has to do the renewing, and if nothing does, the token dies on schedule.

This is the decisive test, and it is one command. It distinguishes an expired token from a broken auth method without changing any policy.

# Run inside the workload, using the credential the process itself reads.
ROLE_ID=$(cat /run/secrets/role_id)
SECRET_ID=$(cat /run/secrets/secret_id)
bao write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"

Three outcomes, three different runbooks. A login that succeeds and is followed by a working read means the old token had expired: fix the renewal, not the policy. A login that succeeds and is followed by the same refusal means the identity is fine and the grant is not, which is an authorisation problem. A login that fails means the credential or the auth method is the fault, and step 5 is next.

Step 5 - Check the SecretID constraints before re-issuing anything

An AppRole may constrain its SecretID by lifetime, by number of uses, and by source network. All three refuse a login that looks perfectly valid from the client side.

bao read auth/approle/role/app-role

secret_id_ttl. The SecretID aged out. Common after a long-lived pod stayed up across a credential rotation it never noticed.
secret_id_num_uses. The use count is exhausted. A crash loop burns a five-use SecretID in under a minute, which is why the failure often appears immediately after an unrelated restart storm.
secret_id_bound_cidrs. The source address no longer matches. This is the one that survives every restart and every redeploy, because nothing about the workload changed. The network did.

Step 6 - Re-deliver the credential through the intended path

OpenBao recommends pull mode, in which the platform generates the SecretID and the client fetches it, specifically so that response wrapping keeps the value confidential from the systems that carry it. Push mode, where some other system already holds the full credential pair, exists for legacy compatibility.

# Re-run the platform job that delivers the credential.
# It writes the wrapped SecretID to the path the workload reads.
systemctl start secret-delivery.service
systemctl status secret-delivery.service --no-pager

Then restart the consumer, and only the consumer. Restarting OpenBao to clear an application authentication failure reseals a Shamir-sealed node and turns one broken workload into an estate-wide outage.

Step 7 - Watch the audit trail as the workload reconnects

The application log says the application is happy. The audit log says OpenBao issued a token to the identity you expected, carrying the policies you expected. Only the second is independent evidence.

# On the active node, follow the audit device destination.
tail -n 20 /openbao/audit/audit.log

Look for a login record naming the role, and check the policy list on the issued token against what that workload is supposed to hold. A login that succeeds with more policies than expected is a finding, even though it has made the alert stop.

Common pitfalls

Symptom	Cause	Action
Works from your shell, fails in the pod	You are testing with your own token	Re-test from inside the workload with its own credential
Fails exactly one hour after every deploy	`token_max_ttl` ceiling reached, no renewal	Add renewal, or shorten the read-cache lifetime
Fails after a network migration only	`secret_id_bound_cidrs` no longer matches	Correct the binding in version control, do not delete it
Requests hang instead of failing	Audit destination blocked; auditing is fail-closed	Free the audit sink, then re-test
Login succeeds, read still refused	Authorisation, not authentication	Switch to the authorisation runbook
Every workload fails at once	Sealed node, or the auth mount is gone	Escalate as a service incident

Verification

Verification is complete when the workload reads its own secret using its own credential from inside its own container, and when an independent channel agrees. That independent channel is the audit record of the login, which names the role and the policies attached to the issued token, plus the downstream dependency showing a connection from the application. Both are observations of something other than the file you just wrote. Confirm in the same pass that Sealed false still holds, that the audit device is still listed, and that an untouched workload on the same mount continues to authenticate. Finally, compare the running role configuration against the version-controlled definition, because a fix that lives only in memory will be undone by the next deployment and the incident will repeat.

Rollback

Rollback is per-change and in reverse order. If a role configuration was edited, write back the values recorded in the pre-checks and re-test before touching anything else. If a SecretID was delivered and turned out to be unnecessary, revoke the tokens it produced rather than waiting for them to expire. If an emergency token was minted, revoke it in the same session and record that it existed. The prefix revocation bao lease revoke -prefix auth/approle/ is available and will end the incident, but it revokes every lease created through that mount, so it is a service owner decision rather than an operator reflex. If the consumer was reconfigured by hand, redeploy the previous manifest from version control so the rollback survives the next scheduling event.