← All runbooks in Secrets, PKI & Certificates

medium risksecurity relevant~40 min

Runbook: Troubleshoot Secret Manager Authorisation Failure

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-lab-20-write-least-privilege-secret-policy
secrets-pki-rb-17-troubleshoot-secret-manager-authentication
A login through the workload auth method that already succeeds, so the identity is not in question
Read access to the audit device destination on the active node, which is where the decisive evidence lives
The version-controlled policy document that is supposed to govern this mount, and the authority to change it there
Knowledge of which KV version the mount runs, because the policy path differs between version 1 and version 2
A change record, because every outcome of this runbook is a change to who may read what

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Prove the identity first. Run a fresh login with the workload credential. If the login itself fails, this is the wrong runbook and the authentication procedure owns the incident. Authorisation problems always have a valid token behind them.
· Capture the refusal verbatim, including the URL line. The command line prints the API path it actually called. Error listing kv/metadata/app after a bao kv list kv/app is the single most useful line in this whole procedure, and it is the line people paraphrase away.
· Determine the KV version of the mount. A version 2 mount serves data under a data/ prefix and metadata under a metadata/ prefix. A policy written for version 1 paths parses cleanly, uploads cleanly, and grants nothing.
· List every policy attached to the token, not just the one you expect. bao read auth/approle/role/app-role shows the token policies for the role. The built-in default policy is always attached as well and cannot be removed.
· Confirm an audit device is enabled before you rely on it. bao audit list -detailed must show a device with a destination you can read. Without it you are guessing at which path was refused.
· Record the current policy exactly as the server holds it. bao policy read app-read prints the running document. Save it; it is your rollback.
· Decide in advance what a correct answer looks like. Write down the single path and the single capability the workload legitimately needs before you look at the policy, so that you are not tempted to match the policy to the symptom.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Reproduce the refusal from the workload and keep the whole message. The URL line in the error names the API path OpenBao evaluated, which is frequently not the path the operator typed. This one line resolves most authorisation incidents on its own.
2Translate the command-line path into the API path. A version 2 KV mount rewrites kv/app/config into kv/data/app/config for reads and writes, and into kv/metadata/app for a list. Use the -mount form, bao kv get -mount=kv app/config, so the display path and the API path stop being confused with one another.
3Find the denial in the audit log and read the policy result. A denied response is recorded with the requesting policies and with "policy_results":{"allowed":false}, alongside the exact path that was evaluated. This is the decisive evidence, because it is the server stating what it evaluated rather than the client guessing.
4Confirm the secret value did not reach the log. Tokens are stored HMACed and secret values are not recorded in clear. Verify with a targeted grep for a known value before you paste an audit excerpt into a ticket.
5Compare the evaluated path against every path stanza on the policy. A stanza matches by prefix only when the glob is the final character. + matches a single path segment and nothing across segments. A stanza that looks close enough is not a match.
6Compare the requested operation against the capability list. Read is GET, list is LIST, create and update are both POST or PUT, patch is PATCH, delete is DELETE. A policy that grants read on a path still refuses a write to that same path, which is correct behaviour and a frequent surprise.
7Look for an explicit deny anywhere on the token, not only in the policy you are reading. A deny capability takes precedence over every other grant including sudo, and it cannot be overridden by another attached policy. An unset capability is silent; an explicit deny is absolute.
8Decide whether the policy is wrong or the request is wrong. If the workload is asking for a path it has no business reading, the correct fix is in the application, not in the policy. Write that conclusion down before anyone reaches for the policy file.
9Draft the narrowest correction that satisfies the legitimate request. Grant list on the metadata prefix if listing is genuinely required. Do not grant delete on a metadata prefix to make a soft delete work, because that capability destroys every version and the metadata with it.
10Change the version-controlled policy document first, then apply it. bao policy write app-read /path/to/app-read.hcl uploads the file. A policy edited only on the server will be overwritten by the next deployment and the incident will recur with no trace of why.
11Re-test as the workload, using the workload credential, for both the allowed and the previously refused operation. A fix that is only verified for the failing case tells you nothing about what else it opened.
12Prove the grant did not widen. Re-run a request that should still be refused and confirm it is still refused, with a fresh audit record showing allowed:false. A policy change with no remaining denials is almost always too broad.
13Record the before and after policy documents in the change record. The diff is the artefact an auditor will ask for, and it is trivial to produce now and impossible to reconstruct later.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The workload performs the legitimate operation from inside its own container using its own token, and the command exits 0.
✓A fresh audit record for that request shows the evaluated path and a policy result that is no longer a denial.
✓A deliberately chosen neighbouring path that the workload must not reach is still refused with Code: 403, and the refusal has its own audit record.
✓The running policy returned by bao policy read app-read is byte-for-byte the document held in version control.
✓A grep of the audit destination for a known secret value returns no matches, confirming values are still not logged in clear.
✓A second workload on the same mount, chosen because its policy was not touched, still behaves exactly as it did before the change.
✓The application health endpoint reports ready from an external caller, proving the secret was used and not merely fetched.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Re-upload the policy document captured in the pre-checks with bao policy write, then re-test both the allowed and the refused case to confirm the previous behaviour is back.
↶If the correction was committed to version control, revert the commit as well; a server rolled back without a matching revert will drift back on the next deployment.
↶If the change was applied to the wrong policy name, restore that policy from version control before touching the intended one, so two mistakes do not overlap.
↶If a capability was granted and then removed, revoke the tokens issued while it was in force rather than assuming nothing used it.
↶If the workload was reconfigured to request a different path, redeploy the previous manifest so the running state matches the policy that is now in force.
↶If rollback restores the failure and the pressure is to widen the policy instead, stop and escalate. A deliberately unresolved incident is better than an undocumented standing grant.

6 · Escalation

When the runbook isn't enough, contact:

· The workload is asking for a path that belongs to another team or another tenant: stop, do not grant it, and escalate to the data owner. This is a design question wearing an incident costume.
· The refusal has no matching audit record at all: escalate to the platform team, because either the audit device is not capturing responses or the request never reached this node.
· An explicit deny is present and nobody can say who added it or why: escalate to the security owner before removing it, since a deny is usually the residue of a previous incident.
· The correct fix requires a broader grant than the current standard allows: that is a policy exception and belongs to the security owner in writing, with an expiry date.
· Multiple workloads on the mount begin failing after the change: revert immediately and treat it as a service incident rather than continuing to tune the policy live.
· The mount is a version 1 KV store that nobody intended to still be running: escalate as a migration item, because the policy path shape will keep producing this incident.

The login worked. The token is valid. The read is refused anyway. That is an authorisation failure, and it is a different animal from the expired credential in the previous runbook, because nothing here is broken. The system is doing exactly what it was told; the disagreement is about what it was told.

Almost every authorisation incident against a version 2 KV mount comes down to one thing: the path in the policy is not the path the server evaluates. The command line lets an operator type kv/app/config, and the server evaluates kv/data/app/config for a read and kv/metadata/app for a list. A policy written against the display path parses, uploads, reports success, and grants nothing at all.

The hard discipline in this procedure is that its output is a change to who may read what. There is always a shortcut available, and the shortcut is always broader than the fix. Decide what a correct grant looks like before you open the policy file.

When this runbook applies, and when it does not

It applies when an identity authenticates successfully and a subsequent operation is refused with a permission denied against a specific path.

It does not apply when:

The login itself fails. No token, no policy evaluation, no authorisation question. Work the authentication runbook and come back if the refusal survives a fresh token.
Everything is refused, including operations that never worked. A workload that has never had a working grant is a configuration task with a design review attached, and rushing it under incident pressure is how over-broad policies are born.
The refusal is a 503 naming the seal state. The node is sealed. No policy evaluation is happening at all.
The request is legitimate but belongs to another tenant. The answer is not a wider policy. It is a conversation with the data owner, and possibly a separate mount.

Blast radius

Action	Reversible?	What it costs if wrong
Reading the policy and the audit records	Yes	Nothing. These are reads.
Re-testing a refused operation	Yes	One more denial record in the audit log.
Uploading a corrected policy	Only if you kept a copy	The previous document is replaced in place. Capture it first.
Granting `list` on a metadata prefix	Yes	Reveals the names of every secret under that prefix.
Granting `delete` on a metadata prefix	No	Destroys every version and the metadata, not the latest version.
Removing an explicit deny	No	A deny is usually the residue of a previous incident. Removing it silently repeats that incident.

Step 1 - Read the URL line, because it names the real path

The client prints the API path it called. On a version 2 mount, that path is not the one that was typed, and the difference is the whole diagnosis.

Read-only / Safescoped token, listing a prefix its policy never granted

$ bao kv list -mount=kv app

Error listing kv/metadata/app: Error making API request.

URL: GET http://127.0.0.1:8200/v1/kv/metadata/app?list=true
Code: 403. Errors:

* 1 error occurred:
* permission denied

Illustrative output

The operator asked to list app. The server evaluated kv/metadata/app. A policy that grants read on kv/data/app/config has nothing whatever to say about that path, so the refusal is correct. Reading data and listing names are separate grants on separate prefixes, by design.

Step 2 - Confirm the capability, not just the path

The same policy refuses a write to the exact path it permits reading. This surprises people, and it is the system working properly.

Read-only / Safethe same token, writing to the one path it may read

$ bao kv put -mount=kv app/config x=y

Error writing data to kv/data/app/config: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/kv/data/app/config
Code: 403. Errors:

* 1 error occurred:
* permission denied

Illustrative output

Read is GET, list is LIST, create and update are both POST or PUT, patch is PATCH, delete is DELETE. OpenBao also offers scan, which recursively lists values beneath a path and has no equivalent in Vault Community Edition, so capability lists copied from Vault documentation omit it.

Step 3 - Take the decisive evidence from the audit record

The client says it was refused. The audit record says what the server evaluated, which policies were on the token, and what the outcome was. That is the difference between a theory and a diagnosis.

{
  "time": "2026-08-26T21:26:20.555636512Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:da33377eba1c6df88a7614495b44539cc6e08718a1d11077db8640f07718ae91",
    "display_name": "token",
    "policies": ["app-read", "default"],
    "token_policies": ["app-read", "default"],
    "policy_results": {"allowed": false},
    "token_type": "service",
    "token_ttl": 1800
  },
  "request": {
    "operation": "read",
    "mount_point": "kv/",
    "mount_type": "kv",
    "path": "kv/data/app/other",
    "remote_address": "127.0.0.1",
    "remote_port": 54616
  },
  "response": {
    "data": {"error": "hmac-sha256:b0a0f5321dbc96ac1c397860a52f0cea3a1afa827d535de367de97434ed30ce8"}
  },
  "error": "1 error occurred:\n\t* permission denied\n\n"
}

That record is reformatted for reading and trimmed to the fields that matter; no value in it has been altered. Four things are settled at once: the operation was a read, the evaluated path was kv/data/app/other, the token carried exactly app-read and the built-in default, and the policy result was not allowed. Notice also what is absent. The token is stored as an HMAC rather than in clear, and the secret value never appears: a grep of the audit destination for the known lab value returned zero matches.

Step 4 - Put the policy and the evaluated path side by side

# The running policy: one path, one capability, nothing else.
path "kv/data/app/config" {
  capabilities = ["read"]
}

Match it against the evaluated path character by character. A * glob is honoured only as the final character of a path and behaves as a prefix match, so kv/data/app/* covers kv/data/app/config and everything below it. A + matches exactly one path segment and never spans a separator, so kv/data/+/config covers kv/data/app/config but not kv/data/app/db/config. When several stanzas match, OpenBao documents an ordering in which earlier wildcards lose, a pattern ending in * loses to one that does not, more + segments loses, and a shorter path loses.

Step 5 - Hunt for an explicit deny before you add a grant

An unset capability grants nothing quietly. An explicit deny is absolute: it takes precedence over every other capability including sudo, and no other policy attached to the same token can override it. That asymmetry means a correct-looking grant can sit next to an invisible veto.

# Read every policy the token carries, including the built-in default.
bao policy read app-read
bao policy read default

If a deny is present and nobody can explain it, stop. A deny is usually the residue of a previous incident, and removing it without finding that incident is how the same outage happens twice.

Step 6 - Correct the document in version control, then upload it

# Edit the file in the repository, commit, then apply the committed file.
bao policy write app-read /etc/openbao/policies/app-read.hcl

A successful upload reports Success! Uploaded policy: app-read. A policy edited only on the server survives exactly as long as the next deployment, after which the incident recurs with no record of the correction. Make the repository the source of truth even when the incident is loud.

Step 7 - Prove the grant did not widen

Re-test the legitimate operation as the workload, then deliberately re-test one that must still fail.

# Must succeed.
bao kv get -mount=kv app/config
# Must still be refused, and must still produce an audit denial.
bao kv get -mount=kv app/other

A policy change that leaves no remaining denials is nearly always broader than intended. The second command is not a formality; it is the only evidence that least privilege survived the incident.

Common pitfalls

Symptom	Cause	Action
Policy looks right, grants nothing	Version 1 path shape on a version 2 mount	Insert the `data/` prefix and re-upload
Read works, list refused	`list` lives on the `metadata/` prefix	Grant `list` on the metadata path if listing is genuinely needed
Read works, write refused	`read` does not imply `update` or `create`	Add the specific capability, not a wildcard
Grant added, still refused	An explicit `deny` on another attached policy	Find the deny and the incident that produced it
Fix disappears next week	Policy edited on the server only	Apply from the version-controlled document
Requests hang rather than fail	Audit destination blocked; auditing is fail-closed	Free the audit sink first, then resume

Verification

Verification here has two halves, and skipping the second is how a fix becomes a finding. The first half is that the legitimate operation now succeeds when run from inside the workload with the workload token, and that a fresh audit record shows the evaluated path with a policy result that is no longer a denial. The second half is that a neighbouring path the workload must not reach is still refused, with its own new audit record carrying allowed:false. Confirm alongside these that the running policy matches version control byte for byte, that a grep of the audit destination for a known secret value still returns nothing, and that an untouched workload on the same mount behaves exactly as it did this morning. The application health endpoint, read from an external caller, closes the loop by showing the secret was used rather than merely fetched.

Rollback

Rollback is a single upload of the policy document captured in the pre-checks, followed by re-testing both the allowed and the refused case so that the previous behaviour is demonstrated rather than assumed. If the correction reached version control, revert that commit in the same motion, because a server rolled back without a matching revert drifts forward again at the next deployment. If the wrong policy name was written, restore that policy first so two mistakes do not overlap. Any capability that was in force even briefly should be treated as used: revoke the tokens issued during that window rather than assuming nothing took advantage of it. If rolling back restores the original failure and the room starts arguing for a wider grant instead, that is the moment to escalate rather than to type.