Skip to main content
RunBook Academy

← All break/fix scenarios in Secrets, PKI & Certificates

intermediatesecret-policy~40 min

A release fails on a secret path that the policy appears to grant

Reported symptoms

  • ●At 14:06 UTC the reporting-worker release rolled out and every worker aborted during its startup discovery step, before processing a single job.
  • ●A rollback to the previous release restored service within four minutes, which pointed the investigation at the application code rather than at the secret manager.
  • ●An operator checked the policy by reading the known secret path by hand, it returned the record, and the channel recorded the policy as verified and moved on.
  • ●The release notes mentioned a change of mount syntax in the client configuration, so the mount was blamed for the next twenty minutes.
  • ●Three other services using the same token role were entirely unaffected, because none of them enumerate anything.
  • ●The secret manager was unsealed, healthy and answering thousands of successful reads a minute for other callers throughout.

Evidence

  • · The worker log shows the job aborting at the discovery step with a 403 and the message permission denied, before any secret value has been read.
  • · Listing the secret prefix by hand reproduces the failure, and the error names an API path under the metadata prefix rather than the data prefix.
  • · Reading the single known secret by name with the same token succeeds and returns the record, which is why the policy was wrongly recorded as verified.
  • · Reading a second secret added by the same release fails with 403, and that error names an API path under the data prefix that the policy does not contain.
  • · The policy file grants the read capability on exactly one data path and contains no other stanza at all.
  • · The audit record for the denied read carries a populated policies array for the caller and a policy results object stating that the request was not allowed.
  • · The same audit record shows the operation as read and the path as the data prefix path, which confirms the request reached policy evaluation rather than failing authentication.
  • · The release diff adds a discovery step that enumerates the prefix and a second secret read, and changes nothing about authentication.
Diagnosis and resolutionclick to reveal

Root cause

Two independent gaps in one policy, both of them the same mistake made twice. The policy grants the read capability on a single leaf data path and nothing else. Policies are deny by default, so every path the policy does not name is refused silently, without any indication that a stanza is missing rather than wrong. The release added two new operations. The first is enumeration of the prefix, and this is the subtle one. The key value store version two splits its API into a data prefix and a metadata prefix, and the list capability is only ever evaluated against the metadata prefix. A policy that grants read on the data path therefore cannot grant list on anything, and adding the list capability to the data stanza would not have helped either, because the client never requests a list against that path. The second operation is a read of a sibling secret under the same prefix, which is denied for the ordinary reason that the policy names one leaf and not a prefix. The enumeration failure is what made this an outage, because discovery runs before any work and its failure aborts the whole worker. The sibling read would have been the next failure four lines later. The command line interface concealed both, because it accepts a path without the data segment and inserts it, so operators compare a path they typed against a policy written in a different vocabulary.

Remediation

Do not start by widening the policy. Start by establishing exactly which API paths the application calls, and take that list from the audit device rather than from the source code, because the client library inserts path segments that no line of application code contains. With the real list in hand, add the minimum stanzas that cover it: the list capability on the metadata path for the prefix being enumerated, and the read capability on the data paths actually read. Prefer naming paths explicitly. Where a prefix genuinely is required, remember that the glob is only meaningful as the final character of a path and matches by prefix, so a data stanza and a metadata stanza are still two separate stanzas. Do not grant the delete capability on the metadata prefix to an application. Delete on a data path soft-deletes the latest version; delete on the corresponding metadata path destroys every version and the metadata with it, which is an enormously larger grant that looks almost identical in a diff. Do not attach a broad administrative policy or a root token as a temporary measure, because the temporary measure will outlive the incident and will not appear in any future review as an exception. Before deploying, check that no other policy attached to the same role contains an explicit deny for these paths: deny always wins, regardless of any other capability, so a grant that looks correct can still be overridden. Re-authenticate the workers after the change so that the policy set in force is unambiguous, then confirm from the audit record rather than from the application.

Verification

Verify from the audit device, because it is the only place that records what the server decided rather than what the client experienced. Re-run the discovery step as the application, from an application host, using the application token, and confirm the audit record for the metadata path no longer carries the permission denied error and is no longer recorded as not allowed. Repeat for each data path the release reads. Running the same commands as an operator with a different token proves nothing, and that substitution is exactly how this policy came to be recorded as verified at 14:11. Then run the negative half, which is the half teams skip: pick a path deliberately outside the intended grant, request it with the application token, and confirm it is still refused and still recorded as not allowed. A policy change that fixes the failure and also silently opens a prefix will pass every positive test ever written. Confirm the worker completes a full job cycle end to end rather than merely getting past discovery, since the release added two operations and only one of them aborts startup. Finally, confirm the audit device is actually recording, as a policy verified against an audit stream that stopped days ago is a verification of nothing.

Prevention

Write policies against API paths and keep the translation visible. Maintain a short table in the policy repository mapping each command line form to the API path it becomes, so that a reviewer comparing a policy against a ticket is comparing the same vocabulary. Test policies in continuous integration with both halves: for every role, assert the list of paths that must be permitted and the list that must be refused, and fail the build on either. A policy suite with only positive cases cannot detect over-granting, which is the failure mode that a rushed incident fix introduces. Make a policy review a required step in any release that adds, moves or enumerates a secret path, and attach the audit extract showing the paths the release actually calls. Alert on denials directly: any request recorded as not allowed for a production role warns, and five within five minutes pages, which would have surfaced this at 14:06 rather than at 14:31. Never grant delete on a metadata prefix to an application, and treat any such grant as a finding. Review each role quarterly against what it actually called in the audit stream over the preceding ninety days, and remove grants with no matching requests. Finally, confirm the audit device is configured in the server configuration file, because in this release it cannot be enabled through the API at all.

Reported symptoms

reporting-worker reads its configuration from a key value version two mount on bao.example.com and has done so for two years. Release 6.2 added regional reporting, which meant the worker had to stop reading one hardcoded secret and start discovering which regional secrets exist before reading them. The policy attached to its role had not been touched in eighteen months.

The incident channel filled up faster than the facts did:

  • 14:06, release 6.2 rolls out. Every worker aborts during startup. The queue depth begins climbing immediately.
  • 14:10, the release is rolled back. Service recovers in four minutes, which everybody reads as confirmation that the fault is in the new code.
  • 14:11, an operator reads the known secret path by hand to check the policy. It returns the record. The channel notes that the policy is verified and moves on.
  • 14:14, somebody spots that the release notes mention a change to the mount syntax in the client configuration, and the next twenty minutes go into the mount.
  • 14:31, an engineer opens the audit device output and searches for the worker token accessor.

The policy was never verified at 14:11. An operator read a path that the policy grants, using a different token, in a different form, and concluded something about two paths that the policy does not grant. That one minute of work cost the incident twenty-five.

Evidence provided

# reporting-worker instance 2, startup log
2026-08-26T14:06:41.118Z level=info  component=secrets step=discovery
    prefix=app action=list
2026-08-26T14:06:41.164Z level=error component=secrets step=discovery
    status=403 body="* permission denied"
2026-08-26T14:06:41.164Z level=fatal component=worker
    cause="discovery failed, refusing to start with an unknown secret set"
Read-only / SafeOperator host, 14:33 UTC, using the worker token: the enumeration the worker performs at startup
$ bao kv list -mount=kv app
Error listing kv/metadata/app: Error making API request.

URL: GET https://bao.example.com:8200/v1/kv/metadata/app?list=true
Code: 403. Errors:

* permission denied

Illustrative output

Read-only / SafeOperator host, 14:11 UTC, using the worker token: the read that convinced the channel the policy was fine
$ bao kv get -mount=kv app/config
Read-only / SafeOperator host, 14:35 UTC, using the worker token: the second secret the release reads
$ bao kv get -mount=kv app/other
Error reading kv/data/app/other: Error making API request.

URL: GET https://bao.example.com:8200/v1/kv/data/app/other
Code: 403. Errors:

* permission denied

Illustrative output

Three requests, one token, three different outcomes. Two of them are refused and the two refusals name different API paths. Neither of those paths resembles what was typed.

# /etc/openbao/policies/app-read.hcl, unchanged since February 2025
path "kv/data/app/config" {
  capabilities = ["read"]
}

That is the entire policy. One stanza, one capability, one path.

{"time":"2026-08-26T14:35:02.555636512Z","type":"response",
 "auth":{"client_token":"hmac-sha256:da33377eba1c...","accessor":"hmac-sha256:5eb6ce9e...",
         "policies":["app-read","default"],"policy_results":{"allowed":false}},
 "request":{"operation":"read","mount_point":"kv/","mount_type":"kv",
            "path":"kv/data/app/other","remote_address":"198.51.100.62"},
 "response":{"data":{"error":"hmac-sha256:b0a0f532..."}},
 "error":"1 error occurred:\n\t* permission denied\n\n"}

This record is the decisive piece of evidence and it is worth reading field by field. The auth object is populated, and it names the policies the caller carries, which means the token authenticated successfully and the server knew who it was. policy_results says allowed is false, which means the request reached policy evaluation and lost there. The path is the API path, not the path anybody typed. A token that had expired would not have produced this record at all, because there would have been no policy set to evaluate.

# release 6.2 diff, secrets client (abridged)
+  regions := secrets.List("app")            // new: enumerate before reading
   cfg    := secrets.Read("app/config")
+  other  := secrets.Read("app/other")       // new: shared regional settings

Work the evidence before reading on

The trap in this incident is that one of the three requests succeeds, and the successful one is the request an operator will naturally reach for when asked whether the policy is correct.

  1. Write down the three API paths the errors and the successful read actually reach. Now compare them, character by character, against the single path in the policy file. How many of the three appear there?
  2. The command that succeeded named kv/app/config and the policy names kv/data/app/config. Something inserted a path segment. What, and what else might it be inserting elsewhere?
  3. The failing enumeration names a path under metadata, which the policy does not mention at any capability. If you added list to the existing stanza, which of the three requests would change?
  4. The audit record contains a populated policies array and a policy result of not allowed. What does each of those two facts independently rule out?

Before continuing: name the two separate grants this policy is missing, and say why granting one of them on the path the application reads from would achieve nothing.

Root cause

The policy grants one leaf, and policies are deny by default

The stanza names kv/data/app/config and grants read. Every other path in the system is refused, and refused silently: there is no warning at write time that a policy is narrower than the workload, and no diagnostic at request time distinguishing a path that was denied from a path that was never mentioned. Release 6.2 added a read of kv/data/app/other, a sibling of the one granted path, and it was refused for the ordinary reason that the policy does not contain it. This is the easy half of the incident and the half that was never reached, because the worker aborted before it got that far.

The list capability lives on a different prefix entirely

Version two of the key value engine splits its API. Secret values are read and written under a data prefix; version history, deletion state and enumeration live under a metadata prefix. Enumeration is only ever evaluated against the metadata path, so the worker enumerating app produces a request for kv/metadata/app. Granting list on kv/data/app/config, or even on kv/data/app/*, would not have made the slightest difference, because no client ever asks to list that path. The two prefixes require two stanzas, and no amount of editing one of them produces the other.

The command line interface hides the split from the person reviewing the policy

Operators type kv/app/config and the client inserts data. Policies are written against the API, so they say kv/data/app/config. The two strings look similar enough to pass review and different enough that nobody can compare them mechanically, and the mismatch only becomes visible in an error message or an audit record, both of which print the real path. That is why the 14:11 check appeared to verify a policy it had not tested, and why the incident spent twenty minutes on the mount configuration named in the release notes.

Resolution

  1. Take the list of API paths from the audit device, not from the source code. The client library inserts path segments that appear nowhere in the application, which is the entire reason this policy passed review, and reading the code will reproduce the original mistake exactly.
  2. Add the enumeration grant as its own stanza on the metadata path for the prefix being listed, with the list capability and nothing else. It cannot be folded into the data stanza, because the two prefixes are different paths and the client requests only one of them for each operation.
  3. Extend the read grants to cover the data paths the release actually reads. Name them explicitly where the set is small and known. If a prefix is genuinely required, remember the glob is meaningful only as the final character of a path and matches by prefix, so kv/data/app/* still leaves kv/metadata/app untouched.
  4. Do not grant delete on any metadata path to an application. Delete on a data path soft-deletes the latest version; delete on the matching metadata path destroys every version and the metadata with it. In a diff those two lines are almost identical, so state the rule rather than relying on review to catch it.
  5. Check every other policy attached to this role for an explicit deny covering these paths. An explicit deny always takes precedence over any other capability, including administrative ones, so a correct grant can still produce an identical 403 and an identical audit record.
  6. Deploy the policy, then re-authenticate the workers so that the policy set in force on their tokens is unambiguous rather than inferred. Record the new accessors so the verification step has something specific to search the audit stream for.
  7. Roll release 6.2 forward again rather than patching the rolled-back release. The application code was never wrong, and carrying a divergent build forward guarantees that the next person to deploy 6.2 reproduces the incident.
  8. Write the two new stanzas into the policy repository with a comment beside each explaining which application operation requires it. A stanza with no stated purpose is a stanza that survives every future least-privilege review by default.

Verification

  1. Re-run the discovery step from an application host, as the application, with the application token. Verifying a policy with an operator token in an operator shell is what produced the false clearance at 14:11, and repeating it now would produce the same worthless result.
  2. Read the audit record for the metadata path and confirm the denial is gone: no permission denied error, and no result recording the request as not allowed. This is the server stating what it decided, which is independent of whatever the client reports.
  3. Repeat that audit check for each data path the release reads, including the sibling secret that the worker never reached during the incident because discovery aborted first.
  4. Run the negative case. Request a path deliberately outside the intended grant with the application token and confirm it is still refused and still recorded as not allowed. A change that fixes the failure and quietly opens a prefix passes every positive test that will ever be written against it.
  5. Let a worker complete a full job cycle rather than stopping at a successful startup. Discovery and the sibling read are two separate operations added by the same release, and only the first one aborts the process.
  6. Confirm the audit device is currently recording by locating a request you made yourself within the last minute. A policy verified against a stream that stopped writing days ago has been verified against nothing.
  7. Compare the deployed policy against the repository copy after the incident closes, to confirm no emergency edit was left in place on the server that the repository does not know about.

Prevention

  • Write policies against API paths and publish the translation. Keep a table in the policy repository mapping each command line form to the path it becomes. Reviewers who compare a ticket against a policy are otherwise comparing two different vocabularies.
  • Test both halves in continuous integration. For every role, assert the paths that must be permitted and the paths that must be refused. A suite with only positive cases cannot detect the over-granting that a rushed incident fix introduces.
  • Gate secret-path changes on a policy review. Any release that adds, moves or enumerates a secret path attaches the audit extract showing the paths it actually calls, and the policy diff is reviewed against that extract rather than against the description.
  • Alert on denials. One request recorded as not allowed for a production role warns, five within five minutes pages. The evidence that resolved this incident existed at 14:06 and was first read at 14:31.
  • Forbid delete on metadata prefixes for applications. Treat any such grant as a finding with an owner, not a style preference. The blast radius is every version of every secret under the prefix.
  • Review roles against what they actually called. Once a quarter, compare each role grant list against ninety days of audit history and remove grants with no matching requests. Policies grow when incidents are fixed and never shrink unless somebody schedules it.
  • Verify as the caller, never as an operator. Record in the runbook that a policy is only tested when the request is made from the workload host, with the workload token, in the form the client library produces. The check at 14:11 met none of those three conditions, and it cost the incident twenty-five minutes and a rollback that was never necessary.