The architecture of a secret manager, before any product

A secret manager is not an encrypted database with an API in front of it. It is a small distributed system whose entire job is to turn a proven identity into a scoped, time-bounded credential, and to leave a durable record that it did so. Every serious product in this space implements the same six parts. Once you can name them, product documentation stops being a vocabulary lesson and becomes a list of configuration choices.

The six parts, and the question each one answers

Six components appear in every implementation, whether the product is OpenBao, a cloud vendor’s managed offering, or an in-house service somebody wrote in 2019 and nobody wants to touch.

• The client is the workload, not the human. It runs on a host, in a container, or as a scheduled job, and it has no hands to type a password. Almost every design mistake in this field starts by forgetting that.
• The authentication method converts something the client already possesses into a session. That something might be a platform-issued token, a client certificate, a signed instance document, or a pair of role credentials.
• The policy is evaluated against a path and an operation. It is not a role name attached to a person; it is a rule that says which API paths this session may read, write, list or delete.
• The secret engine is a plugin mounted at a path. Some engines store what you gave them. Others manufacture a fresh credential in a downstream system at the moment of the request.
• The credential is what the client receives. In a mature design it carries a lease: an issue time, a time to live, and a revocation handle.
• The audit device writes a structured record of the request and the response, including the ones that were refused.

The request path has two halves, and they are separate

Authentication and authorisation are two different operations separated in time, and conflating them is the single most common conceptual error operators bring to this subject.

flowchart LR
    C["Workload"] -->|"1 present platform credential"| A["Auth method"]
    A -->|"2 session token\nplus policy names"| C
    C -->|"3 request path\nplus operation"| P["Policy evaluation"]
    P -->|"4 permitted"| E["Secret engine"]
    E -->|"5 credential\nplus lease"| C
    P -.->|"denied"| D["403 permission denied"]
    A --> L["Audit device"]
    P --> L
    E --> L

The first half runs once. The workload presents whatever platform-issued proof it already holds, the authentication method verifies it against the issuing authority, and the workload receives a session token together with the list of policy names attached to it. The second half runs on every single request afterwards. The token is presented, the named policies are looked up and evaluated against the requested path, and only then does the secret engine get involved. Both halves emit audit records, and so does the refusal.

Written as API calls the shape becomes obvious, because each half is a different set of paths:

POST  /v1/auth/approle/login          -> a session token
GET   /v1/kv/data/app/config          -> the value at a path
GET   /v1/kv/metadata/app?list=true   -> the child keys

The first line runs once per session. The other two run whenever the application needs something, each carrying the token and each evaluated against policy on its own merits, as the differing prefixes hint.

That separation is what makes a secret manager operationally different from a configuration file. A configuration file is evaluated once, at process start, and its contents remain valid until somebody redeploys. A policy is evaluated at request time, so tightening it takes effect on the next call without touching a single workload.

Where a credential comes from changes everything downstream

Secret engines fall into two families, and the distinction is the most consequential one in the entire architecture.

A storing engine keeps what you put into it. You write a database password, the engine encrypts and persists it, and readers get back exactly the value you wrote. The engine has no opinion about whether that value is still correct, and it cannot change the downstream system, so rotation remains a human project with a change window attached.

A generating engine holds a privileged bootstrap credential for a downstream system and manufactures a new, unique credential per request. The value returned did not exist a second earlier. Because the engine created it, the engine can also destroy it, which is what makes revocation a single API call rather than an incident.

# Where does this host keep credentials today, before any
# secret manager exists? Run this before designing anything.

# 1. Files under /etc that look like they hold a credential.
sudo grep -rlIE '(password|secret|token|api[_-]?key)[[:space:]]*[=:]' /etc \
  2>/dev/null | head -20

# 2. Of the configuration files present, which are readable
#    by every account on the host?
sudo find /etc -type f -perm -o=r -name '*.conf' | head -20

# 3. Credentials handed to services through the environment.
systemctl show --all --property=Environment \
  | grep -iE 'password|token|secret' | head -20

Run that inventory on a production host before you introduce a manager. The output is the list of things that must eventually move, and the third command usually surprises people: an environment variable set in a unit file is readable by anyone who can run systemctl show, and it is inherited by every child process the service spawns.

The architecture predicts its own failure modes

Because the request path is known, the outage classes are known before deployment, and each one has a different owner.

• Authentication fails. The platform proof expired, was rotated, or the authentication method can no longer reach the issuing authority. The workload never gets a token and reports a login error, not a secrets error.
• Authorisation fails. The token is valid and the path is refused. The status code is 403 and the message is deliberately unhelpful, because a detailed refusal is an oracle for path discovery.
• The engine is unavailable. A generating engine cannot create a credential when the downstream database is down. This looks like a secrets outage and is a database outage.
• The manager is unavailable. Every request fails at once, across every workload, which makes the secret manager a tier-zero dependency the moment the first application adopts it.
• Audit cannot write. A manager that is required to record every request has to decide what to do when it cannot. The honest answer, and the one OpenBao takes, is to stop serving requests rather than serve them unrecorded.

The status code the client received is enough to place the failure in the right column before anyone opens a dashboard:

# Triage a secrets failure from the status the client saw.
STATUS=403

case "$STATUS" in
  403) echo "authorisation: compare the exact path against the policy" ;;
  503) echo "availability: the manager is unavailable or sealed" ;;
  400) echo "the endpoint refused the request itself; read the body" ;;
  *)   echo "collect the response body before changing anything" ;;
esac

Escalate on that classification rather than on the symptom the application reported. An application that logs only that it could not start its database pool has told you nothing about which of the five failures above it hit, and every one of them has a different owner.

Production discipline

1. Name the first credential before choosing a product. The authentication method you can actually operate is determined by what your platform already issues to workloads, not by which method has the best documentation.
2. Write policy against paths, never against people. A policy that mirrors your org chart will be wrong within a quarter. A policy that mirrors your application layout survives reorganisations.
3. Decide the failure posture in writing. Fail-closed on audit is correct and it is also an availability risk. Both statements are true, and the operations team needs to know which one you chose before three in the morning.
4. Treat every read as observable. If a value can be read without leaving a record, it is not managed, it is merely stored somewhere newer.

Cross-course references

• Linux for Production Sysadmins - Part LXXII (Secrets) covers how credentials leak through scripts, shell history and world-readable configuration, which is the estate a secret manager is brought in to clean up.
• Kubernetes for Production Sysadmins - Part XXI (Secrets) covers the platform-issued service account token that becomes the workload first credential in the authentication step described here.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLII (CI Secrets) covers the masking limits that make a pipeline a poor place to hold long-lived credentials.

Quiz