Where credentials really leak

Almost nothing in this course leaks because an algorithm failed. Credentials escape through ordinary plumbing: a commit, a log line, a state file, an image layer, a screenshot in a ticket. Each of those channels has documented behaviour that explains why, and knowing the behaviour changes what an engineer does during design and during the first ten minutes of an incident.

The exposure inventory

An exposure inventory is a list, per secret, of every location it currently rests in and every channel it crosses. Not the locations the architecture diagram shows. The locations it is actually in, discovered by looking. The list is useful because each entry has a different owner, a different retention period and a different chance of anyone noticing a read.

flowchart LR
    S["One database password"] --> A["Secret store"]
    S --> B["Repository history\nfrom a 2024 commit"]
    S --> C["Pipeline log\nfrom a debug run"]
    S --> D["Terraform state\nin object storage"]
    S --> E["Image layer\nfrom a build argument"]
    S --> F["Incident channel\nmessage"]

The diagram shows one credential and six independent compromise paths, which is a realistic count for a value that has been in production for two years. Only the first entry is governed by the access policy anyone reviews. The other five are governed by whoever administers a repository, a log platform, an object store, a registry and a chat workspace, and none of those people were consulted when the secret was classified.

Building the inventory is mechanical. Search the repository history rather than the working tree. Search the log platform for the value and for its base64 and URL-encoded forms. Read the state files. List the image layers. Search the chat archive. Every hit is an entry, and the total is the number that belongs in the risk conversation.

Version control keeps what you deleted

Git stores content, addressed by hash. A commit references a tree, and the tree references blobs. Deleting a file in a later commit produces a new tree that omits the blob, but the earlier commit still references it, so the blob remains in the object database, in every clone anyone has taken, and in every mirror. The working tree is clean and the secret is exactly as present as it was the day it was committed.

# Search the whole history, not the current checkout.
git log --all --oneline -- config/credentials.yaml

# Count every object that has ever been reachable in this repository.
git rev-list --objects --all | wc -l

The second command is worth running once on a repository you believe is clean, because the number is usually far larger than the file count suggests, and every one of those objects is readable by anyone who can clone. History rewriting reduces the population for future clones, but it does not reach the clones that already exist, the forks, the build caches or the mirrors.

Redaction is best effort, and the gaps are documented

Pipeline platforms mask registered secret values in log output, and teams reasonably infer that a printed secret is therefore harmless. The masking is real and the inference is wrong, for reasons the platforms themselves document.

• Exact matching breaks on structure. A secret printed inside a JSON or YAML document may be escaped, quoted or wrapped across lines, and the redactor is matching a literal string, so the transformed form passes straight through.
• Derivatives are not covered. A base64 or URL-encoded form of the value is a different string and is not masked unless it was registered separately.
• Only the current job is covered. Values not referenced by the running job are not in the redaction set, so a secret echoed from an unexpected source is printed intact.
• Late registration does not reach earlier output. Adding a mask during a job does not retroactively redact what has already been written.

The related trap is which pipeline triggers can reach secrets at all. A job triggered by an ordinary pull request from a fork does not receive them. The trigger that runs in the context of the target repository does, together with write permissions, even when the change came from a public fork. That distinction is one word in a workflow file and is the difference between a contribution and a credential handover.

Configuration management has its own version of the same lesson. Encrypting variables at rest protects the file, not the run: once decrypted the value is an ordinary variable, and the suppression flag that hides task output has documented gaps around debugging output, difference display and values that appear in dictionary key names.

Artefacts and state files carry credentials forward

Infrastructure state is the quietest of these channels because it is machine-written and rarely read by a human. Marking a variable as sensitive controls how the value is displayed on the command line and in the user interface; the value is still recorded in state, and the commands that print outputs in JSON or raw form print it in clear regardless of the marking. Remote state in object storage is encrypted only when encryption was explicitly configured. The modern answers are the ephemeral values and ephemeral resources introduced in Terraform 1.10 and the write-only arguments introduced in 1.11, which pair a write-only attribute with a version attribute so the value is sent to the provider without ever being persisted.

Container images carry credentials just as durably, and for a structural reason. Layers are additive, so removing a file in a later instruction records a deletion marker while the earlier layer still ships the original bytes. A value passed as a build argument also persists in image history and in provenance attestations. The correct mechanism mounts the secret for the duration of one instruction.

# The value is mounted for one command and never becomes a layer.
RUN --mount=type=secret,id=npm_token,required=true \
    NPM_TOKEN="$(cat /run/secrets/npm_token)" npm ci

The required=true setting matters more than it looks. Without it the mount is optional, so a build that was never given the secret proceeds with an empty file and fails later in a way that looks like a registry problem rather than a missing credential.

Platform stores and the human channels

A Kubernetes Secret is encoded, not encrypted. Its contents are base64 and are stored unencrypted in etcd unless encryption at rest has been configured. More importantly, the isolation is weaker than the object name suggests: anyone authorised to create a pod in a namespace can use that access to read any Secret in that namespace, including indirectly by creating a deployment that mounts it, and a privileged container can read every Secret in use on its node. Namespace boundaries are an authorisation boundary, not a containment boundary.

The remaining channels are the ones that never appear in a threat model. Application telemetry captures request headers, and an authorisation header captured into a trace is a credential in the observability platform, retained for whatever that platform retains. Crash dumps and heap dumps contain process memory, which is where every decrypted secret lives. Database dumps taken for a staging refresh carry whatever the credential tables hold. And an engineer pasting a working command into an incident channel at 02:00 puts a live credential into a searchable archive with a different retention policy and a different membership list from anything the security review examined. Runtime exposure on the host itself, through process environment and command-line arguments, adds two more entries to the same inventory.

Production discipline

1. Scan history, not the working tree. A repository check that inspects only the current checkout will report clean on the exact repositories that are not.
2. Measure time to rotate and rehearse it. The number that bounds a leak is how long invalidation takes, so it deserves the same practice as a restore drill.
3. Search for derivatives during an incident. Look for the base64 and URL-encoded forms of a leaked value in logs and artefacts, because the redactor did not.
4. Keep secrets out of build inputs entirely. Mount them for a single instruction rather than passing them as build arguments, and require the mount so a missing secret fails loudly.
5. Add the quiet channels to the inventory. Telemetry, crash dumps, state files and chat archives are storage locations with owners and retention, and they belong on the list beside the secret store.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part XCIV (IncSecretLeak) covers the incident procedure that follows the discovery this lesson describes, including scope assessment and rotation ordering.
• Terraform for Production Sysadmins - Part XIX (Security) covers the state file as a credential store and the newer language features that keep values out of it.
• Docker & Containers for Production Sysadmins - Part XXXVI (Supply-Chain) covers image provenance and the metadata that travels with an artefact long after the build host is gone.

Quiz