Distribution without proliferation - counting the copies

Storage gets the attention, but delivery is where the damage is done. A value sitting in one hardened store is a manageable risk. The same value after it has been rendered into a template, written to a file, inherited by four child processes and captured by a nightly backup is a different object entirely, and the difference was created by the act of getting it to work.

A copy is anywhere the plaintext can be recovered

Define the term precisely, because a loose definition is what lets teams believe they have one copy of something they have nine of. A copy is any location from which the plaintext can be recovered without the cooperation of the running process. That definition deliberately includes places nobody chose.

• The source of truth. The intended copy.
• The transport. Whatever buffered the value in flight, including the intermediate that rendered a template around it.
• The destination. The file, the environment block, the mounted tmpfs entry.
• The derived copies. The backup of the destination, the snapshot of the volume, the container image layer if the file was present at build time.
• The incidental copies. The log line that recorded the transport, the core dump, the swap page, the crash report that serialised the process state.

flowchart LR
    S["Secret store\n(intended copy)"] --> T["Transport\nor renderer"]
    T --> F["File or env\non the host"]
    F --> P["Process memory"]
    F --> B["Backup and\nvolume snapshot"]
    T --> L["Transport log"]
    P --> C["Core dump\nor swap page"]

The store on the left is the copy a team defends. The five boxes to its right are copies a team usually inherits. Any credible distribution design is an argument about which of those boxes exist at all, not an argument about how well the leftmost box is protected.

The six patterns, compared honestly

No pattern is free. Each buys a reduction in one column by paying in another.

The first row deserves its reputation. Deleting a file in a later build step does not remove it from the image: removals in the OCI layer format are recorded as whiteout entries that apply to lower layers, and the earlier layer blob still contains the plaintext and is still pushed, pulled and cached by digest. Build arguments are no better, because Docker documents that build arguments and environment variables are inappropriate for passing secrets since they are exposed in the final image, and that they persist in image history and provenance attestations.

# ANTI-PATTERN: the token is now in the layer, the history and
# the provenance attestation, and no later RUN can remove it.
# docker build --build-arg API_TOKEN=not-a-real-token .

# The documented mechanism: a build-time mount that leaves no layer.
docker build --secret id=api_token,src=./api_token.txt .

Inside the build, that secret appears by default at a path under /run/secrets named for the secret id, and it exists only for the duration of the instruction that mounts it. Set the mount option required=true, because the default is false and a mistyped id otherwise yields an empty file and a green build.

The middle rows are where most estates actually live, and they are where the honest comparison matters. Runtime injection by the orchestrator looks the same in configuration across platforms, which is exactly what makes it misleading.

services:
  api:
    image: registry.example.com/api:1.4.2
    secrets:
      - db_password
secrets:
  db_password:
    external: true

That declaration causes the value to appear inside the container as a file under /run/secrets named for the secret, rather than in the environment. The composition file itself carries no plaintext, which is a genuine improvement over an inline value. What differs between platforms is everything behind the declaration. An orchestrator-injected secret has genuinely good properties in some platforms and genuinely poor ones in others. Docker Swarm stores the value in an encrypted Raft log, mounts the decrypted secret into an in-memory filesystem inside the container, and flushes it from the node when the task stops. Kubernetes, by contrast, stores Secrets unencrypted in etcd by default, and its own documentation states that anyone authorised to create a Pod in a namespace can use that access to read any Secret in that namespace, including indirectly by creating a Deployment. Those are two different security models wearing similar-looking YAML.

The bootstrap problem and where the chain ends

Every pattern below the first row needs the workload to prove who it is before the store will answer. That proof requires a credential, which needs to be delivered, which is the same problem again. The chain has to terminate in something that is not a delivered secret at all.

Three things can legitimately terminate it. A hardware root, such as a key that never leaves a security module on the machine. A platform attestation, where the infrastructure that created the workload vouches for it: a projected service account token bound to the Pod, an instance identity document, a workload identity issued by the platform. Or a federated assertion, where an external identity provider signs a short-lived statement that the resource is willing to trust.

# The job asks its platform for an identity assertion. Nothing
# long-lived is stored anywhere in the repository.
permissions:
  id-token: write
  contents: read

That two-line permission block is the whole of the credential storage in a federated pipeline. The platform mints a token for the job, the cloud provider validates the claims in it and returns access valid only for that job. The trust policy on the other side must pin the audience and the subject claim, because pinning the repository alone allows any workflow in that repository, including one added by a pull request, to assume the role.

The one pattern that never terminates the chain is delivering the bootstrap credential by the mechanism you were trying to avoid. An agent credential baked into the image so the agent can fetch everything else has moved the problem rather than solved it, and has made it worse, because that one credential now unlocks all of the others.

Reducing the copy count in an estate you inherited

The realistic path is not a migration to the bottom row. It is a sequence of reductions that each removes one box from the diagram.

Start by removing the derived copies, because they are the cheapest to eliminate and the hardest to find later. Move rendered secrets off durable filesystems onto memory-backed mounts so that the volume snapshot stops capturing them. Exclude the paths from backup selectors deliberately rather than by accident. Then remove the incidental copies: disable core dumps for processes that hold key material, and audit the transport for anything that logs its payload. Only then attempt to change the delivery mechanism itself, because that step requires the consumer inventory from the lifecycle model and will stall without it.

Production discipline

1. Count the copies in the design review, in writing. The number belongs in the design document beside the diagram, and a later change that increases it should require the same review as the original.
2. Never deliver the bootstrap credential the way you refused to deliver the others. If the agent credential is baked in, the agent has weakened the estate rather than strengthened it.
3. Prefer memory-backed delivery for anything a backup could otherwise capture. The snapshot is the copy nobody remembers making and nobody can enumerate later.
4. Set the build secret mount to required. The default is permissive and turns a typo into a silently empty credential and a passing build.
5. Write down the availability coupling you just accepted. Fetch-at-start couples the workload to the store at restart time, which is exactly when an incident is already restarting things.

Cross-course references

• Kubernetes for Production Sysadmins - Part LX (ServiceAccounts) covers projected token volumes and audiences, which is the platform attestation that legitimately terminates the bootstrap chain inside a cluster.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLIII (OIDC) covers the federation flow and the trust policy claims that make the bottom row of the comparison table workable in a pipeline.
• Linux for Production Sysadmins - Part LXXII (Secrets) covers the file ownership and mode decisions that determine which local accounts become unrecorded holders once a rendered file lands on a host.

Quiz