External secret managers — Vault, AWS SM, Doppler, Infisical · Docker & Containers

For serious production secrets — database credentials, cloud API keys, signing keys — an external secret manager is the right answer. The manager is the source of truth; the container fetches on demand.

That sentence hides the question this lesson is really about. If the container fetches a secret from the manager, it has to prove it is entitled to. What does it prove that with, and where did that credential come from?

The bootstrap problem

You cannot store the manager's credential in the manager

Every secret-manager deployment has exactly one credential that is not in the secret manager, because it is what opens the secret manager. This is the whole design problem, and how well a manager solves it is more important than its feature list.

There are four families of answer, and they are not equally good.

1. Platform identity — no credential at all. The cloud gives the compute instance an identity and the manager trusts it. An EC2 instance role, a GCE service account, an Azure managed identity. The container asks the instance metadata service for a short-lived token and presents that. There is no long-lived secret anywhere, which is why this is the best available answer whenever you are on a platform that offers it.

The Docker-specific caveat: the metadata endpoint is reachable from inside the container by default, so every container on that host inherits the host’s identity, not just the one that should. Scope the instance role to the union of what all containers on that host need — which is an argument for not co-locating unrelated workloads — or block 169.254.169.254 for containers that have no business reaching it.

2. A response-wrapped or single-use token. Vault’s AppRole splits authentication into a role_id (not secret, ships with the deployment) and a secret_id (secret, single-use or short-TTL). Something trusted delivers the secret_id at start-up; the container exchanges both for a token and the secret_id is spent. An attacker who reads it later gets nothing, and an attempted reuse is a detectable event rather than a silent compromise.

3. A long-lived token in a file. Honest, common, and much better than nothing: mode 0400, owned by the container’s UID, delivered by configuration management, rotated on a schedule. You have not eliminated the secret; you have reduced the problem to one credential with one purpose that you can audit and rotate independently of everything else.

4. A long-lived token in an environment variable. This is where most first implementations land, and it undoes the point of the exercise. The bootstrap credential can read every other secret, and you have put it in the one place that leaks to docker inspect, /proc, every child process and every crash reporter.

The honest framing: a secret manager does not remove secrets from your infrastructure. It reduces N distributed secrets to one bootstrap credential with a short life and an audit trail. That is a very large improvement, and it is not the same as zero.

The three delivery topologies

How the fetched value reaches the application matters as much as where it came from.

	App-native SDK	Sidecar agent	Init container
Who fetches	The application	A long-running companion	A container that runs and exits
Secret lands in	Application memory	A shared tmpfs file	A shared tmpfs file
Handles rotation	Yes, if you wrote it to	Yes, re-renders on change	No, only at start
Application changes needed	Substantial	None	None
Failure blocks start	Yes	Yes, with `service_healthy`	Yes, with `service_completed_successfully`

App-native is the cleanest and the most work. The application holds a client, refreshes on a timer, and never writes the value anywhere. Nothing is on disk at any point. It requires touching the application, which rules it out for anything you did not write.

Sidecar is the workhorse. Vault Agent is the canonical implementation: it authenticates once, keeps its token renewed, renders secrets into files from templates, and re-renders when the value changes. The application reads a file and knows nothing about Vault.

Init container is the simplest and covers most cases. A container fetches the secret, writes the file, exits. Compose waits for it with service_completed_successfully. There is no rotation — the value is fixed until the next deploy — which is fine for credentials with a 90-day life and unacceptable for a one-hour dynamic lease.

A sidecar that is correct

Here is the shape, with the details that matter marked.

services:
  vault-agent:
    image: hashicorp/vault:1.18
    command: ["vault", "agent", "-config=/etc/vault/agent.hcl"]
    environment:
      VAULT_ADDR: https://vault.example.com:8200
    configs:
      - source: agent_config
        target: /etc/vault/agent.hcl
    secrets:
      - source: approle_secret_id
        target: secret-id
        mode: 0400
    volumes:
      - rendered:/rendered
    healthcheck:
      test: ["CMD", "test", "-s", "/rendered/db_password"]
      interval: 5s
      timeout: 2s
      retries: 30

  api:
    image: myorg/api:1.4.0
    user: "1001:1001"
    environment:
      DB_PASSWORD_FILE: /rendered/db_password
    volumes:
      - rendered:/rendered:ro
    depends_on:
      vault-agent:
        condition: service_healthy

configs:
  agent_config:
    file: ./vault/agent.hcl

secrets:
  approle_secret_id:
    file: ./vault/secret-id

volumes:
  rendered:
    driver: local
    driver_opts:
      type: tmpfs
      device: tmpfs
      o: "size=1m,mode=0700,uid=1001"

The shared volume is the mistake, and it writes your secrets to disk

The obvious way to share the rendered file between the sidecar and the application is a named volume. A named volume backed by the local driver is a directory under /var/lib/docker/volumes/<name>/_data on the host disk.

So the whole exercise — fetch from Vault, never put it in the image, never put it in an environment variable — ends with the plaintext credential written to the host filesystem, where it is picked up by the host backup, replicated to the backup target, and left behind after docker compose down because down does not remove named volumes.

The driver_opts block above is the fix. It uses the same type, o and device pass-through that the Compose reference documents for NFS volumes, but hands mount(8) a tmpfs instead of a directory. The rendered secret lives in RAM, and it is gone when the volume is removed. Size it small — a 1 MB tmpfs holds any credential and caps the damage if something loops.

The same trap catches the init-container pattern, and it catches it more often, because there the volume is the only thing the two containers share and reaching for a named volume is instinctive.

And note the tmpfs caveat from the storage part: tmpfs pages can be swapped. If this is a compliance requirement rather than a preference, the host needs no swap or encrypted swap.

Read-only / Safeverify it is really tmpfs— The check that distinguishes a secret in RAM from a secret on the host disk.

VOLUME=$(docker compose ps -q vault-agent | xargs -r docker inspect --format '{{range .Mounts}}{{.Name}} {{end}}')

# Where the daemon thinks it is
docker volume inspect rendered --format '{{.Options}} {{.Mountpoint}}'

# What the kernel actually mounted there. Expect FSTYPE tmpfs.
findmnt -no FSTYPE,SIZE,OPTIONS "$(docker volume inspect rendered --format '{{.Mountpoint}}')"

# And the definitive check: is the plaintext on the host disk?
grep -rl --binary-files=without-match "$(cat ./vault/known-test-value)" /var/lib/docker/volumes 2>/dev/null || echo 'not on disk'

Read-only / Safethe two possible answers— Top: the driver_opts tmpfs took effect. Bottom: a plain named volume, and the plaintext is on disk.

$ findmnt -no FSTYPE,SIZE,OPTIONS $(docker volume inspect rendered --format '{{.Mountpoint}}')

# correct — the volume is RAM-backed
tmpfs   1M  rw,relatime,size=1024k,mode=700,uid=1001,inode64

# wrong — the volume is a directory on the host filesystem
ext4  879G  rw,relatime

Illustrative output

FSTYPE tmpfs passes. FSTYPE ext4 means every rendered secret is in your host backups — and the giveaway is the SIZE column, which reports the whole host filesystem rather than the small cap you thought you had set.

The managers

HashiCorp Vault

The most capable option and the most operational work. What you buy:

Dynamic secrets. Vault creates a Postgres role on demand with a one-hour lease and drops it when the lease expires. Nobody, at any point, knows a long-lived database password — there isn’t one. This is the single strongest reason to run Vault.
Transit. Encryption as a service. The application sends plaintext and receives ciphertext; key material never leaves Vault.
PKI. Issue and rotate certificates with short lives.
A real audit device. Every read is logged with who, what and when.

# Store and read a static secret
vault kv put secret/database/prod password=REPLACE_ME
vault kv get -field=password secret/database/prod

The cost is that Vault is a stateful, quorum-based service you now have to run, unseal, back up and upgrade. Vault being down is an outage in everything that starts.

AWS Secrets Manager and Parameter Store

For AWS-native deployments, and the bootstrap problem largely disappears: the instance or task role is the identity.

aws secretsmanager create-secret \
  --name prod/database/password \
  --secret-string REPLACE_ME

aws secretsmanager get-secret-value \
  --secret-id prod/database/password \
  --query SecretString --output text

Secrets Manager supports scheduled rotation with a Lambda function, including managed rotation for RDS. Parameter Store is cheaper and simpler for values that do not need rotation. Both are billed per secret and per API call, which matters if a fleet re-fetches on every container start.

Google Secret Manager

printf '%s' REPLACE_ME | gcloud secrets create prod-database-password --data-file=-
gcloud secrets versions access latest --secret=prod-database-password

Note printf '%s' rather than echo. echo appends a newline and the newline becomes part of the secret — the failure from the previous lesson, arriving here at creation time instead of read time.

Authentication is via Application Default Credentials, which on GCE means the instance service account and no stored credential.

Doppler and Infisical

Simpler operationally, less feature-rich. Both do versioned key-value secrets with environments, access control and audit logs, and both integrate with CI through a service token. Infisical is open source and can be self-hosted; Doppler is SaaS with a free tier.

Neither does dynamic database credentials, which is the honest dividing line: if what you need is “one place for our secrets, with history and access control”, they are a much better fit than Vault. If you need credentials that expire on their own, you need Vault or a cloud-native equivalent.

Choosing

Need	Pick
Dynamic database credentials, transit, PKI	Vault
AWS-native, want the instance role to be the identity	AWS Secrets Manager
GCP-native	Google Secret Manager
One shared store with history and RBAC, minimal ops	Doppler or Infisical
Air-gapped or on-prem, no SaaS	Vault or Infisical, self-hosted

What happens when the secret manager is unreachable at start-up?

Nothing starts. That is the correct behaviour and it is also a new failure domain you did not have before, so it needs planning rather than discovery.

The single-container case is benign. depends_on with service_healthy holds the application until the agent has rendered a file. The container restarts, fails again, backs off. Recovery is automatic when the manager returns.

The mass-restart case is not. A host reboot, a daemon upgrade, or a rolling deploy across a fleet starts every container at once, and every one of them authenticates and fetches at once. A Vault cluster sized for steady-state reads gets a thundering herd; a cloud manager with per-call billing and a request quota starts returning throttling errors. Containers fail, restart, and retry — which is a retry storm on a service that is already saturated, and it does not converge on its own.

Mitigations, in order of effectiveness: stagger restarts rather than restarting a whole fleet; use restart: on-failure with a backoff rather than always with a tight loop; give the agent a token TTL long enough that a normal restart renews rather than re-authenticates; and cache. Vault Agent can cache and serve tokens and leases, which turns N container fetches into one upstream call.

The circular-dependency case is the one that ruins a night. If Vault’s own storage backend, or the monitoring you would use to diagnose Vault, or the DNS that resolves vault.example.com runs in a container that fetches its secrets from Vault, then Vault being down means you cannot start the thing that would tell you why Vault is down.

Draw that graph before you need it. The manager, its storage, DNS and your break-glass access must not depend on the manager. And keep a documented break-glass path — a sealed envelope, an offline recovery key, a root token in a safe — because the alternative is discovering that recovery requires a credential only the dead system can issue.

Choose by the bootstrap story, not the feature list. If the platform can give the container an identity with no stored credential, that beats every feature comparison.
Deliver as a file, never as an environment variable. Including the bootstrap credential itself.
Render into a tmpfs. Verify with findmnt, not by reading the Compose file.
Set a small tmpfs size and a restrictive mode. size=1m,mode=0700, with uid= set to the UID the application runs as.
Make the failure explicit. depends_on with service_healthy or service_completed_successfully, so a missing secret is a container that will not start rather than one that starts wrong.
Plan the mass restart. Token TTLs, backoff, caching, staggered rollout.
Read the audit log. A secret manager whose audit log nobody looks at is a filesystem with extra steps. Alert on reads from unexpected identities and on rotation failures.