Secrets - the credential management problem

A secret is anything that grants access: a password, a key, a token, a certificate. Secret management is the discipline of keeping secrets safe and available.

Why secrets are a problem

Secrets are everywhere in production:

• Database passwords.
• API keys for cloud services.
• TLS private keys.
• SSH keys.
• OAuth tokens.
• Service account credentials.
• Encryption keys.

Each is a target. Each can leak. Each can be exploited.

Common leak paths

• Code repositories: secrets in git history.
• Configuration files: passwords in plain text.
• Environment variables: readable through /proc/PID/environ, systemctl show -p Environment, container inspect output, and core dumps.
• Command lines: readable through /proc/PID/cmdline, visible to every user in ps, and recorded in shell history.
• Logs: secrets in error messages.
• Backups: secrets in unencrypted backups.
• Memory dumps: secrets in core dumps.
• Network: secrets over unencrypted protocols.

The first step is to identify where secrets live in your infrastructure.

The two leak paths you can demonstrate right now

Environment variables and command lines are the leaks people name most often and describe least accurately. Both are kernel-visible, and both are easy to prove.

/proc/PID/environ

A process’s environment is a file:

# Your own process: always readable
tr '\0' '\n' < /proc/self/environ | head

# Another process: readable by root, and by the owning user
sudo tr '\0' '\n' < /proc/$(pgrep -f myapp)/environ | grep -i pass

The environment is captured at exec time and stays readable for the life of the process. So a secret passed as an environment variable is not “gone after the shell exits” - it sits in kernel memory until the service stops, and any root process, any core dump, and systemctl show -p Environment myapp.service can read it back.

/proc/PID/cmdline

Arguments are worse, because /proc/PID/cmdline is world-readable by default:

mysql -h db -u app -phunter2 &     # do not do this
ps -eo pid,args | grep mysql       # any user on the host sees it
tr '\0' ' ' < /proc/$!/cmdline     # so does this

mysql -pPASSWORD, curl -u user:pass, mysqldump -p..., wget --password=, helm --set password= and kubectl create secret --from-literal= all leak this way. The window is short but it is not zero, and on a shared host or a CI runner “short” is enough.

The safe forms take the value from a file or from stdin:

# MySQL: a 0600 option file, not an argument
printf '[client]\npassword=%s\n' "$PW" > ~/.my.cnf
chmod 600 ~/.my.cnf
mysql -h db -u app

# curl: netrc or stdin, never -u user:pass
curl --netrc-file /run/secrets/netrc https://api.example.com
curl -H @/run/secrets/auth-header https://api.example.com

Principles of secret management

• Never in code or config: secrets are not in the source repository.
• Encrypted at rest: secrets are encrypted when stored.
• Encrypted in transit: secrets are transmitted over TLS.
• Access logged: every access to a secret is logged.
• Rotated regularly: secrets change periodically.
• Scoped: each secret has the minimum permissions needed.
• Audited: who accessed what, when.

Choose the right store

For most production:

• HashiCorp Vault: self-hosted, dynamic secrets, policies. The standard.
• AWS Secrets Manager: managed, KMS-integrated. For AWS workloads.
• Azure Key Vault: managed, Azure-integrated. For Azure workloads.
• GCP Secret Manager: managed, GCP-integrated. For GCP workloads.
• sops + age: file-based, git-friendly. For config-in-git workflows.
• Kubernetes Secrets: simple, in-cluster. For K8s.

For self-hosted, Vault. For cloud, the cloud provider’s secret manager. For config-in-git, sops.

Write to the store without leaking on the way in

Every one of these tools has an obvious invocation that puts the secret value in argv, and a slightly less obvious one that does not. Use the second:

# Vault: read the value from stdin (the trailing "-"), never from argv
printf '%s' "$PW" | vault kv put secret/myapp/db password=-

# or from a 0600 file, with the "@" file reference
vault kv put secret/myapp/db password=@/run/secrets/db.pw

# AWS: file:// reference, not an inline literal
aws secretsmanager create-secret --name myapp/db \
    --secret-string file:///run/secrets/db.json

# sops: --age takes an age RECIPIENT PUBLIC KEY, not a number
sops --encrypt \
     --age age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p \
     --in-place secrets.yaml

/run is a tmpfs, so a staging file there never reaches disk. Create it mode 0600 and remove it when you are done.

Secrets in git

Git history is append-only by design. Deleting a secret in a new commit does not remove it - the blob is still reachable from every earlier commit, from every clone, from every fork, and from the provider’s cached views of pull requests.

Two separate jobs follow, and the order matters:

1. Rotate the secret first. Issue a new credential, deploy it, revoke the old one. This is the only step that closes the exposure.
2. Then rewrite the history with git filter-repo (the supported successor to filter-branch) or BFG, to remove the blob from every commit.
3. Expire the old objects on the server (git reflog expire --expire=now --all then git gc --prune=now) and ask the hosting provider to purge cached pull-request views.
4. Force-push, and tell every clone holder to re-clone: every object ID has changed.
5. Add a pre-commit scanner so the next one never lands.

# Remove a file from every commit in history
git filter-repo --invert-paths --path config/secrets.yml

# Scan before committing, and scan the whole history
gitleaks protect --staged --redact
gitleaks detect --source . --redact

Getting a secret onto a Linux host

Choosing a store is the easy half. The mechanics of delivering a secret to a running service - without it landing in a world-readable file, a command line, or an image layer - is the half that people skip.

systemd credentials: the native path

systemd can hand a service a secret as a file that only that service can read, decrypted at start time with a key bound to the host:

# Encrypt once, as root, on the host that will run the service
systemd-creds encrypt --name=dbpw plaintext.pw \
    /etc/credstore.encrypted/dbpw
shred -u plaintext.pw

# /etc/systemd/system/myapp.service
[Service]
LoadCredentialEncrypted=dbpw
ExecStart=/usr/bin/app --password-file=%d/dbpw

%d expands to the credentials directory, which systemd mounts as a private tmpfs visible only to that unit. Nothing appears in the environment, nothing in argv, nothing on disk in plaintext, and the ciphertext is useless on any other host because the key is sealed to this one (to the TPM where one is available).

Inspect it with:

systemd-creds list
systemd-analyze security myapp.service

Configuration in git: ansible-vault and sops

When the secret must live alongside the configuration that uses it, encrypt it in place:

# Encrypt one variables file
ansible-vault encrypt group_vars/prod/vault.yml

# Encrypt a single value inline
ansible-vault encrypt_string --stdin-name db_password

# Read the vault password from a file or a script, not a prompt
ansible-playbook site.yml --vault-password-file /run/secrets/vault-pw

The trade-off is real and worth stating: the encrypted blob is in git forever, so its strength is only the strength of the vault password, and rotation means re-encrypting. That is acceptable for a small team and a handful of values. It is not a substitute for a real store at scale.

Dynamic secrets: vault-agent templating

The strongest pattern removes the long-lived secret entirely. vault-agent authenticates with the host’s own identity, fetches a short-lived credential, renders it into a file, and signals the service when it changes:

template {
  source      = "/etc/vault/db.env.tpl"
  destination = "/run/secrets/db.env"
  perms       = "0400"
  command     = "systemctl reload myapp"
}

The credential now lives for hours rather than years, on a tmpfs, mode 0400. A leak has an expiry date.

File-mode discipline

Whatever delivers the secret, the file it lands in decides who can read it:

install -o myapp -g myapp -m 0400 -D /dev/null /run/secrets/db.pw
umask 077                      # before writing any secret
find /etc /run/secrets -type f -name '*.pw' -perm /o=r

Mode 0400, owned by the service account, on a tmpfs, and never inside a directory that a backup job copies unencrypted.

Knowledge check