Secret mounting risks — how Secrets leak

Mounting a Secret into a Pod is the operational moment of risk: the Secret value is delivered to a container that may have arbitrary code. The mounting choice (env var vs volume mount) determines the exposure surface; the file permissions determine who in the container can read it. This lesson covers the mounting options, the leak vectors, and the production patterns.

The two mounting options

A Secret can be delivered to a container in two ways:

1. Environment variable. The Secret value is injected as an env var:

spec:
  containers:
  - name: api
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

2. Volume mount. The Secret is mounted as a file:

spec:
  containers:
  - name: api
    volumeMounts:
    - name: db-credentials
      mountPath: /etc/secrets/db
      readOnly: true
  volumes:
  - name: db-credentials
    secret:
      secretName: db-credentials
      defaultMode: 0400

flowchart LR
    A[Secret] --> B{How mounted?}
    B -->|env var| C[Every process sees it]
    B -->|volume mount| D[File in mount path]
    C --> E[Visible in /proc/<pid>/environ]
    D --> F[File permissions limit access]

The leak vectors

Five vectors:

1. /proc/<pid>/environ. Any process in the container can read the env vars of any other process. A malicious sidecar or a compromised workload can dump every Secret.

2. Shell history. A shell that sources an env var (e.g., export $(cat /etc/secrets/db/password)) may save the value in ~/.bash_history. The fix is to read the value into a variable without echoing it.

3. Crash dumps. A workload that crashes may include the env vars in the crash report. The fix is to scrub the env vars from the crash report.

4. Logs. A workload that logs its configuration may include the Secret value. The fix is to scrub the Secret value from logs.

5. Container escape. A privileged container that escapes its mount namespace can read the host’s /proc/<pid>/environ, which includes the workload’s env vars. The fix is PSS restricted.

Volume mounts are safer

A volume mount with defaultMode: 0400 is safer than an env var:

volumes:
- name: db-credentials
  secret:
    secretName: db-credentials
    defaultMode: 0400  # readable only by the file's owner

The kubelet creates the file with mode 0400. Only the process that runs as the file’s UID can read it. Other processes in the container cannot read it.

The kubelet audit

The kubelet’s --audit flag records every Secret mount:

# Verify the kubelet flags
cat /var/lib/kubelet/config.yaml | grep -E "read-only-port|anonymous-auth"

The audit log records the Pod’s UID, the Secret name, and the mount path. The SIEM alerts on unexpected Secret mounts.

Production patterns

1. Volume mounts with 0400 for production Secrets. Env vars are acceptable for non-sensitive configuration.
2. runAsUser non-zero. The file owner is a non-root user; the Secret is not readable by root.
3. readOnly: true on the mount. The workload cannot write to the mount path; the Secret is immutable from the workload’s perspective.
4. No kubectl exec with the Secret. The workload’s audit log records every kubectl exec that accesses the Secret; the operator verifies.

Common failure modes

1. Env vars used for high-value Secrets. The Secret is visible in /proc. The fix is a volume mount with 0400.
2. defaultMode: 0644 on the mount. The file is world-readable; any process in the container can read it. The fix is 0400.
3. runAsUser: 0. The file is owned by root; any process in the container can read it. The fix is runAsUser: <non-zero>.
4. Workload logs the Secret. The Secret value appears in the workload’s logs. The fix is a redaction list in the logger plus a drop rule at log ingest; the value otherwise survives in the log store long after the Secret is rotated.

Production failure modes

1. Secret mounted but not needed. A workload that does not use the Secret has the Secret in its environment. The fix is to remove the mount.
2. Multiple Secrets in one Pod. A Pod with many Secrets has many files in /etc/secrets. The fix is to scope the Secrets per-container.
3. Secret rotation not handled. A workload that reads the Secret at startup does not pick up the rotated value. The fix is to re-read periodically.
4. Crash dump leaks. A workload that crashes dumps the env vars in the crash report. The fix is to scrub the report.

Cross-course references

• The Linux course covers file permissions, /proc, and shell history.
• The Observability course covers the audit log entries for Secret access.

Quiz

Production discipline

The mounting choice determines the exposure surface. A defensible Secret programme uses volume mounts with 0400 permissions and runAsUser: <non-zero> for high-value Secrets, env vars for non-sensitive configuration, and scrubs Secret values from logs, crash dumps, and shell history. The audit log records every Secret access; the SIEM alerts on unexpected access. A cluster whose Secrets are mounted safely has a Secret programme that is auditable.