External secret managers — Vault, External Secrets Operator, and patterns

A Kubernetes Secret is a rendering of a credential; the source of truth is outside the cluster. The cluster stores a copy that the workload consumes. This pattern — external secret manager as source of truth, Kubernetes Secret as rendering — is the standard production approach for high-value credentials. This lesson covers the integrations that make the pattern work.

Why external secret managers

flowchart LR
    A[Workload] -->|reads Secret| B[Kubernetes API]
    B --> C["Kubernetes Secret<br/>in etcd"]
    D["External secret<br/>manager"] -->|sync| C
    D -->|Vault / AWS SM| E[Source of truth]

The reasons to externalise:

• Rotation. A Vault or AWS Secrets Manager supports automatic rotation; the cluster’s Secret is refreshed from the source.
• Audit. External managers record every read. A Secret read in Kubernetes is not audited by default; an external manager logs every credential access.
• Multi-cluster. A single source of truth serves many clusters. Rotation in the source propagates everywhere.
• Compliance. Some compliance regimes (PCI DSS, HIPAA) require credential storage in dedicated systems, not cluster etcd.

The integration patterns

Three patterns are common in production:

flowchart TB
    A["External secret<br/>manager"] --> B{Integration}
    B -->|ESO| C["External Secrets Operator<br/>syncs to K8s Secret"]
    B -->|Vault Agent| D["Sidecar / init container<br/>writes to FS"]
    B -->|CSI driver| E["CSI secret driver<br/>tmpfs mount"]

Pattern 1: External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: prod
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials   # K8s Secret name
  data:
  - secretKey: password    # K8s Secret key
    remoteRef:
      key: database/app    # Vault path
      property: password

The External Secrets Operator (ESO) is a controller that periodically reads from the external store and writes a Kubernetes Secret. The K8s Secret is refreshed every refreshInterval.

flowchart LR
    A["Vault<br/>secret/database/app"] -->|every 5m| B[ESO controller]
    B -->|writes| C["K8s Secret<br/>db-credentials"]
    C -->|mounts| D[Pod]

The Pod consumes the K8s Secret normally. The credential is in etcd but it is a rendering; the source is Vault.

Pattern 2: Vault Agent sidecar

spec:
  containers:
  - name: vault-agent
    image: hashicorp/vault:1.16
    args:
    - agent
    - -config=/etc/vault/config.hcl
    volumeMounts:
    - name: vault-config
      mountPath: /etc/vault
    - name: secrets
      mountPath: /etc/secrets
  - name: web
    image: web:v1
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: vault-config
    configMap:
      name: vault-agent-config
  - name: secrets
    emptyDir:
      medium: Memory

Vault Agent runs as a sidecar. It authenticates to Vault, fetches the secret, and writes it to a shared emptyDir: medium: Memory volume. The main container reads the file.

flowchart LR
    A["Vault Agent<br/>sidecar"] -->|fetch from Vault| B[Vault]
    A -->|writes to| C["/etc/secrets/password<br/>tmpfs"]
    C --> D["Web container<br/>reads file"]

The credential never enters Kubernetes etcd. It exists only on the node’s memory and in the Vault server.

Pattern 3: CSI secret driver

spec:
  containers:
  - name: web
    image: web:v1
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: vault-db-credentials

The CSI secret driver (secrets-store.csi.k8s.io) is a CSI driver that fetches secrets from an external store and mounts them as a tmpfs volume. The application reads the files.

flowchart LR
    A["CSI driver<br/>secrets-store"] -->|fetch from Vault| B[Vault]
    A -->|mounts tmpfs| C["/etc/secrets/password"]
    C --> D[Web container]

The credential never enters Kubernetes etcd. It exists only on the node’s memory.

Comparison of patterns

The trade-off:

• ESO is the simplest. The credential is in etcd but rotation is automatic and audit is centralised.
• Vault Agent / CSI keeps the credential out of etcd entirely. Better for high-value credentials but more operational complexity.

Real-world: Vault + ESO

The most common production pattern. A Vault server is the source of truth; ESO syncs to Kubernetes Secrets; the workload consumes the K8s Secret.

flowchart TB
    A["Vault server<br/>source of truth"] -->|sync every 5m| B[ESO controller]
    B -->|writes| C["K8s Secret:<br/>db-credentials"]
    C -->|mounts| D[Pod]
    A -->|audit log| E[Audit pipeline]
    B -->|sync logs| E

The Vault server stores the credential with policy-based access:

vault kv put secret/database/app \
  username=app \
  password=s3cr3t
vault policy write app-read - <<EOF
path "secret/data/database/app" {
  capabilities = ["read"]
}
EOF

The ESO’s ServiceAccount is bound to a Vault role that allows read on secret/database/app. The K8s Secret is refreshed every 5 minutes; rotation in Vault propagates within the refresh interval.

Real-world: AWS Secrets Manager + ESO

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-sm
  namespace: prod
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: prod
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-sm
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: prod/db
      property: password

AWS Secrets Manager is the source; ESO syncs; the workload consumes the K8s Secret. Rotation in AWS SM propagates to the cluster.

Trust boundaries

flowchart TB
    A[Vault server] -->|TLS| B[ESO controller]
    B -->|TLS| C[API server]
    C -->|writes| D[etcd]
    B -->|ServiceAccount token| E[K8s API]
    E -->|RBAC| F{Allowed?}
    F -->|yes| G[Sync Secret]

The trust boundary: the ESO controller authenticates to Vault using Kubernetes ServiceAccount tokens (the Vault Kubernetes auth method). The token is verified by Vault; the K8s Secret is written. The token’s scope determines which Vault paths the controller can read.

A compromised ESO controller can read every Vault path it is bound to. The mitigation: tight Vault policies (only the paths the controller needs), short-lived tokens, and rotation cadence that matches the rotation of credentials.

Quiz

Production discipline

• Use an external secret manager for production credentials. Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault. The K8s Secret is a rendering.
• Tighten Vault policies. Every ServiceAccount bound to a Vault role has a least-privilege policy; an over-permissioned role reads too much.
• Audit external access. Every credential read in the external manager is logged; the K8s Secret read is not. Surface the external log in the audit pipeline.
• Rotate credentials on a cadence. The external manager’s rotation is the rotation; the K8s Secret refresh follows.
• Document the recovery path. A Vault outage means the K8s Secret is not refreshed. The K8s Secret still works (it’s in etcd); new Pods cannot fetch fresh credentials until Vault is back.

External secret managers are the production pattern for high-value credentials. The integration tooling (ESO, Vault Agent, CSI driver) handles the rendering; the discipline is in the policies, the rotation cadence, and the audit pipeline.