External Secrets Operator — Secrets outside Kubernetes

For high-value credentials, the Kubernetes Secret is the wrong primitive: it lives in etcd, in the Git history (if committed), and in every cluster backup. The right primitive is an external secret store (Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) and a controller (External Secrets Operator, ESO) that mirrors the value into a Kubernetes Secret on demand. This lesson covers the ESO workflow, the configuration, and the production patterns.

The ESO architecture

ESO has three components:

1. SecretStore — a namespaced CRD that references the external provider (Vault, AWS, GCP, Azure).
2. ExternalSecret — a namespaced CRD that references the SecretStore and the path to fetch.
3. Controller — a Deployment that watches the CRDs and reconciles the Kubernetes Secret.

flowchart LR
    A[Vault/AWS/GCP/Azure] --> B[ESO controller]
    B --> C[Kubernetes Secret]
    D[SecretStore CRD] --> B
    E[ExternalSecret CRD] --> B

The controller reads the value from the external store and writes it to a Kubernetes Secret. The workload reads the Kubernetes Secret as usual.

A SecretStore for Vault

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-store
  namespace: prod
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "prod-api"
          serviceAccountRef:
            name: "api-sa"

The SecretStore references Vault’s Kubernetes auth method. The api-sa SA authenticates to Vault via a projected token; Vault returns a Vault token; ESO uses the token to read secrets.

An ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: prod
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: vault-store
    kind: SecretStore
  target:
    name: db-credentials  # the Kubernetes Secret to create
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: prod/db
      property: password

The ExternalSecret reads prod/db’s password field from Vault and writes it to the db-credentials Kubernetes Secret every 5 minutes. The Kubernetes Secret is owned by the ExternalSecret (the controller deletes it if the ExternalSecret is deleted).

Rotation

The rotation cycle:

1. The external value changes (Vault rotates the credential, or a cloud provider rotates the API key).
2. ESO detects the change at the next refreshInterval (5 minutes by default).
3. The Kubernetes Secret is updated with the new value.
4. The workload reads the new value (either immediately, if it re-reads, or on the next Pod restart).

For workloads that need immediate rotation, the Secret can be mounted as a volume and re-read on every API call. For workloads that cache the value, the rotation is delayed until the next Pod restart.

A SecretStore for AWS Secrets Manager

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-store
  namespace: prod
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: api-sa

The SecretStore uses IRSA (IAM Roles for Service Accounts) to authenticate to AWS. The api-sa SA assumes an IAM role; the role grants access to the Secrets Manager API.

RBAC for ESO

ESO’s controller SA needs broad RBAC (it creates Secrets across namespaces). The principle of least privilege does not apply to the controller itself; it applies to the workloads that read the mirrored Secrets.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eso-controller
subjects:
- kind: ServiceAccount
  name: eso-controller
  namespace: external-secrets
roleRef:
  kind: ClusterRole
  name: eso-controller
  apiGroup: rbac.authorization.k8s.io

The controller’s ClusterRole allows it to create, update, and delete Secrets cluster-wide. The workloads that read the Secrets have their own minimum-surface RBAC.

Production patterns

1. Vault or cloud KMS as the source of truth. The high-value credentials live in an external store.
2. ESO mirrors on a regular interval. 5 minutes is the common default; shorter intervals increase load on the external store.
3. creationPolicy: Owner. The Kubernetes Secret is owned by the ExternalSecret; deletion is automatic.
4. Workloads re-read the Secret. Volume mounts are better than env vars for fast rotation.

Production failure modes

1. External store is down. ESO cannot refresh; the Kubernetes Secret is stale. The fix is HA external store and observability.
2. ESO RBAC is wrong. The controller cannot create Secrets; the ExternalSecret is in SecretSyncedError. The fix is to verify the controller’s RBAC.
3. Refresh interval too long. A credential is rotated but the Kubernetes Secret is stale for the duration of the interval. The fix is a shorter interval or event-driven refresh.
4. Workload caches the value. A workload that reads the Secret at startup does not pick up rotation. The fix is to re-read or to restart the Pod.

Cross-course references

• The Linux course covers Vault’s auth methods and IRSA.
• The Observability course covers the SIEM rules for ESO refresh failures.

Quiz

Production discipline

External Secrets Operator is the right primitive for high-value credentials. A defensible ESO programme uses Vault or a cloud KMS as the source of truth, sets a refreshInterval that balances freshness and load, configures the controller’s RBAC for cluster-wide Secret management, and observes every ExternalSecret’s sync status. The Kubernetes Secret is a mirror; the external store is the source of truth. A cluster whose high-value credentials are mirrored from an external store has a Secret programme that is auditable.