External secret systems — Vault, cloud KMS, and the controller that bridges them

The external-system class separates the two things that GitOps otherwise forces into one place: the desired configuration (which includes a credential by name) and the credential value (which is material the trust boundary cannot release). The repository holds the former; the external store holds the latter; a controller reconciles the cluster from both.

What goes in Git

A manifest in the repository does not contain the credential value. It contains:

• A reference to the external store (a SecretStore or ClusterSecretStore resource naming Vault, AWS, Azure, or GCP).
• A reference name that identifies which entry in the store the target Secret should be populated from.
• A refresh interval and a rotation policy.

The cluster Secret resource is created by the controller, not committed by the engineer. The engineer’s commit creates the ExternalSecret resource; the controller creates the Secret.

flowchart LR
    A["Git commit: ExternalSecret manifest"] --> B["Flux / Argo: applies manifest"]
    B --> C["External Secrets Operator: reads store"]
    C --> D["Vault / AWS SM / AKV / GSM"]
    D --> C
    C --> E["Kubernetes Secret created"]
    E --> F["Pod consumes Secret"]

The cluster Secret exists only as long as the controller can reach the external store. If the store is unavailable, the Secret is either retained with the previous value or marked for refresh, depending on the configured policy.

The four major external stores

The External Secrets Operator interfaces with all four major managed stores through a provider abstraction:

• HashiCorp Vault. Dynamic secrets, lease-based rotation, PKI, transit encryption. The most operationally heavy option but the most flexible.
• AWS Secrets Manager. Native to AWS; integrates with IAM for both the controller and the workload identity. Rotation is replica-based and slower than Vault’s lease model.
• Azure Key Vault. Native to Azure; integrates with Managed Identity and Azure AD. Backed by HSMs in the Premium tier.
• Google Cloud Secret Manager. Native to GCP; integrates with Workload Identity. Versioning is per-secret and visible in the manifest.

The choice between them is rarely a security choice — all four are credible trust boundaries — and usually an operational choice: where is the workload, what identity model does the team already operate, and what rotation cadence is required.

The reconcile cycle

A typical flow:

1. The engineer commits an ExternalSecret manifest naming a store and a reference. The reference is a string — a path in Vault, a name in AWS Secrets Manager.
2. The GitOps controller (Flux or Argo CD) applies the manifest to the cluster.
3. The External Secrets Operator reconciles. It authenticates to the store using its workload identity, fetches the value, and creates a Secret whose data matches the reference.
4. On the configured refresh interval, the controller re-fetches. If the store value has rotated, the cluster Secret is updated; pods that mount the Secret see the new value on next mount or environment reload.

The GitOps commit is auditable. The store fetch is auditable in the store’s own log. The two audits together — “which manifest references which entry” and “which entry was returned at which time” — produce the same evidence a plaintext-in-Git system would, with the value never crossing the repository boundary.

Failure modes

The class has three failure modes that are not present in encrypted-at-rest or plaintext systems:

• Store outage. The controller cannot fetch. Depending on policy, the cluster Secret is either retained with the previous value, or deleted, or marked as stale. The application either continues with a stale credential or fails.
• RBAC drift. The controller’s workload identity loses access to the store. The reconcile fails with an authentication error. The cluster Secret is not refreshed.
• Reference mismatch. The manifest names a path that does not exist in the store. The reconcile fails with a not-found error. The cluster Secret is not created.

All three surface in the ExternalSecret resource’s status field. A monitoring system that watches the status field catches all three.

Production discipline

1. The store is the trust boundary, not the controller. Compromise of the controller yields a window of unauthorised fetches; compromise of the store yields the values. Harden the store first.
2. Refresh intervals are short. A rotation in the store that takes 24 hours to reach the cluster is a rotation that is not yet complete.
3. The manifest does not contain the reference’s value. A pull request that adds an ExternalSecret must not contain a test value, an example value, or a comment that quotes the store entry. Reviewers reject on sight.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers rotation patterns; the same cadence applies at the cluster boundary.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers Ansible Vault; the threat-model split is the same.
• Terraform for Production Sysadmins - Part XV (SensitiveVars) covers Terraform Cloud’s variable store; the same pattern.

Quiz