The secret reference pattern — what goes in Git, what never should

The third class of GitOps secret management starts from a different premise: the value never enters the GitOps pipeline at all. The repository holds a name. The store holds the value. The runtime fetches the value by name. The audit trail is reconstructed from three independent logs — the Git history, the store’s access log, and the cluster’s audit log — each of which records a different aspect of the same operation.

The pattern

A reference-pattern manifest names a Secret by identifier. It does not contain the value and does not contain a ciphertext. The cluster reconciles the reference into a runtime Secret that the workload can mount, by talking to the store directly.

flowchart LR
    A["Git commit: name only"] --> B["GitOps controller applies"]
    B --> C["cluster holds workload identity"]
    C --> D["fetch by name from store"]
    D --> E["store returns value"]
    E --> C
    C --> F["Kubernetes Secret materialised"]
    F --> G["Pod consumes"]
    H["Vault / AWS SM / AKV / GSM"] --> D

The repository sees only name: rds-password. The store sees the cluster’s workload identity. The cluster sees the value, briefly, during materialisation. The application sees the value, in memory or in a tmpfs mount, for the lifetime of the pod.

The three collaborators and their boundaries

The pattern has three collaborators, each with a distinct boundary:

• Repository. Owns the name and the refresh policy. Can be read by anyone with repository access. Should never contain the value, in plaintext or ciphertext, because the threat boundary is too broad.
• Store. Owns the value. Authenticates the workload identity. Logs every fetch. Holds rotation policy and rotation history. The trust boundary is the workload identity; a stolen repository credential yields nothing.
• Runtime. Owns the materialised Secret. Receives the value over a TLS channel from the store, holds it in cluster memory (or in a short-lived Secret), and presents it to the workload through the normal Kubernetes Secret mechanism.

The boundaries are clean: a compromise of one boundary yields a different blast radius than a compromise of another. A repository compromise yields names, not values. A store compromise yields values, not the names of workloads that will fetch them. A cluster compromise yields both, briefly.

When the pattern is the right answer

The reference pattern fits when:

• Rotations must not require a code change. A compliance mandate that passwords rotate every 24 hours, with no engineer in the loop, fits the pattern. The rotation happens in the store; the next reconcile cycle picks it up.
• Multi-cluster shared credentials. A credential that must be the same across dev, staging, and production clusters fits the pattern. Each cluster fetches from the same store entry; the repository has no copy to drift.
• The audit requirement is finer than the commit log. The store’s access log records every fetch; the cluster’s audit log records every materialisation; the repository’s history records every reference change. Three logs together produce an audit trail that the other classes cannot.

When the pattern is the wrong answer

The reference pattern fails when:

• Air-gapped or restricted runtime. A cluster that cannot reach the store cannot fetch. The reconcile fails. An external-system with a controller that caches the value survives a network outage; the reference pattern does not.
• The team does not yet operate a store. Adding the reference pattern on top of an empty store is a large project. Encrypted-at-rest with SOPS gets a team to GitOps-conformant secret management faster.
• The workload identity is hard to manage. Without a workload identity bound to the cluster, the store cannot authenticate the cluster, and the pattern collapses to long-lived static credentials — at which point the encrypted-at-rest class is simpler.

The failure modes

Three failure modes are specific to the reference pattern:

• Store outage. The cluster cannot fetch; the materialised Secret expires; the workload fails. The application needs to handle a cold start when the store recovers, not a steady-state failure.
• Identity drift. The workload identity bound to the cluster is rotated, removed, or restricted. The fetch fails with an authentication error. The fix is to re-bind the identity, not to rotate the value.
• Reference typo. A name in the manifest does not exist in the store. The reconcile fails with a not-found error. A monitoring system that watches the controller’s status field catches this immediately.

Production discipline

1. The repository holds names, not values. A pull request that contains a value — even a comment that quotes a store entry’s name verbatim — is rejected. The discipline is strict because the threat boundary is broad.
2. Workload identities are first-class. The store authentication is not an afterthought. A team that adopts the pattern without an identity story is adopting a weaker version of it.
3. The store is the source of truth. When the store says the value is X, the cluster says X. When the repository says the name is Y, the store is asked for Y. The repository never wins a disagreement with the store.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers rotation patterns and the discipline of “the store is the source of truth”.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers Ansible’s lookup plugins; the reference pattern is the same idea at cluster scope.
• Terraform for Production Sysadmins - Part XV (SensitiveVars) covers Terraform Cloud’s variable references; the same pattern.

Quiz