The GitOps secret problem — why plaintext credentials cannot live in Git

A GitOps controller reads the repository and converges the cluster to what the repository declares. The repository is the desired state. The same controller that reads the deployment manifest also reads the Secret manifest — and a Kubernetes Secret manifest in YAML, by default, contains the secret value in plaintext, base64-encoded at best. The GitOps model collides with the secret-handling model at exactly this point.

The collision

A GitOps pipeline, taken literally, says: everything required to reproduce the running system is in Git. A secret-handling pipeline says: confidential material does not leave the trust boundary it was issued in. The two are in tension when the running system needs a database password, a TLS private key, or a cloud-provider credential.

flowchart LR
    A["GitOps controller reads repo"] --> B["applies manifests"]
    B --> C["Kubernetes Secret exists"]
    C --> D["Pod consumes secret"]
    A --> E["plaintext credentials in Git?"]
    E -->|yes| F["leak surface: history, forks, backups"]
    E -->|no| G["how does controller reconcile?"]

Plaintext in Git violates the threat model because the repository, by design, is replicated: every clone, every mirror, every CI runner, every backup. The deletion lesson in Part XXXV applies with full force.

The four properties to preserve

A working GitOps secret-management approach must preserve all four of:

• Auditability. The audit trail still ends at a Git commit; the change to a credential is reviewable.
• Reproducibility. A fresh cluster bootstrap can still pull everything it needs from the repository alone.
• Confidentiality. The credential value is never readable by anyone with read access to the repository.
• Revocability. Rotation can still happen without a code change, or with a controlled code change reviewed like any other.

The classical “do not commit secrets” answer preserves confidentiality by sacrificing the other three. The GitOps question is: can all four be preserved at once?

The three classes of solution

The remaining lessons in this part cover three classes of solution, each preserving the four properties by different means:

• External secret systems. The credential lives in Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager. The repository holds a reference, not a value. The cluster resolves the reference at apply time.
• Encrypted-at-rest in Git. The credential value is encrypted before commit; the repository holds ciphertext. The decryption key lives elsewhere — a KMS, a per-cluster key, or an age recipient. The controller decrypts on apply.
• The reference pattern. The repository holds neither the value nor the ciphertext; it holds a name. The cluster, the CI pipeline, and the runtime cooperate via a name-resolution layer.

None of the three is universally correct. The decision is a threat-model decision: what does the team trust, who can read the repository, who can read the cluster, and what is the cost of compromise?

Why this part exists

The rest of this part walks through each of the three classes in order: external systems (LXXXII-02), encrypted-at-rest (LXXXII-03 through LXXXII-05), and the reference pattern (LXXXII-06). The lesson at the end of the part is that the “GitOps secret problem” is solved not by a single tool but by a deliberate choice about which trust boundaries own which bytes.

Production discipline

1. The repository is the threat boundary. Anyone with read access to the repository — including read-only deploy keys, CI runners, and forks — is inside the threat boundary. Plan accordingly.
2. Choose a class, then choose a tool. External system (LXXXII-02), Sealed Secrets (LXXXII-04), SOPS (LXXXII-05), or reference pattern (LXXXII-06) — the threat model picks the class before the tool picks the implementation.
3. The wrong class is more dangerous than the right tool. A SOPS-encrypted file in a repository readable by every engineer is closer to plaintext than to a secret.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers /etc/shadow and file-permission boundaries; the same logic applies at repository scope.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers Ansible Vault; the threat model is the same as for SOPS.
• Terraform for Production Sysadmins - Part XV (SensitiveVars) covers sensitive = true and state-file exposure; the GitOps equivalent is the encrypted manifest.

Quiz