Encrypted Git workflows — encrypt the value, leave the keys elsewhere

The encrypted-at-rest class changes what is committed, not where it is reconciled. The repository still holds the manifest; the manifest still describes the desired state; the controller still reads the manifest and applies it. The difference is that the credential value in the manifest is ciphertext, decryptable only with a key that lives outside the repository.

The model

Asymmetric cryptography makes the model work. The encryption side uses the public key; the decryption side uses the private key. The public key is safe to commit; the private key is not.

flowchart LR
    A["plaintext manifest"] --> B["encrypt with public key"]
    B --> C["ciphertext committed"]
    C --> D["Git repository"]
    D --> E["GitOps controller applies"]
    E --> F["controller holds private key"]
    F --> G["decrypt on apply"]
    G --> H["Kubernetes Secret"]

Three properties follow:

• Read-only repository access is safe. A reader of the repository has the public key (or has it implicitly) and the ciphertext. They cannot derive the value.
• The private key is the trust boundary. A reader of the repository who also has the private key can decrypt. The model is only as good as the key-rotation and key-storage discipline.
• The encryption is reversible only by the controller. To rotate, the engineer re-encrypts with the public key; the ciphertext changes; the controller decrypts with the same private key.

The three key-holder options

The class has three mature implementations of the public-key holder:

• age. A modern, simple replacement for PGP. Single recipient, no key servers, no web of trust. Suited to small teams and per-cluster keys. SOPS uses age as a first-class recipient (LXXXII-05).
• PGP / GnuPG. The legacy option. Web of trust, key servers, revocation infrastructure. Heavier than age but more flexible for multi-recipient encryption and organisational trust graphs.
• Cloud KMS. AWS KMS, GCP KMS, Azure Key Vault. The private key never leaves the HSM. The controller calls Decrypt over the network; the key is never in cluster memory. The threat model that fits KMS is “the cluster itself is compromised” — KMS keeps the key out of reach even then.

The choice between them is a key-management choice: how is the private key stored, who can use it, how is rotation performed, and what happens on compromise.

What encrypted-at-rest is good for

The class fits a specific threat model:

• The repository is broad-read. Many engineers, CI runners, and forks have read access. They should not see the value.
• The cluster is narrow-trust. Only the controller and the workload identities have access to the decrypted Secret. The key holder sits behind that boundary.
• Rotation cadence matches commit cadence. A rotation that requires a code change is acceptable.

The class is weaker than external systems (LXXXII-02) when rotation must happen without a commit: a compromised credential that needs revocation in five minutes cannot wait for an engineer to re-encrypt and push. The class is weaker than the reference pattern (LXXXII-06) when the value should never be in the cluster’s view at all.

Failure modes

The class has three failure modes specific to it:

• Key loss. The private key is destroyed or made inaccessible. The cluster can no longer decrypt; every Secret in the repository must be re-encrypted with a new key and pushed. This is recoverable, but it is a cluster-wide outage for any Secret reconciled by the lost key.
• Key compromise. The private key is exfiltrated. Every ciphertext ever committed is now readable by the attacker. The response is rotation of the key and rotation of every value encrypted by the old key — a much larger rotation than the external-system case.
• Partial encryption. A manifest contains some encrypted fields and some plaintext fields. The plaintext fields leak. SOPS and Sealed Secrets both default to per-value encryption, which makes partial encryption a configuration mistake rather than a default.

Production discipline

1. The private key never enters the repository. Not as a variable, not as a file, not as a comment, not as a reference. The same rules as for any other secret apply, plus the rule that an error message quoting the key path is also a leak.
2. Key rotation has two halves. Rotating the key without rotating the values encrypted by the old key is incomplete; the old ciphertext is still readable by anyone holding the old key.
3. Encryption is per-value, not per-file. A YAML file containing both encrypted and plaintext fields is a YAML file that will eventually leak.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers disk-level encryption; the threat-model framing is the same.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers Ansible Vault; the model is conceptually identical to SOPS with a file-level key.
• Terraform for Production Sysadmins - Part XV (SensitiveVars) covers state-side encryption.

Quiz