Secrets, PKI & CertificatesXIX · Production ArchitectureArchitecture
What belongs in Git, and what never does
What you'll learn
- Explain which properties of Git storage make it excellent for policy and unsuitable for secret values
- Separate the artefacts that belong in a repository from the values that must be referenced rather than stored
- State what an encrypted-secrets-in-Git approach still depends on and which protections it stops providing
- Apply a single decisive test to settle whether a repository is holding secrets
Prerequisites
Verified against OpenSSL 3.5.x teaching target; 3.0+ minimum · OpenSSH 10.x teaching target; 8.2+ minimum for certificate workflows · OpenBao 2.6.x · Smallstep step-ca 0.30.x · Certbot / Pebble Certbot current release; Pebble 2.10.x ACME test server · Kubernetes (cross-course target) 1.36.x · PostgreSQL 17.x · 2026-08-26
Version control is the best tool an infrastructure team has for describing intent, and one of the worst available for holding values. Both statements come from the same three properties: a repository replicates completely, retains permanently, and authorises at a granularity no finer than the repository itself. Design around those properties and the boundary draws itself.
What Git’s storage model guarantees, and to whom
A clone is not a view of a repository, it is a complete independent copy of its history. That is the property that makes distributed development work and the property that makes a committed value irrecoverable in the security sense. From the moment a value is committed and pushed, the number of copies is the number of clones, forks, mirrors, continuous integration caches and backup snapshots taken since, and no action on the origin server reduces that number.
Retention behaves the same way. Removing a file from the tip changes which objects the tip references; it does not remove the objects.
# The tip no longer references the file. The value is still an object.
git log --all --oneline -- config/app.env
git rev-list --all --objects -- config/app.env
Both commands answer from history rather than from the working tree, which is why they are the first things to run when someone reports that a credential was “removed”. A blob is named by the hash of its contents, so anyone holding that name can read it back for as long as the object survives, and rewriting history creates new commits without destroying the old ones.
Authorisation is the third property and the least discussed. Repository permissions are close to binary: a reader reads everything, including every historical version of every file. There is no path scoping, no per-file grant, and no way to let a build system read one value without letting it read the rest. Any design that depends on “only the platform team can see that directory” has confused a convention with a control.
flowchart LR
C["One commit\ncontaining a value"] --> O["Origin repository"]
O --> D["Developer clones"]
O --> F["Forks and mirrors"]
O --> P["Pipeline checkout cache"]
O --> B["Backup snapshots"]
D --> R["Rotation is the only\nremediation that reaches\nall of these"]
F --> R
P --> R
B --> R
The diagram is the argument in one picture. A single commit fans out into copies held by parties with their own retention, their own permissions and, in the case of a fork, their own owner. History rewriting reaches the origin and nothing else on that page, which is why rotating the value is the only remediation that terminates every branch of the fan-out at once.
What genuinely belongs
The things that belong in Git are the ones whose value comes from being reviewed, diffed, reverted and applied identically everywhere.
- Policy. Secret manager policy documents, cloud trust policies, role definitions and the capability lists that express a boundary. These are the enforcement points from the first lesson of this part, and they should never change without a diff and an approver.
- Certificate profiles. The extension sets that decide what an issued certificate is allowed to be used for.
- Automation. Renewal timers, reload hooks, issuance scripts, revocation list generation, the code that talks to the secret manager.
- The inventory. Which credential exists, which system accepts it, who owns it, how long it lives and where it is stored. Names, owners and locations, never values.
A certificate profile is the clearest example of something that is security-critical, belongs under review, and contains nothing secret at all:
[ server_cert ]
basicConstraints = critical,CA:FALSE
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:app.lab.example,DNS:www.app.lab.example
Every line there is a constraint the issuing authority will apply, and
every line is worth arguing about in review. Changing serverAuth to
add client authentication, or dropping the critical flag from basic
constraints, are changes with real consequences that deserve to be
visible. None of it helps an attacker who reads the repository.
Encrypted secrets in Git, honestly
Encrypting values before committing them is a real and legitimate pattern, and it deserves a fair account rather than a slogan. What these approaches genuinely provide is worth stating first: a change to a credential becomes a reviewable diff with an author and an approver; one artefact carries configuration and credential together, so an environment can be rebuilt from the repository alone; and the deployment path acquires no new runtime dependency, because decryption happens once at apply time rather than on every request.
Four things they still depend on, and none of them are edge cases.
- A key that is not in Git. Every scheme relocates the bootstrap problem rather than solving it. File-level encryption needs an external private key or a grant on a cloud key service; a cluster-side controller needs its own private key; transparent filters need a symmetric or personal key distributed some other way. Trace that key back and you are in the trust diagram again, usually standing on an engineer’s laptop.
- Permanent ciphertext with no forward secrecy. Every version ever committed is retained in every copy. A wrapping key compromised three years from now decrypts the value you rotated away from today, along with every intermediate version, in every fork.
- Rotation that is a rewrite, not a revocation. Re-encrypting under a new key adds a commit. It does not invalidate the old ciphertext, which is still present and still decryptable by the old key. Rotating the wrapping key achieves nothing at all until the wrapped value is also changed at the system that accepts it.
- Protection that ends at decryption. A controller-decrypted manifest becomes an ordinary platform Secret, which is base64-encoded rather than encrypted and stored unencrypted by default; anyone authorised to create a Pod in that namespace can use that access to read any Secret in it. Vault-style file encryption in a configuration management tool protects data at rest only: once decrypted the value is an ordinary variable, the tool decrypts every referenced encrypted file rather than only the ones a play needs, and encrypted variables cannot be rekeyed at all.
There is a fifth limitation that catches teams during incidents. The repository records who changed the ciphertext. It cannot record who decrypted it, because decryption happens elsewhere, and that is precisely the question an incident asks.
Use these approaches where their strengths apply: a bootstrap value that has to exist before the secret manager does, an estate that does not yet have one, or a low-rotation operator-supplied value that must survive a rebuild from Git alone. Do not use them for anything whose compromise you would describe as an incident.
The test that settles the argument
Arguments about what may go in a repository become short when they are converted into an operational question with a clock attached.
One entry catches almost every team that runs the exercise for the first time, and it is not a file anyone chose to commit. Infrastructure state records the values of the resources it manages, and marking a variable as sensitive is redaction in the command output and the user interface only, not in the stored state. Reading the outputs in machine-readable form prints them in clear regardless of the flag. State belongs in a backend with encryption explicitly enabled and access controlled per environment; it does not belong in the repository, and the fact that the file is large and boring is not a security property.
Production discipline
- Store references, never values. A repository may say which secret a service needs and where it comes from. The moment it says what the secret is, its threat model changed and nobody updated the review process.
- Put the profile under review and the key under lock. The extension set, the policy and the automation should be hard to change without an approver. The private material should be hard to read at all.
- Choose encrypted-in-Git deliberately and write down the key custody. If you cannot name where the decryption key lives and who can use it, you have not adopted a pattern, you have adopted a file format.
- Run the fourteen-hundred test on a schedule. New repositories and new contributors change the answer. A test that was empty last quarter is not evidence about this one.
Cross-course references
- Git, CI/CD & GitOps for Infrastructure Engineers - Part XXXV (Secrets in Git) covers detection and history rewriting for a credential that has already been committed, which is the recovery side of the boundary this lesson draws.
- Terraform for Production Sysadmins - Part XI (State Security and Lifecycle) covers what the state file records, why marking a variable sensitive does not change that, and how to protect the backend instead.
- Kubernetes for Production Sysadmins - Part XXI (Secrets) covers what a platform Secret actually is once a controller has decrypted a manifest into one, and who in the namespace can then read it.
Quiz
Knowledge check · 4 questions
Q1. A team re-encrypts a committed credential under a new wrapping key and merges the change. What has this achieved with respect to the original exposure?
Q2. Marking a Terraform variable as sensitive keeps its value out of the state file.
Q3. Name the three properties of Git storage that decide what may safely be committed.
Q4. Judge the claim and decide what the first hour looks like.
At 09:40 an engineer reports that a private repository was briefly made public overnight. It contains encrypted application configuration, the pipeline definitions, and a file removed from the tip in January that had held a database password. The team lead says the January file is not a concern because it was deleted, and the encrypted configuration is not a concern because it is encrypted.
Passing score: 75%. Answers are checked in this browser.