Versioned and immutable — Git as the canonical record of desired state

The second OpenGitOps principle - versioned and immutable - is what gives the controller something it can trust. A declarative description in a mutable file on a CI runner’s disk is not GitOps. A declarative description in Git with protected branches and signed tags is GitOps. The difference is the trust the controller can place in what it reads.

Two layers of immutability matter:

1. Content addressing. A commit in Git is identified by the SHA of its tree and parents. The bytes are the identity. Any change to the bytes produces a different SHA. The repository cannot claim to have two different commits at the same SHA.
2. Ref protection. A ref (branch or tag) is a movable pointer. Branch protection rules and signed tags prevent the ref from being moved to a different commit after the fact.

Content addressing protects the data; ref protection protects the pointer. Both are required: content addressing without ref protection lets an attacker rewrite history without changing the data’s identity claims, and ref protection without content addressing lets an attacker swap the contents while keeping the ref’s name.

flowchart LR
    A["Commit SHA"] -->|"identifies"| B["Tree of files"]
    C["Ref protection"] -->|"guards"| D["Branch / tag"]
    D -->|"points at"| A
    B -->|"verified by"| E["Hash check"]
    A -->|"verified by"| F["GPG / SSH signature"]

The diagram shows the two layers and the verifications the controller relies on. The ref names the commit; the commit SHA names the bytes; the signature names the author.

What Git provides for free

• Append-only history. A Git commit cannot be modified once made. To change the bytes, a new commit is required with a new SHA. The original commit survives in the object store.
• Content addressing. The SHA of a blob is the SHA of its bytes. Two commits that include the same file include the same blob, identified by the same SHA, deduplicated in the object store.
• Branches and tags. A branch is a movable ref that moves on every push. A tag is a fixed ref that points at one commit forever (unless explicitly removed and re-created).
• Diff and merge. Git can compute the diff between any two commits and merge two histories, which is what makes the pull-request review of an infrastructure change possible.

What Git does not provide

• Authentication of the author. A Git commit is just an email and a name in the header. Either can be set to anything by the client. The author is trusted only because the host (GitHub, GitLab, Gitea) authenticated the push.
• Ref protection. Git itself allows any committer with push access to force-push any branch. Branch protection is a host feature (GitHub branch protection, GitLab protected branches), enforced by the server, not by Git.
• Tamper evidence. Git can detect that the object store has changed, but only if the verifier has a known-good reference SHA to compare against. Without that reference, an attacker who can rewrite the object store can rewrite the references too.

The three missing pieces are provided by tooling on top of Git: signed commits for author authentication, branch protection for ref protection, and signed tags (or signed manifests, in the Sigstore sense) for tamper evidence.

Signed commits and signed tags

A GPG or SSH signature on a commit binds the bytes to a public-key identity. The verifier can prove that the commit was authored by the holder of the private key. For GitOps, the relevant identity is the CI pipeline’s identity or the release engineer’s identity, not the author’s - the controller cares that the bytes were approved, not who originally wrote them.

A signed tag is stronger than a signed commit for the GitOps use case because the tag is the ref the controller watches. A signature on the tag object binds the tag’s target SHA to the signing identity. Moving the tag to a different commit requires a new signature.

git tag -s v1.4.2 -m "Promote manifests to production"

The -s flag produces a GPG-signed tag (the default key is used). The signature covers the tag object, which includes the target SHA.

What the controller actually verifies

When an Argo CD or Flux controller reads a Git ref, it reads the commit SHA at that ref and the bytes at that SHA. The controller can be configured to:

• Verify the signature on the commit (Git itself supports this; git verify-commit).
• Verify the signature on the tag if the controller is pinned to a tag rather than a branch.
• Verify the SHA matches a known-good value if the controller is pinned to a specific commit (rare; pins the system to a specific deployment).

The deeper the verification, the stronger the trust. A controller pinned to a signed tag with no force-push history is the production-grade pattern; a controller pinned to a unprotected branch with unsigned commits is a placeholder.

Production discipline

1. Pin to a signed tag, not a branch. Branches move; signed tags do not. The controller’s loop is anchored to a SHA, but the SHA is reached through a ref name.
2. Disallow force-push and branch deletion on the production ref. Branch protection is a host feature; it must be configured for the production ref.
3. Sign the production ref with a release key, not an individual key. The signature answers “was this approved?”, not “did Alice write this?”. The release key is held by the CI pipeline.

Cross-course references

• Linux for Production Sysadmins - Parts XII (RepoSecurity) and XXXIV (ConfigMgmt) cover the package-management analogue of signed metadata.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the protected-branch pattern for playbooks.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the immutable state-file model that complements the immutable source-code model.

Quiz