The infrastructure GitOps limitations — state, drift, secrets

The GitOps model is a strong fit for cloud infrastructure, but not a complete answer to the operational problems of running infrastructure. The four OpenGitOps principles describe the desired-state loop; they do not describe the state backend, the secrets store, the cloud provider’s drift surface, or the failure modes that arise when the loop’s assumptions break. This lesson catalogues those limits and the production discipline that bounds them.

The limits fall into four families: state loss, state corruption, drift the loop cannot detect, and secrets the loop must handle but Git cannot store. None is a counter-argument against infrastructure GitOps; they are the edges of the model the controller-shaped answer does not reach.

flowchart LR
    A["Git desired state"] --> B["Loop"]
    B --> C["State backend"]
    C --> D["Cloud"]
    B -.->|"not covered"| E["State loss"]
    B -.->|"not covered"| F["State corruption"]
    B -.->|"partial"| G["Cloud-side drift"]
    B -.->|"partial"| H["Secrets in state"]

The dashed arrows name the limits; the solid arrows name the loop. The four limits are what the loop does not, by itself, solve.

State loss

The state backend is a database. Like every database, it can be lost: an S3 bucket without versioning that is recreated by a cleanup script; a Consul cluster that loses quorum; a Kubernetes secret whose etcd backup is stale. The loop does not protect against state loss because the loop relies on the state to detect drift and compute plans.

The failure mode is invisible at first. The next plan runs against an empty state, queries the cloud for live resources, and produces a non-empty diff that says “everything will be created”. If the engineer approves, the apply attempts to re-create live resources and either fails on naming conflicts or succeeds and produces duplicates.

The production discipline is the discipline of backing up the state backend: S3 with versioning; a Consul snapshot on a schedule; a Pulumi backend with point-in-time recovery. The state backend’s last-modified timestamp is the tripwire; a sudden drop in the file size or a sudden change in the resource count is the alarm.

State corruption

State loss is recoverable from a backup. State corruption is harder because the state file exists but is wrong: a junior engineer ran terraform state rm on a resource that still exists in the cloud; a manual edit introduced a typo in a resource ID; a parallel apply from a laptop wrote a conflicting entry. The next plan produces a diff that does not match the live cloud, and the next apply attempts to “fix” the divergence by recreating or destroying resources.

The loop does not detect state corruption because the loop trusts the state file as the source of truth for “what Terraform created”. The loop cannot tell the difference between a state file that is correct and one that has been edited out of band.

The production discipline is to forbid direct edits to the state backend. All state operations - import, rm, mv, push, pull - run from CI with an audit trail; running them from a laptop is a violation of the same kind as running kubectl edit directly in a Kubernetes GitOps cluster. The tripwire is the state backend’s last-modified timestamp combined with the CI audit log.

Drift that the loop cannot detect

The loop’s reconcile cadence is set by the controller: every three minutes for Argo CD, every minute for Flux, every push for Atlantis. Anything that changes in the cloud between two plan runs is invisible to the loop until the next plan. A direct cloud-console edit, a manual CLI change, a scheduled job that mutates a security group - all of these produce drift the loop will see on the next plan, but not before.

The production discipline is to extend the loop with analysis tools that run on a separate cadence: Batfish for network configuration, Cloud Custodian for cloud policies, Steampipe for SQL-based compliance queries. The loop is the Git-to-cloud channel; the analysis tools are the cloud-to-Git tripwire. A drift the loop misses, the analysis tool catches on its next run.

Secrets in state

Terraform state contains resource attributes that are sensitive: IAM role ARNs, database endpoints, KMS key IDs, sometimes plaintext secrets if a resource is misconfigured. The state file lives in a backend the loop reads on every plan; the loop does not protect the secrets the state contains.

The production discipline is to never store secrets in state in the first place: reference them through data sources, mount them from a secret manager, or use random_password / random_string resources the apply creates and external systems consume. A state file that contains a plaintext password is a state file whose backend is now a secret store, with the corresponding controls: encryption at rest, access logs, separate identities for read and write, no public access.

flowchart LR
    A["Terraform state"] --> B["Attributes"]
    B --> C["IAM ARNs"]
    B --> D["DB endpoints"]
    B --> E["Plaintext secrets"]
    E --> F["Encrypt at rest"]
    E --> G["Access logs"]
    E --> H["No public access"]

The loop does not protect the secrets in state. The backend must.

Production discipline

1. Version the state backend. S3 versioning, Azure Blob immutable storage, Pulumi backend snapshots. Treat the state as a database.
2. Forbid direct state operations. All terraform state subcommands run from CI; running them from a laptop is a violation.
3. Extend the loop with analysis. Batfish, Cloud Custodian, Steampipe catch drift between plan ticks.
4. Treat secrets in state as a security incident. A plaintext secret in a state file is a compromised secret; rotate it, audit the backend logs, and scan CI to prevent recurrence.

Cross-course references

• This course, Part LXXXII (Secrets) - the broader GitOps secret-management model that applies here to state-file contents.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the state-backend discipline in depth.
• Kubernetes for Production Sysadmins - Parts on etcd backups cover the database-discipline analogue.

Quiz