Environment boundaries — dev, staging, and production as separate concerns

Ask where the production boundary is in a GitOps estate and the usual answer points at a directory: clusters/production. That is a label. A boundary is a place where an attempt to cross is refused by something that does not read the label — an allowlist that rejects the destination, a service account that lacks the verb, an API server that never had the credential. This lesson is about the difference, because almost every estate has the labels and rather fewer have the refusals.

Four axes, increasing strength

An environment separation can be drawn on four axes, and they are not equivalent.

flowchart LR
    A["Axis 1: repository path or branch"] --> B["Axis 2: controller scope"]
    B --> C["Axis 3: reconciling identity"]
    C --> D["Axis 4: separate cluster"]
    A -.-> A1["Organises, does not refuse"]
    B -.-> B1["Refuses wrong source or destination"]
    C -.-> C1["Refuses wrong verbs, at the API server"]
    D -.-> D1["Refuses because no credential exists"]

• Repository path or branch. Cheap, readable, and enforcing nothing on its own. A path becomes part of a boundary only when a CODEOWNERS entry and a controller scope both reference it.
• Controller scope. An Argo CD AppProject with a destination allowlist, or a Flux Kustomization confined to a namespace, refuses a manifest that names the wrong target. This is the first axis that says no.
• Reconciling identity. A production service account that no non-production path can impersonate makes the refusal an API server decision rather than a controller decision.
• Separate cluster. The strongest and most expensive: the non-production control plane has no credential for the production API server, so there is no request to refuse.

argocd proj create platform-prod \
  -d https://prod-api.example.com:6443,platform \
  -s https://github.com/example/platform-gitops

kubectl apply -f rbac.yaml

Most production estates should sit at axis three at minimum. Axis four is warranted where the failure model includes control-plane loss or where a compliance regime requires that a non-production credential cannot exist for production.

What each model actually buys

A shared cluster with namespace separation gives you cost efficiency and one upgrade to run. It does not give you blast-radius separation for anything cluster-scoped: one CRD, one admission webhook, one exhausted node pool, one etcd, affecting every environment on it. Testing an upgrade of a cluster component in staging is meaningless when staging shares the component with production.

Separate clusters give real isolation for control-plane failure, CRD and webhook changes, node and network exhaustion, and credential compromise. The price is duplicated platform work, N upgrades, and a strong pull toward drift between environments — which is precisely the problem GitOps is good at, so the cost is lower here than it is for a team managing clusters by hand.

The single sanctioned crossing

State has to move between environments or nothing ever ships. The discipline is that exactly one mechanism crosses, and it moves an immutable artifact reference rather than a rendered result.

• What crosses: an image digest, a chart version, a manifest revision. Content that has been built once and is identical in both places.
• What does not cross: rendered manifests, secrets, credentials, reconciling identities, and cluster credentials. Each environment renders its own overlay and holds its own secret material.
• How it crosses: a commit into the target environment path, reviewed under that path rules, reconciled by that environment controller and identity.

A promotion that works by copying a rendered output from staging into production is not a crossing of the boundary — it is proof there was not one, because the production environment did not decide anything.

Reviewing an existing separation

Four questions, one per axis, answerable from configuration:

1. Which paths reconcile into production, and who owns them in CODEOWNERS?
2. Which project or Kustomization scope names a production destination, and is that list finite?
3. Which service account applies in production, and can any non-production path cause it to be used?
4. Does the non-production control plane hold a credential for the production API server at all?

An estate that answers all four crisply has a boundary. An estate that answers the first and hesitates on the rest has a naming convention.

Production discipline

1. Production has its own reconciling identity, without exception. Shared identities across environments make the other axes cosmetic.
2. Destination allowlists are finite and reviewed. A wildcard destination on any project is a finding, not a convenience.
3. Only digests and versions are promoted. Rendered output never moves between environments.
4. Boundary tests are scheduled. A quarterly deliberate violation, expected to fail, with the failure recorded as evidence.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part LVI (DeployEnvs) covers environment modelling in push pipelines, Part LXXXIV (EnvPromotion) covers the promotion machinery in depth, and Part XCII (ProtectedEnvs) covers the approval side.
• Kubernetes for Production Sysadmins - Part CI (ClusterBoundaries) covers when a separate cluster is the right answer and what it costs.
• Terraform for Production Sysadmins - Part XV (Environment-Architecture) covers the same separation where the state backend and the cloud role are the boundary.

Quiz