No environment separation — one controller, one repo, every cluster

Environment separation is the architecture that says “this change has not yet earned the right to run in production”. The architecture is the boundary that prevents the staging change from reaching production without the explicit act of promotion.

The four failure modes of no separation

A single-controller, single-repo architecture produces four failure modes that recur at every stage.

    flowchart LR
        A["PR to staging"] --> B["Single repo"]
        B --> C["Single controller"]
        C --> D["Every cluster syncs"]
        D --> E["Production sees the change"]
        B -.->|"no boundary"| F["Repo permissions unified"]
        C -.->|"no boundary"| G["Controller credentials unified"]
        D -.->|"no boundary"| H["Drift visible everywhere"]

• Promotion-by-merge. A change is merged to main and immediately appears in every cluster.
• Repository permissions unified. The repo’s permission model is the union of every cluster’s read.
• Controller credentials unified. Broad enough for every cluster, or narrow enough for one.
• Drift visible everywhere. An out-of-band change in staging is a kubectl diff in production.

The team that has no separation pays for the simplicity of one repo and one controller with the cost of distinguishing environments by directory rather than by architecture.

Per-environment AppProjects

The first boundary is per-environment AppProjects. Argo CD’s AppProject scopes an Application to source repositories and destination clusters.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments-staging
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example/payments-staging
  destinations:
    - namespace: payments
      server: https://staging-cluster.example.com
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace

The sourceRepos field names the only repositories the AppProject’s Applications may read from. The destinations field names the only clusters and namespaces the Applications may deploy to. The boundary is in the controller’s admission path.

Per-cluster controllers

The strongest boundary is a controller per cluster. Argo CD’s application controller runs once per cluster; Flux’s controllers run once per tenant namespace.

kubectl config use-context staging
kubectl get applications -n argocd
kubectl config use-context production
kubectl get applications -n argocd

A compromise of one is not a compromise of the other. The cost is operational: more controllers to maintain.

Promotion-by-ref

The third boundary is promotion-by-ref. A team that promotes by merging to main and waiting for every cluster to sync has no promotion flow; every merge is a production deploy. A team that promotes by Git ref has a promotion flow that crosses each boundary explicitly.

    flowchart LR
        A["PR merged to main"] --> B["Staging Application syncs"]
        B --> C["Staging healthy"]
        C --> D["Update production Application spec.sourceRef"]
        D --> E["Production Application syncs"]
        E --> F["Production healthy"]
        B -.->|"fail"| G["Staging never promoted"]
        C -.->|"regress"| H["Production spec not updated"]

The promotion is the explicit update of the production Application’s spec.sourceRef to the SHA staging proved healthy.

Production discipline

1. One AppProject per environment, with sourceRepos and destinations enforced.
2. One controller per cluster.
3. Promotion by ref, not by merge.
4. Credentials are scoped per controller.

Cross-course references

• This course, Part LXXXIII (GitOpsRBAC) - AppProject roles.
• This course, Part LXXXIV (Promotion) - promotion by ref.

Quiz