GitOps controller points at wrong environment (sync target swapped)

Reported symptoms

●Staging cluster resources show production-tier labels: `kubectl get deploy -n <staging-ns> -o jsonpath={.spec.template.metadata.labels.environment}` returns `production`
●`argocd app list -n argocd` shows every Application pointing at `https://github.com/<org>/gitops-prod` instead of `https://github.com/<org>/gitops-staging`
●The staging cluster `Secret` objects contain production-tier credentials (`kubectl get secrets -n <staging-ns>` shows secrets named `<svc>-prod-*` that reference the production vault)
●Customer-impacting features were activated in staging because the production feature-flag ConfigMap was synced; staging tests passed on the wrong configuration
●The change was made via `kubectl edit application -n argocd` on the staging controller, not via a PR to the staging GitOps repo
●The production controller `repoURL` was unaffected; production is healthy
●The Argo CD notification controller sent a `Sync succeeded` alert to the `#deploys-production` Slack channel for a staging sync (cross-channel noise)
●`argocd app manifests <app>` returns the production manifest tree, including production-tier `external-secrets` references that resolve to the production `SecretStore`

Evidence

· `kubectl get application -n argocd -o custom-columns=NAME:.metadata.name,REPO:.spec.source.repoURL` lists `https://github.com/<org>/gitops-prod` for every Application
· `kubectl get cm -n argocd argocd-cm -o yaml` shows `repositories:` with a `url: https://github.com/<org>/gitops-prod` entry
· `kubectl get secrets -n argocd` includes a Secret whose `metadata.annotations.argocd.argoproj.io/secret-type` is `repository` and whose `data.url` (base64-decoded) is `https://github.com/<org>/gitops-prod`
· The kube audit log shows `verb=patch, user.username=<teammate>, object=Application/argocd/<app>, requestPatch:{"spec":{"source":{"repoURL":"https://github.com/<org>/gitops-prod"}}`
· The production controller Argo CD has no equivalent audit entry; production is unaffected
· `kubectl get secretstore -A` in the staging cluster references `vault.prod.example.com`; the staging `SecretStore` should reference `vault.staging.example.com`
· `argocd app history <app>` on staging shows a sync to the production repo HEAD, with `revision: <production-sha>` and `pruning: true` — staging resources from the previous repo were pruned, production resources were created
· The teammate terminal history shows the edit ran from a context that was pointing at the staging cluster but the copy-pasted URL came from the production repo browser tab

Diagnosis and resolutionclick to reveal

Root cause

The staging GitOps controller was configured with a `repoURL` pointing at the production GitOps repo. Two structural failures enabled the swap. First, the staging controller and the production controller shared an authentication context: the teammate had `kubectl` access to both clusters from the same terminal, and the copy-paste of the repo URL was uncontrolled. Second, the staging controller had broad write access (the `kubectl edit application` succeeded without any change-control gate) — GitOps is supposed to put the desired state in git, but the staging controller `Application` resources were being edited in-cluster rather than via the staging GitOps repo, so the controller itself was drifting from its own desired state. The result is that the staging cluster now mirrors the production GitOps repo, with all the cross-environment consequences (production secrets, production feature flags, production tier references). The fix is to re-establish environment isolation at every layer.

Remediation

Stop the bleeding first: pause Argo CD on the staging cluster to prevent further syncs (`kubectl patch application -n argocd --type merge -p ''{"spec":{"syncPolicy":{"automated":null}}'' --selector app=*` or `argocd app sync <app> --disable-automated` per Application). Identify the production-tier resources that have leaked into staging and remove them: delete the affected Application resources after verifying they are not in active use. Restore the staging controller `repoURL` to the staging GitOps repo via the staging GitOps repo itself: open a PR that reverts the Application manifests to point at the staging repo, get the required review, merge, and let Argo CD reconcile. Re-enable automated sync after the cluster matches git. Then audit the RBAC: the staging controller should not be able to read the production `SecretStore`, and the production controller should not be able to read the staging `SecretStore`. Restrict the staging controller `ServiceAccount` to staging-tier credentials only, and add an admission policy that rejects any `SecretStore` or `ExternalSecret` in a staging namespace that references a production-tier vault path.

Verification

`kubectl get application -n argocd -o custom-columns=NAME:.metadata.name,REPO:.spec.source.repoURL` lists `https://github.com/<org>/gitops-staging` for every Application. `argocd app list` shows `Sync Status: Synced` and `Health Status: Healthy` for every staging Application. The staging cluster secrets reference staging-tier vault paths: `kubectl get secretstore -n external-secrets -o jsonpath={.spec.provider.vault.server}` returns `https://vault.staging.example.com`. The production cluster is unaffected: production Applications still point at the production repo, and production secrets reference the production vault. The kube audit log for the staging controller shows the patch was reverted by a subsequent PR merge; no further direct edits to the controller Applications are recorded.

Prevention

Environment isolation is structural. Staging and production GitOps repos should be distinct repositories (or distinct branches with separate controllers), and the staging controller credentials should grant access only to the staging repo. Restrict the controller ServiceAccount IAM to a role that can read only the staging repo and the staging registry; the production controller gets a separate IAM role for the production repo. Add an admission policy (Kyverno / OPA) that rejects any `Application` in the staging Argo CD whose `spec.source.repoURL` does not match an allowlist of staging repos. For the controller-side, ensure `Application` resources are managed in the staging GitOps repo and reconciled by the staging controller, not edited in-cluster — a direct `kubectl edit application` on a controller whose Applications are managed in git is a drift event that must trigger an alert. Use separate kube contexts (`.kube/config` with cluster-specific contexts) and require a deliberate context switch (`kubectx <staging>`) before any controller edit. The principle is that a GitOps controller that can read both repos is a GitOps controller that can serve the wrong one.

A GitOps controller that can read both repos is a controller that can serve the wrong one. Restrict each controller to its environment repo, its environment registry, and its environment secret store. Reject cross-repo repoURL values at admission, manage controller Application resources in their own GitOps repo, and require a deliberate context switch before any controller edit. The fix is structural, not procedural — the controller that can serve the wrong environment is the controller that will.