Wrong deployment environment (staging hit prod)

Reported symptoms

●The production cluster shows resources from a commit that was only supposed to deploy to staging
●The Argo CD `Application` for the production environment shows `Synced` to a revision that matches the staging commit SHA, not the last approved production revision
●GitHub Actions workflow log shows `Deploying to environment: production` with no required-reviewer check
●The GitHub `production` environment has no protection rules configured (no required reviewers, no wait timer, no branch restriction)
●The workflow's `vars.TARGET_ENV` was set at the org level to `production` after a copy-paste from a previous incident
●The PR that triggered the deploy was not labelled for production; the workflow's branch filter allowed any branch to target any environment
●Customer-facing dashboards show the staging release version

Evidence

· `gh api repos/<org>/<repo>/environments` returns no entry for `production`, or returns `protection_rules: []`
· `kubectl get application -n argocd <app-prod> -o yaml | yq .spec.source.targetRevision` matches the staging commit SHA, not the last-known-good production SHA
· `kubectl get pods -n <prod-ns> -l app.kubernetes.io/version=<staging-version>` shows pods with the staging version label
· The Actions workflow log shows the `vars.TARGET_ENV` value at job-start: `TARGET_ENV=production`
· `gh variable list --org <org>` shows `TARGET_ENV` set to `production`
· `argocd app history <app-prod>` shows a sync event for the staging revision with no prior approval record
· The PR that triggered the deploy is labelled `area/staging` and the workflow had no label-based environment selection
· No human approved the deploy; the `production` environment's "Required reviewers" list was empty

Diagnosis and resolutionclick to reveal

Root cause

The workflow introduced an indirection — `environment: ${ vars.TARGET_ENV }` — that let a configuration error at the variable store level silently route a staging deploy to the production cluster. The indirection had no companion control: the `production` environment had no required reviewers, no wait timer, no branch restriction, and no separate workflow file. The same single workflow was used for every environment, with the only difference being the value of one variable — which means a typo, a copy-paste, or a malicious override at the variable store has the same authority as a deliberate production deploy. The structural failure is the absence of per-environment workflows with explicit, hardcoded targets, and the absence of environment-protection rules that would have caught the missing approval and stopped the deploy.

Remediation

Roll back the production cluster immediately: `argocd app rollback <app-prod>` to the last known-good revision, or `kubectl rollout undo` on the affected Deployment. Notify stakeholders, declare the incident, and pause all production deploys until the protection rules are in place. Then make the indirection impossible: split the workflow into `deploy-staging.yml` and `deploy-prod.yml`, each with a hardcoded `environment:` name and a hardcoded target cluster/kubeconfig. Configure GitHub environment protection rules for `production`: required reviewers from a designated team, a wait timer if your deployment policy allows one, a branch restriction to `main`, and a deployment branch policy that requires linear history. Audit every Argo CD `Application` to confirm the destination cluster, repo URL, and target revision are scoped per environment and cannot be reached by a workflow targeting another environment.

Verification

`kubectl get application <app-prod>` shows the last known-good production revision. The staging release is not present in the production cluster (`kubectl get pods -l app.kubernetes.io/version=<staging-version> -n <prod-ns>` is empty). `gh api repos/<org>/<repo>/environments/production` returns `protection_rules` with at least one required reviewer. A test PR targeting `production` via `vars.TARGET_ENV=production` is refused before any kubeconfig is read, because the deploy job does not exist for that combination — `deploy-staging.yml` cannot deploy to production.

Prevention

Per-environment workflows, hardcoded. No variable indirection for the environment name, the cluster, the kubeconfig, the registry, or the notification channel: each of these is a place an error can be made, and the fix is to make the error structurally impossible rather than to write a more careful workflow. Configure GitHub environment protection rules for every environment that touches production data: required reviewers, branch restrictions, and a deployment-branch policy that requires linear history and signed commits if your signing policy allows it. For Argo CD, use a separate `Application` per environment with an explicit destination server and project, and enable `syncWindows` and `prune: false` on production. The principle is "if a parameter controls where the deploy goes, the parameter is load-bearing and must be protected by more than a single variable lookup".

A single workflow deploying to multiple environments with a variable controlling the target is one typo away from staging hitting production. Split the workflows per environment, hardcode the targets, and let the GitHub environment protection rules refuse any deploy that does not have the right reviewer and the right branch.