Missing production approval (auto-deploy skipped required reviewers)

Reported symptoms

●The workflow log shows `Deploying to environment: production` with no preceding approval gate
●The GitHub `production` environment "Required reviewers" list was modified recently (or had been emptied) by an admin who added a bypass actor
●The production deploy ran in seconds, not minutes: there is no `Waiting for reviewer` event in the workflow log
●`gh api repos/<org>/<repo>/environments/production` returns `protection_rules` whose `required_reviewers` list is empty, or whose `reviewers` block has been replaced by a `bypass_actors` entry
●`gh api repos/<org>/<repo>/environments/production/secrets` returns production-tier secrets that the workflow can read without any approval step
●The PR that triggered the deploy was opened by a service account or by a user that owns the bypass actor; the deploy job ran on push to `main`
●No notification (Slack, email) was sent because the approval gate was skipped
●Customer-facing dashboards show a version that was not approved by anyone outside the merging team

Evidence

· `gh api repos/<org>/<repo>/environments/production` returns JSON with `protection_rules: required_reviewers empty, bypass_actors with actor_id`
· `gh api repos/<org>/<repo>/environments/production` audit log shows a `modify` event by `<admin-username>` two days ago that emptied the reviewers list and added the bypass actor
· The workflow log shows the `production` job started immediately after the build job; there is no `hold` step and no `Waiting for reviewer to approve` message
· `gh api repos/<org>/<repo>/actions/runs/<run-id>` returns `event: push`, `head_branch: main`, `conclusion: success`, `created_at` approximately equal to `updated_at` (no human review window)
· GitHub audit log shows the production deploy event `environment: production, environment_action: deploy, approved_by: null`
· The team PR template requires two reviewers for any change to `production` overlay manifests; the PR was merged with one reviewer
· `argocd app history <app>` shows the production Application synced to the new commit immediately after the workflow run; no `SyncWindow` was active
· The Slack notification rule for production deploys fired `succeeded`, not `awaiting_approval`, confirming the gate was bypassed

Diagnosis and resolutionclick to reveal

Root cause

GitHub environment protection rules are the deploy-time approval gate: required reviewers must approve before the job can access environment secrets and run. The structural failure is twofold. First, the production environment `required_reviewers` list was emptied by an admin who believed a bypass actor was a safer pattern (it is not — a bypass actor is an opt-out from approval, not a substitute for it). Second, the workflow did not include its own approval gate as defence-in-depth; it relied entirely on the environment rule. The result is a deploy that reached production without a human reviewer, on a commit that was reviewed only by the merging author. The bypass actor had been added for "emergency break-glass" use cases, but the gate was not reinstated after each use, so the bypass became the default.

Remediation

Roll back the production deploy to the previous approved SHA immediately: `argocd app rollback <app>` or a manifest revert in the GitOps repo, with the revert PR itself going through the normal review process. Restore the required reviewers list via the GitHub API for the production environment, specifying the reviewers (typically two from the SRE on-call rotation), the wait timer (if your policy allows one), the branch restriction (`main` only), and removes or restricts the bypass actor. A bypass actor should be allowlisted by role (e.g. only the Security Incident Response team), set to `bypass_mode: pull_request` (so it only bypasses for PRs from that team, not for arbitrary users), and timestamped with a `bypass_until` so it expires automatically. Add a CI verifier on every PR that opens a workflow targeting production: the workflow itself must include an `environment` declaration and a step that calls `gh api repos/<org>/<repo>/deployments/...` to confirm a reviewer approved before any kubeconfig is read. Treat the environment protection rule as a backstop, not as the only control — every workflow that deploys to production should have its own approval gate.

Verification

`gh api repos/<org>/<repo>/environments/production` returns `required_reviewers` with at least two named users or teams, and `bypass_actors` is empty or restricted to a named Security IR team with `bypass_mode: pull_request`. A test PR that opens a workflow targeting production via `environment: production` enters `Waiting for reviewer to approve` in the workflow log; no reviewer approves, and the deploy job does not start. After one reviewer approves, the deploy job starts. The rollback returns the cluster to the previous approved SHA, and the Application is `Synced` and `Healthy`. The bypass actor addition is recorded in the audit log, and a follow-up runbook item exists to review and remove bypass actors at the end of every incident.

Prevention

Required reviewers are a backstop, not a substitute for a workflow-level approval gate. Every workflow that targets production must declare `environment: production`, and the environment must require at least two reviewers from a designated team, with a wait timer if your deployment policy permits one. Bypass actors must be exceptional, not routine: restrict them to a named incident-response team, set them to `bypass_mode: pull_request` so they apply only to PRs from that team, and give them an automatic expiry (`bypass_until`). Alert on every bypass actor addition and every bypass actor use, with a daily review of bypass actors that have not expired. Add a CI verifier that fails any PR that modifies the `production` environment protection rules unless the PR is authored by a designated admin and reviewed by a designated auditor. The principle is that bypass is the exception, and the structural controls must make the exception expensive enough that it is used only when warranted.

Required reviewers are a load-bearing control, and a bypass actor is an opt-out, not a substitute. Empty the reviewers list or leave a bypass actor enabled past its incident, and the next push to main will reach production without a human reviewer. Restrict bypass actors to a named team, give them automatic expiry, and alert on every use. The approval gate must exist in two places: the environment protection rule, and the workflow itself.