Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high riskservice affecting~60 min

Runbook: Investigate a Wrong-Environment Deployment

1 · Prerequisites

Confirm every item is in place before any state change.

  • git-cicd-gitops-rb-16-roll-back-deployment
  • Access to the CI/CD system (GitHub Actions, GitLab CI, Jenkins) and GitOps controller (Argo CD, Flux)
  • Access to the affected clusters/environments
  • Knowledge of the environment topology (dev/staging/prod) and the expected routing rules

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Confirm the misrouting. The deployment went to the wrong environment: production code in staging, or staging code in production. The confirmation sources are: CI/CD run logs (which target was selected), GitOps controller sync history (which source revision was applied), and the live cluster (what is actually running)
  • · Identify the wrong-environment blast radius. Is the wrong-environment code serving real traffic? kubectl get svc -n <ns> and the ingress controller to confirm. If yes, the incident is service-affecting; if not, it is contained to a non-production environment
  • · Capture the current state of both environments BEFORE making any changes. kubectl get all -n <ns> -o yaml > /tmp/wrong-env-state.yaml and the correct environment too. The capture proves what was running and enables forensic comparison
  • · Identify the deploy initiator: the commit SHA, the workflow run ID, the Argo CD sync ID, the operator who triggered it (if manual). gh run list --workflow=<wf> --limit=5 (GitHub Actions); glab ci list --limit=5 (GitLab); argocd app history <app> --limit=5 (Argo CD). The initiator is the audit trail anchor
  • · Determine the routing rule that failed. Common causes: a workflow environment URL was wrong, a Git branch pattern matched the wrong target, a GitOps Application's destination was misconfigured, a webhook pointed to the wrong cluster. The routing rule is what needs to be fixed
  • · Confirm the fix is authorized. Stopping a deployment and rolling back are changes; the decision must be made by the incident commander or the on-call lead, not unilaterally. A wrong-environment deployment is not always an emergency — sometimes it is intentional and the misroute was a configuration mistake that can be reversed without rollback

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Stop further deploys to the wrong target. For GitHub Actions: gh workflow disable <wf> or push a no-op commit to stop pending runs. For GitLab CI: glab ci cancel --pipeline-id <id>. For Argo CD: argocd app set <app> --sync-policy none to disable automated sync. The stop is the first priority — every minute of continued deploy makes the blast radius worse
  2. 2STEP 2 - If the wrong code is serving real traffic, roll back immediately. For Kubernetes: see git-cicd-gitops-rb-16-roll-back-deployment — use the previous known-good revision. For container deployments: redeploy the previous image tag. The rollback is a service-affecting change; communicate before doing
  3. 3STEP 3 - If the wrong code is NOT serving traffic (it deployed but nothing is consuming it): the rollback can wait. Investigate first, then roll back. The investigation produces a clearer picture of what went wrong and prevents the same misroute from happening again
  4. 4STEP 4 - Determine the deploy mechanism. CI/CD push: a workflow pushed directly to the target (kubectl apply, helm upgrade, ssh deploy). GitOps pull: a GitOps controller reconciled a manifest to the target. Manual: an operator ran a command. The mechanism determines the investigation path
  5. 5STEP 5 - For CI/CD push: examine the workflow run. gh run view <run-id> --log for GitHub Actions; glab ci trace for GitLab CI; JENKINS_URL/job/<job>/<build>/console for Jenkins. Identify: the commit SHA, the triggering branch, the environment URL selected, the deploy job that ran, the credentials used, the target cluster. Every one of these is a potential misroute source
  6. 6STEP 6 - For GitOps pull: examine the controller's sync history. argocd app history <app> --limit=10 for Argo CD; flux get kustomization -A and flux events for Flux. Identify: the source revision, the manifest path, the destination cluster/namespace, the sync initiator (auto vs manual)
  7. 7STEP 7 - Compare the deploy to the expected deploy. The expected deploy is in the change ticket or the deploy log. The actual deploy is in the live cluster. The diff names the misroute: wrong commit, wrong branch, wrong image tag, wrong target cluster, wrong namespace
  8. 8STEP 8 - Identify the root cause. Common patterns: (a) environment URL variable was wrong in the workflow (typo, stale secret); (b) the Git branch pattern matched too broadly (refs/heads/* matched a feature branch); (c) the GitOps Application's source path was wrong (a directory move was not propagated); (d) a webhook pointed to the wrong cluster; (e) a human ran the wrong command in the wrong context
  9. 9STEP 9 - Fix the routing rule. For workflow environment URLs: correct the secret/variable and redeploy. For branch patterns: tighten the pattern (e.g., refs/heads/main only, or refs/heads/release/*). For GitOps source paths: update the Application CR in Git and push. For webhooks: update the webhook configuration on the Git provider
  10. 10STEP 10 - Verify the routing fix. Trigger a test deploy to each environment and confirm it goes to the correct target. Do NOT use production for the test — use a non-production environment or a dry-run flag
  11. 11STEP 11 - Re-enable deploys to the wrong target after the fix is verified. gh workflow enable <wf>; argocd app set <app> --sync-policy automated. Monitor the next several deploys to confirm the routing is correct
  12. 12STEP 12 - Record the incident. Open a ticket with: the wrong-environment target, the commit SHA that deployed, the routing rule that failed, the fix applied, the operator, the timeline, the blast radius (was real traffic affected?), the recovery time. The record is the audit trail and the basis for the post-incident review

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓The wrong-environment target no longer serves the wrong code: kubectl get pods -n <ns> -o jsonpath='{.items[*].spec.containers[*].image}' shows the expected image
  • ✓The correct environment is unaffected (or restored to expected state if it was affected): kubectl get pods -n <ns> -o jsonpath='{.items[*].spec.containers[*].image}' shows the expected image for that environment
  • ✓The CI/CD routing rule is verified correct by a test deploy to a non-production environment
  • ✓The GitOps Application's source path and destination are correct: argocd app get <app> -o yaml shows the expected path and cluster
  • ✓A workflow trigger on the correct branch deploys to the correct environment: gh run list --workflow=<wf> --limit=3 shows the expected target
  • ✓A workflow trigger on a feature branch does NOT deploy to production: the branch pattern is tightened and verified
  • ✓The fix is captured in the GitOps repo (if the fix was a manifest change) or in the CI/CD config (if the fix was a workflow change): git log --oneline -- <path> shows the fix commit

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the rollback fails or the wrong-environment code continues to serve traffic: escalate per the incident response process. The misroute is now a multi-regression (wrong code AND broken rollback). Engage the platform team
  • ↶If the routing fix causes a different misroute (the fix was applied to the wrong workflow): revert the fix immediately. The misroute is now worse. Use the workflow's history (gh run list) to identify the previous good config
  • ↶If the GitOps manifest fix cannot be pushed (the Git provider is also down): see git-cicd-gitops-rb-21-recover-git-hosting-dependency. The fix must wait for Git access
  • ↶If the rollback deployed the wrong code to the correct environment (e.g., the rollback target was also wrong): the misroute is now bidirectional. Stop everything and engage the platform team; do not attempt further automated changes
  • ↶If the wrong-environment deploy was a security-sensitive resource (secrets, IAM, network policy): the rollback is not enough. The wrong-environment code may have left credentials, opened ports, or changed access controls. Engage the security team to audit the blast radius
  • ↶If the misroute was caused by a feature flag, an A/B test, or a canary that flipped wrong: see the application's flag system. The flag is the source of truth, not the cluster. Revert the flag and the cluster follows
  • ↶If the operator who triggered the misroute is unavailable: the audit trail (commit SHA, workflow run ID) is the truth. Do not attempt to "reproduce" the misroute to understand it — read the logs instead

6 · Escalation

When the runbook isn't enough, contact:

  • · The wrong-environment code is in production and is leaking data, opening ports, or changing authentication: this is a security incident, not a deployment incident. Engage the security team immediately. The audit must capture what data was exposed, to whom, and for how long
  • · The wrong-environment deploy was caused by a compromised credential or a malicious commit: see git-cicd-gitops-rb-07-respond-to-secret-committed and git-cicd-gitops-rb-28-respond-to-supply-chain-compromise. The misroute is a symptom of a larger compromise
  • · The misroute caused a customer-facing outage (wrong-environment code is serving real traffic and failing): engage the incident response team. The fix must be coordinated with customer communication, status page updates, and stakeholder notification
  • · The fix requires changes to multiple systems (CI/CD workflow + GitOps Application + webhook): the misroute has multiple causes. Engage the platform team and the application owner; the fix is not a single change
  • · The misroute is recurring (it has happened before): the routing rule has been fixed but the underlying process (branch naming, environment URL management, change control) is still broken. Engage the engineering leadership to fix the process
  • · The wrong-environment code is still being served by a CDN cache or a downstream system that does not auto-refresh: the rollback is incomplete. Engage the platform team to purge the cache and verify the downstream system is updated

A wrong-environment deployment is a configuration mistake: the code went where it should not have, or did not go where it should have. The blast radius varies — staging code in production can be catastrophic; production code in staging is usually recoverable. The investigation identifies the routing rule that failed, fixes it, and prevents recurrence.

1. Confirm the misroute and capture state

Read-only / Safe
$ WRONG_NS="prod"
RIGHT_NS="staging"
APP="checkout-api"
echo "--- what is running in the WRONG environment? ---"
kubectl get deploy "$APP" -n "$WRONG_NS" -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}'
echo "--- what should be running in the RIGHT environment? ---"
kubectl get deploy "$APP" -n "$RIGHT_NS" -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}'
echo "--- capture state of both ---"
kubectl get all -n "$WRONG_NS" -o yaml > /tmp/wrong-env.yaml
kubectl get all -n "$RIGHT_NS" -o yaml > /tmp/right-env.yaml

The diff between the two environments is the misroute. The capture is the forensic baseline for the investigation.

2. Stop further deploys

Read-only / Safe
$ echo "--- GitHub Actions: disable the workflow ---"
gh workflow disable deploy.yml
echo "--- GitHub Actions: cancel in-progress runs ---"
gh run list --workflow=deploy.yml --limit=5
gh run cancel REPLACE_WITH_RUN_ID
echo "--- Argo CD: disable automated sync ---"
argocd app set checkout-api --sync-policy none
echo "--- Flux: suspend reconciliation ---"
flux suspend kustomization checkout-api
echo "--- verify paused ---"
argocd app get checkout-api | head -10
flux get kustomization -A | grep checkout-api

The stop is the first priority. Every minute of continued deploy makes the blast radius worse.

3. Identify the deploy mechanism

Read-only / Safe
$ APP="checkout-api"
echo "--- Argo CD history ---"
argocd app history "$APP" --limit=10
echo "--- GitOps source revision ---"
argocd app get "$APP" -o jsonpath='{.status.sync.revision}{\"\n\"}'
echo "--- CI/CD: GitHub Actions runs ---"
gh run list --workflow=deploy.yml --limit=10
echo "--- CI/CD: GitLab CI pipelines ---"
glab ci list --limit=10
echo "--- the deploy initiator ---"
gh run view REPLACE_WITH_RUN_ID --json event,headBranch,actor,createdAt

The mechanism (CI/CD push, GitOps pull, manual) determines the investigation path. The deploy initiator is the audit anchor.

4. For CI/CD push: examine the workflow run

Read-only / Safe
$ RUN_ID="1234567890"
echo "--- full run log ---"
gh run view "$RUN_ID" --log | head -100
echo "--- the trigger ---"
gh run view "$RUN_ID" --json event,headBranch,headSha,triggeringActor
echo "--- the environment selected ---"
gh run view "$RUN_ID" --json jobs | jq '.jobs[] | {name, steps: [.steps[] | select(.name | contains("env"))]}'

The full run log shows every step and the variables used. The trigger identifies the branch, commit, and actor. The environment selection is the routing decision.

5. For GitOps pull: examine the controller sync

Read-only / Safe
$ APP="checkout-api"
echo "--- Argo CD sync history ---"
argocd app history "$APP" --limit=10 -o json
echo "--- the source ---"
argocd app get "$APP" -o jsonpath='{.spec.source.repoURL} {.spec.source.path} {.spec.source.targetRevision}{\"\n\"}'
echo "--- the destination ---"
argocd app get "$APP" -o jsonpath='{.spec.destination.server} {.spec.destination.namespace}{\"\n\"}'
echo "--- live vs desired ---"
argocd app diff "$APP" | head -40

The source revision tells you which commit deployed. The destination tells you which target it went to. The diff tells you what changed.

6. Roll back if real traffic is affected

Read-only / Safe
$ APP="checkout-api"
WRONG_NS="prod"
PREVIOUS_REVISION=4
echo "--- confirm the previous revision is healthy in the correct env ---"
kubectl get deploy "$APP" -n staging -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}'
echo "--- kubectl rollout undo ---"
kubectl rollout undo deploy/"$APP" -n "$WRONG_NS" --to-revision="$PREVIOUS_REVISION"
kubectl rollout status deploy/"$APP" -n "$WRONG_NS" --timeout=300s
echo "--- verify ---"
kubectl get deploy "$APP" -n "$WRONG_NS" -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}'

See git-cicd-gitops-rb-16-roll-back-deployment for the full procedure. Communicate before the rollback — it is a service-affecting change.

7. Fix the routing rule

Read-only / Safe
$ echo "--- workflow environment URL was wrong ---"
gh variable set DEPLOY_TARGET --body "staging" --repo myorg/myrepo
echo "--- branch pattern was too broad ---"
sed -i 's|branches: [.*|branches: [main]|' .github/workflows/deploy.yml
echo "--- GitOps source path was wrong ---"
yq -i '.spec.source.path = "apps/checkout-api/v2"' applications/checkout-api.yaml
git add -A && git commit -m "fix(routing): correct checkout-api deploy target" && git push
echo "--- webhook pointed to the wrong cluster ---"
gh api repos/myorg/myrepo/hooks --jq '.[] | select(.name=="deploy-webhook") | .config.url'

Each routing rule has a different fix. Apply only the fix that addresses the root cause; do not apply multiple fixes in parallel (it confuses the post-incident review).

8. Verify the fix and re-enable deploys

Read-only / Safe
$ echo "--- test deploy to a non-production environment ---"
gh workflow run deploy.yml --ref test/fix-verification -f environment=staging
echo "--- verify it went to the right place ---"
kubectl get deploy checkout-api -n staging -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}'
echo "--- re-enable deploys ---"
gh workflow enable deploy.yml
argocd app set checkout-api --sync-policy automated
flux resume kustomization checkout-api
echo "--- monitor the next few deploys ---"
gh run list --workflow=deploy.yml --limit=5

The test deploy must be to a non-production environment. The re-enable happens only after the test confirms the routing is correct.

Verification

The wrong-environment target no longer serves the wrong code. The correct environment is unaffected or restored. The CI/CD routing rule is verified correct by a test deploy to a non-production environment. The GitOps Application”s source path and destination are correct. A workflow trigger on the correct branch deploys to the correct environment, and a workflow trigger on a feature branch does NOT deploy to production. The fix is captured in the GitOps repo or in the CI/CD config.

Rollback

If the rollback fails, escalate — the misroute is a multi-regression. If the routing fix causes a different misroute, revert the fix immediately. If the GitOps manifest fix cannot be pushed because the Git provider is down, see the Git hosting dependency runbook. If the rollback deployed the wrong code to the correct environment, stop everything. If the wrong-environment code is security-sensitive, the rollback is not enough — engage security. If the operator is unavailable, read the audit trail instead of attempting to reproduce.

References

  1. GitHub Actions — Environments
  2. GitHub Actions — Workflow Run History
  3. GitLab CI/CD — Environments
  4. Argo CD — Application History
  5. Argo CD — Sync Windows
  6. Flux — Reconcile Events