Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

critical riskcluster affecting~60 min

Runbook: Recover Git Hosting Dependency (Multi-Remote Failover)

1 · Prerequisites

Confirm every item is in place before any state change.

  • git-cicd-gitops-rb-17-troubleshoot-argocd-auth
  • A documented mirror/backup of the Git repository (read-only mirror, off-site backup, or a warm-standby secondary)
  • DNS records that can be repointed (CNAME swap or alternate A record) for the failover
  • Credentials for the mirror repository (deploy keys, PAT) pre-staged
  • For GitOps controllers: knowledge of the controller namespace and credential secrets

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Confirm the primary Git provider is genuinely down, not just slow. Check the provider's status page: GitHub https://www.githubstatus.com/, GitLab https://status.gitlab.com/, Bitbucket https://status.bitbucket.org/. Self-hosted: check the provider's health endpoint and the team's status page. The status page is the source of truth for "is it really down"
  • · Check the provider's status API for an outage indicator: curl -fsS https://www.githubstatus.com/api/v2/status.json | jq -r '.status.indicator, .status.description' or the provider's equivalent. Key the decision off .status.indicator: minor may not require a failover; major or critical does
  • · Verify the failover target is reachable and has the expected content. For a read-only mirror: git ls-remote <mirror-url>. For an off-site backup: restore the latest snapshot to a temporary clone and verify the latest commit hash matches the last known commit on the primary
  • · Identify every system that depends on the Git provider. CI/CD runners pull from Git, GitOps controllers sync from Git, code-search indexes Git, deploy keys and webhooks terminate on Git. The list determines the failover scope
  • · Identify the DNS records that need to be repointed. For CNAME: the Git provider's canonical hostname (e.g., github.com, gitlab.com) cannot be repointed by the team; the failover must use a different hostname. For self-hosted: the team's CNAME (e.g., git.internal.example.com) can be repointed
  • · Confirm the failover is authorized. A failover is a significant change — it affects every consumer of the Git provider. The decision must be made by an incident commander or the on-call lead, not unilaterally

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Declare the failover and notify all consumers. Post to the on-call channel: "GitOps failover initiated from <primary> to <mirror>. Provider status: <url>. Reason: <outage>. ETA: <minutes>. Consumers affected: <list>". The notification is the audit trail
  2. 2STEP 2 - For self-hosted Git (Gitea, GitLab self-managed): repoint the DNS. dig git.internal.example.com and update the CNAME/A record to point to the mirror host. Verify with dig +short git.internal.example.com and git ls-remote git@git.internal.example.com:<org>/<repo>.git
  3. 3STEP 3 - For managed Git (GitHub, GitLab SaaS): the canonical hostname cannot be repointed. The failover must go through every consumer. Update the GitOps controller's repo URL to the mirror URL. Argo CD: argocd repo add <mirror-url> --username <user> --password <token> then argocd app set <app> --repo <mirror-url>. Flux: edit the GitRepository.spec.url field
  4. 4STEP 4 - For CI/CD runners: update the runner's clone_url config. GitHub Actions: the actions/checkout action reads GITHUB_SERVER_URL — this is set per-repository and cannot be overridden at the runner level. The failover requires updating every workflow to use the mirror URL via with.repository: <mirror-org>/<repo> and with.ref: <branch>. GitLab CI: update GITLAB_HOST in the runner config
  5. 5STEP 5 - For deploy keys and webhooks: deploy keys are bound to a specific remote URL. The mirror must have its own deploy key, generated against the mirror's account. Add the new deploy key to the mirror repository. Update the GitOps controller's secret to reference the new key (see git-cicd-gitops-rb-17-troubleshoot-argocd-auth)
  6. 6STEP 6 - For webhooks: the webhook URL on the Git provider side points back to the CI/CD system (e.g., https://ci.internal.example.com/hooks/github). The webhook does not change during failover — it points to the CI/CD system, not to Git. What changes is what the CI/CD system pulls from. Verify the webhook fires by pushing a test commit to the mirror
  7. 7STEP 7 - Test the failover. Push a test commit to the mirror and verify every consumer receives it. CI/CD: a test workflow runs against the mirror and produces the expected output. GitOps controller: argocd app get <app> --refresh shows the new commit. Code-search: re-indexes the mirror
  8. 8STEP 8 - Operate in degraded mode. The mirror is the source of truth until the primary returns. All commits during this period go to the mirror. Document the failover window in the incident ticket. Do not push to the primary (it may be out of sync, and pushing to it will create a divergent history that must be reconciled later)
  9. 9STEP 9 - When the primary returns: verify it is fully consistent with the mirror. git fetch origin && git fetch mirror && git log --oneline origin/main..mirror/main shows the commits made during the failover. The primary may have lost those commits (e.g., a database corruption on the primary); reconcile by force-pushing the mirror to the primary, OR by cherry-picking the missed commits
  10. 10STEP 10 - Repoint consumers back to the primary. Reverse STEP 3, STEP 4, STEP 5. Update every consumer's URL/key/secret back to the primary. Verify with a test push. The failover is complete; the primary is the source of truth again
  11. 11STEP 11 - Record the failover. Open an incident ticket: provider, time of failover, time of recovery, consumers affected, commits made during the failover, who authorized the failover, who executed it, the reconciliation performed after recovery. The record is the audit trail
  12. 12STEP 12 - Post-incident: review the failover process. Were the mirror credentials available? Was the DNS swap fast enough? Were all consumers covered? The review produces a list of improvements (pre-staged secrets, automated DNS failover, multi-remote Argo CD)

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓git ls-remote <mirror-url> returns the expected refs from the failover target
  • ✓CI/CD runners pull successfully from the failover target and produce the expected output
  • ✓GitOps controllers (Argo CD, Flux) reconcile against the failover target: argocd app list shows Synced + Healthy; flux get kustomization -A shows Ready=True
  • ✓A test commit pushed to the mirror is received by every consumer (CI/CD, GitOps, code-search)
  • ✓After primary recovery: git fetch origin && git log --oneline origin/main..mirror/main shows the commits made during the failover are accounted for
  • ✓DNS resolves to the correct target (mirror during failover, primary after recovery): dig +short <git-hostname>
  • ✓Deploy keys and webhooks function against the active target

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the failover fails (mirror is unreachable, credentials are wrong, DNS does not propagate): the original Git provider is still down but the failover is also down. Escalate per the incident response process; the failover is blocked until the underlying issues are fixed
  • ↶If consumers cannot be repointed because they hardcode the primary URL (e.g., a CI/CD workflow with actions/checkout and no override): the failover requires editing every workflow. This is a code change that must be reviewed and merged. Use the GitOps repo for the workflows and let the controller propagate the change
  • ↶If the primary recovers during the failover but the mirror has divergent commits: do NOT force-push the mirror to the primary without a review. The divergent commits may include in-flight changes that the primary recovered but the mirror did not. Cherry-pick or merge carefully
  • ↶If the failover reveals that the mirror is not up to date (the mirror sync is broken): the failover is operating on stale data. Some commits made on the primary during the failover window are lost on the mirror. Re-pull from the primary (if reachable) or restore from the off-site backup. The mirror sync should be fixed after the failover
  • ↶If DNS repointing fails (TTL is too high, the record is locked, the provider does not allow it): the failover requires updating every consumer's URL directly. This is a longer process; expect hours of degraded operation
  • ↶If the failover causes a GitOps controller to lose its credential (the controller's secret references the primary's deploy key, not the mirror's): see git-cicd-gitops-rb-17-troubleshoot-argocd-auth. The failover must include rotating the controller's secret to the mirror's deploy key

6 · Escalation

When the runbook isn't enough, contact:

  • · The Git provider outage is also affecting the team's identity provider (SSO, OIDC) and the team's CI/CD infrastructure is single-factor authenticated against the same provider: the outage is multi-system. Engage the security team and the platform team. Do not attempt "creative" workarounds (e.g., using a developer's personal token for CI/CD)
  • · The mirror is unavailable because it depends on the same provider (the mirror is itself on GitHub, GitHub is down): the failover has no failover target. The team is offline. Engage the incident commander and the leadership team. Document the outage and prepare a public-facing status update
  • · The failover forces a force-push on the primary to reconcile: the force-push is a destructive operation (see git-cicd-gitops-rb-06-respond-to-force-push-incident). Engage the security and audit teams before the force-push. The audit trail of the force-push must be captured before, during, and after
  • · The outage lasts longer than the team's recovery time objective (RTO): the team cannot deploy for an extended period. Engage the leadership team; the business impact must be communicated. The failover is partial — operations continue in degraded mode, but new feature deployment is paused
  • · The failover is required for a planned maintenance window (e.g., the provider's database migration): this is a planned failover, not an emergency. Use the same procedure but with a pre-staged change ticket, a pre-tested rollback, and a pre-communicated timeline
  • · The provider is up but the team's access is blocked (suspension, billing issue, security hold): the failover cannot be triggered by a status page check. Engage the account team or the support team. A suspension may take days to resolve; the failover is the only way to continue operations

When the Git hosting provider is down, every system that depends on Git stops: CI/CD cannot clone, GitOps controllers cannot reconcile, deploy keys cannot authenticate. The failover is to a mirror or off-site backup — every consumer must be repointed, and when the primary returns, every consumer must be repointed back.

1. Confirm the outage and check the failover target

Read-only / Safe
$ echo "--- provider status ---"
curl -fsS https://www.githubstatus.com/api/v2/status.json | jq '.status.indicator,.status.description'
echo "--- can we reach the primary? ---"
git ls-remote https://github.com/myorg/myrepo 2>&1 | head
echo "--- can we reach the mirror? ---"
MIRROR="git@git-mirror.internal.example.com:myorg/myrepo.git"
git ls-remote "$MIRROR" 2>&1 | head
echo "--- is the mirror up to date? ---"
LAST_COMMIT_PRIMARY=$(git ls-remote https://github.com/myorg/myrepo HEAD | cut -f1)
LAST_COMMIT_MIRROR=$(git ls-remote "$MIRROR" HEAD | cut -f1)
[ "$LAST_COMMIT_PRIMARY" = "$LAST_COMMIT_MIRROR" ] && echo "MIRROR IN SYNC" || echo "MIRROR OUT OF SYNC - investigate"

The status page and the actual reachability tests are independent confirmations. The mirror”s last commit must match the primary”s last commit (or be a known-acceptable divergence).

2. Repoint DNS (self-hosted Git only)

Read-only / Safe
$ GIT_HOSTNAME="git.internal.example.com"
MIRROR_IP="10.20.30.40"
echo "--- current record ---"
dig +short "$GIT_HOSTNAME"
echo "--- update the DNS record ---"
aws route53 change-resource-record-sets --hosted-zone-id Z123EXAMPLE --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"'$GIT_HOSTNAME'","Type":"A","TTL":60,"ResourceRecords":[{"Value":"'$MIRROR_IP'"}]}}]}'
echo "--- verify propagation ---"
sleep 30
dig +short "$GIT_HOSTNAME"
echo "--- test clone via DNS ---"
git ls-remote "git@$GIT_HOSTNAME:myorg/myrepo.git" 2>&1 | head

DNS repointing only works for self-hosted Git. For managed Git, the canonical hostname cannot be changed by the team; the failover must go through every consumer.

3. Repoint the GitOps controller (Argo CD)

Read-only / Safe
$ MIRROR_URL="https://git-mirror.internal.example.com/myorg/gitops-prod"
USERNAME="myorg-gitops-bot"
TOKEN="REPLACE_WITH_MIRROR_TOKEN"
echo "--- add the mirror as a repo ---"
argocd repo add "$MIRROR_URL" --username "$USERNAME" --password "$TOKEN" --upsert
echo "--- repoint every Application ---"
for app in $(argocd app list -o name); do
argocd app set "$app" --repo "$MIRROR_URL"
done
echo "--- refresh ---"
argocd app list -o wide | head
for app in $(argocd app list -o name); do
argocd app get "$app" --refresh
done

Argo CD supports multiple repositories. The mirror is added with --upsert (overwrites any existing repo with the same URL). Every Application is repointed with argocd app set --repo.

4. Repoint the GitOps controller (Flux)

Read-only / Safe
$ NS="flux-system"
MIRROR_URL="https://git-mirror.internal.example.com/myorg/gitops-prod"
kubectl get gitrepositories -A -o yaml | yq '.items[] | .metadata.name + " " + .spec.url'
echo "--- patch each GitRepository ---"
for gr in $(kubectl get gitrepositories -A -o name); do
kubectl patch "$gr" -n "$NS" --type merge -p "{\"spec\":{\"url\":\"$MIRROR_URL\"}}"
done
echo "--- force reconcile ---"
flux reconcile source git flux-system
flux get kustomization -A

Flux GitRepository CRs reference the URL in spec.url. Patching the field repoints the controller. The flux reconcile source git forces a refresh against the new URL.

5. Repoint CI/CD runners

Read-only / Safe
$ echo "--- GitHub Actions: update workflow checkout ---"
sed -i 's|uses: actions/checkout@.*|uses: actions/checkout@v4
        with:
          repository: myorg/myrepo
          ref: main
          ssh-key: ${{ secrets.MIRROR_SSH_KEY }}
          server-url: git@git-mirror.internal.example.com|' .github/workflows/*.yml
echo "--- or via environment variable ---"
echo "GIT_MIRROR_URL=git@git-mirror.internal.example.com" >> .github/workflows/ci.yml
echo "--- GitLab CI: update the runner config ---"
sed -i 's|https://gitlab.com|https://gitlab-mirror.internal.example.com|' /etc/gitlab-runner/config.toml
gitlab-runner restart
echo "--- test ---"
git clone "$MIRROR_URL" /tmp/test-clone && rm -rf /tmp/test-clone

CI/CD runners clone via the URL in the checkout action (GitHub) or the runner config (GitLab). The failover requires updating the URL in every workflow and every runner config.

6. Operate in degraded mode and reconcile on recovery

Read-only / Safe
$ PRIMARY="https://github.com/myorg/myrepo"
MIRROR="git@git-mirror.internal.example.com:myorg/myrepo.git"
echo "--- all commits during the failover go to the mirror ---"
git remote set-url origin "$MIRROR"
git push origin main
echo "--- when the primary recovers, verify consistency ---"
git fetch "$PRIMARY"
git log --oneline "$PRIMARY/main..main"
echo "--- reconcile: push mirror to primary ---"
git push "$PRIMARY" main --force-with-lease
echo "--- then repoint consumers back ---"
git remote set-url origin "$PRIMARY"

During the failover, the mirror is the source of truth. When the primary recovers, verify it is consistent and repoint every consumer back. The git push --force-with-lease reconciles the primary only after the consistency check.

Verification

git ls-remote <mirror-url> returns the expected refs. CI/CD runners pull successfully and produce the expected output. GitOps controllers reconcile against the failover target (Synced + Healthy). A test commit pushed to the mirror is received by every consumer. After primary recovery, git fetch origin && git log shows the commits made during the failover are accounted for. DNS resolves to the correct target. Deploy keys and webhooks function against the active target.

Rollback

If the failover fails, escalate per the incident response process — the failover is blocked until the underlying issues are fixed. If consumers cannot be repointed because they hardcode the primary URL, the failover requires editing every workflow via the GitOps repo and letting the controller propagate the change. If the primary recovers with divergent commits, do not force-push without a review — cherry-pick or merge carefully. If the mirror is not up to date, re-pull from the primary or restore from the off-site backup. If DNS repointing fails (TTL too high, record locked), update every consumer”s URL directly. If the failover causes the GitOps controller to lose its credential, rotate the controller”s secret to the mirror”s deploy key.

References

  1. GitHub Status
  2. GitLab Status
  3. Bitbucket Status
  4. Argo CD — Multiple Sources for an Application
  5. Flux — GitRepository Mirror
  6. GitHub — Mirroring Repositories
  7. Gitea — Mirror Functionality