Multi-remote failover — two remotes, automatic failover

The multi-remote failover pattern is the simplest Git-side defence against a hosting outage. A single clone carries two remotes - the canonical (the forge where pull requests and reviews happen) and the backup (a mirror in a different provider or a different region). Every push to the canonical is followed by a push to the backup. When the canonical is unreachable, the URL is flipped to the backup and pushes continue.

# Add the backup remote
git remote add backup https://backup.internal/acme/infra.git

# Verify the two remotes
git remote -v
# origin    https://github.com/acme/infra.git (fetch)
# origin    https://github.com/acme/infra.git (push)
# backup    https://backup.internal/acme/infra.git (fetch)
# backup    https://backup.internal/acme/infra.git (push)

The remote name backup is convention, not configuration; the role is determined by the URL pattern. A production script that identifies the canonical remote by URL hostname - github.com or gitlab.com - rather than by name survives a team that renames origin to canonical.

Keeping the backup current

The backup is current when every push to the canonical is followed by a push to the backup. The simplest pattern is a post-push hook:

# .git/hooks/post-push
#!/bin/sh
git push --all backup

Or, in CI, every job that pushes to the canonical runs a second push to the backup as the last step:

git push origin "$BRANCH"
git push --all backup

git push --all backup pushes every local branch to the backup. The --all flag is what distinguishes this from git push backup with no refspec, which would push only the current branch’s upstream.

flowchart LR
    A["engineer commits"] --> B["git push origin"]
    B --> C["canonical remote updated"]
    B --> D["post-push hook"]
    D --> E["git push --all backup"]
    E --> F["backup mirror updated"]
    C --> G["CI runs against canonical"]
    F --> H["CI runs against backup"]

The push-only pattern has a known limitation: the backup is never the source of a fetch unless the canonical is unreachable. The RPO is the gap between the last mirror push and the disaster time. A team that pushes to the backup only from post-push hooks has an RPO measured in seconds; a team that relies on a scheduled mirror job has an RPO measured in the schedule period.

The failover flip

When the canonical is unreachable, the failover is a single command: flip the URL of the canonical remote to the backup:

# Failover: point origin at the backup
git remote set-url origin https://backup.internal/acme/infra.git

# Verify
git remote -v
# origin    https://backup.internal/acme/infra.git (fetch)
# origin    https://backup.internal/acme/infra.git (push)

Every push from this point writes to the backup. Every CI job that reads from origin reads from the backup. The flip is visible in every log line because the URL appears in the network error message and the remote-tracking ref namespace (refs/remotes/origin/*) does not change.

The reverse - git remote set-url origin https://github.com/acme/infra.git

• is the failback. The failback runs after the vendor has recovered and the team has confirmed the recovery with at least two of the four signals from Part XCVIII-01.

A failover script

The production discipline is a script that automates the decision. The script probes the canonical, measures the elapsed time of the failure, and applies the flip when the trigger is exceeded:

#!/bin/bash
# failover.sh — flip origin to the backup when canonical is down
set -euo pipefail

CANONICAL="https://github.com/acme/infra.git"
BACKUP="https://backup.internal/acme/infra.git"
TRIGGER_SECONDS=900

elapsed=0
while [ "$elapsed" -lt "$TRIGGER_SECONDS" ]; do
  if git ls-remote "$CANONICAL" >/dev/null 2>&1; then
    echo "canonical reachable; no failover needed"
    exit 0
  fi
  sleep 30
  elapsed=$(( elapsed + 30 ))
done

echo "canonical unreachable for ${TRIGGER_SECONDS}s; failing over"
git remote set-url origin "$BACKUP"
git remote -v

The script is run by a cron job or by a CI workflow on a short schedule. The trigger threshold is the team’s RTO budget for the failover step alone - the time the team accepts between detecting the outage and switching the canonical remote to the backup.

Push-only mirror vs fetch-back mirror

The push-only mirror has an RPO measured in the gap between pushes. The fetch-back mirror - a remote that is also fetched periodically - has the opposite shape: pushes are cheap (one remote) and pulls are cheap (every remote).

A team that wants the fetch-back shape configures a third remote (or uses --mirror) and runs:

git remote add mirror https://backup.internal/acme/infra.git
git fetch --all

The mirror is fetched every night; pushes go to the canonical and the backup. When the canonical is down, the local clone already has the recent state of the mirror, and the failover flip is purely a URL change.

# Clone a mirror in one step
git clone --mirror https://github.com/acme/infra.git /srv/git/infra.git

git clone --mirror (covered in detail in Part XCVIII-03) creates a bare repository with every ref, including remote branches, refs/remotes, and notes. It is the right primitive for the read-only fallback, not for the failover clone.

Production discipline

1. Identify the canonical and the backup by URL pattern, not by remote name. A script that hard-codes origin is a script that breaks the day someone renames the remote.
2. Push to the backup on every push to the canonical. A mirror that is not current is a mirror that does not recover.
3. Automate the failover flip with a script. Manual failover is slow, error-prone, and inconsistent across engineers.
4. Keep the trigger threshold shorter than the team’s RTO. The failover step is part of the RTO; if the trigger is longer than the RTO, the failover cannot meet the design constraint.
5. Document the failback. The flip back to the canonical when the vendor recovers is the moment the team’s RTO is halved; the failback script is rehearsed as part of the drill.

Cross-course references

• Git, CI/CD & GitOps — Part XX-05 (Multiple Remotes) covers the multi-remote configuration this lesson builds on.
• Git, CI/CD & GitOps — Part XCVIII-01 (The Git Hosting Failure Scenario) names the four scopes of failure and the failover trigger.
• Git, CI/CD & GitOps — Part XCVIII-03 (Mirror Repositories and CDN) covers the read-only fallback this lesson’s mirror is paired with.

Quiz