The Git hosting failure scenario — what happens when the platform is down

A Git hosting failure is the loss of the forge below the layer the repositories survive. The repositories live in clones; the forge is the service that holds the canonical remote, exposes the web UI, serves the git protocol, and runs the API. A regional outage of the forge is the moment every workflow that depends on the forge - CI triggers, pull-request reviews, signed-tag verification, GitOps controllers reading the canonical remote - stops working in the same instant, while the repositories themselves are intact on every clone.

flowchart LR
    A["Git hosting outage"] --> B["control plane lost"]
    A --> C["API lost"]
    A --> D["git protocol lost"]
    A --> E["web UI lost"]
    B --> F["CI triggers stop"]
    B --> G["webhooks stop"]
    C --> H["PR creation blocked"]
    C --> I["status checks stop"]
    D --> J["git fetch / git push fail"]
    E --> K["engineers cannot review"]

The DR question this lesson opens with is: what fails, what survives, and what signal tells you the difference?

The four scopes of failure

A forge has four scopes, and each can fail independently. The production discipline is to know which scope has failed before deciding what to do.

• Control plane. The control plane is the orchestration layer that schedules CI, fires webhooks, and runs the merge queue. A regional outage of the control plane takes CI down even if the git protocol still serves clones. The blast radius is the whole CI pipeline.
• API. The REST and GraphQL APIs back every web UI action, every CLI that authenticates via OAuth, and every CI integration. An API outage that leaves the git protocol up is rare but real; the symptom is clones that succeed while PRs cannot be opened.
• Git protocol. The smart HTTP and SSH endpoints that serve git fetch and git push. A failure here is the most direct: every push fails, every CI run that clones fails. The blast radius is every workflow that reads or writes a commit.
• Web UI. The web UI for browsing repositories, opening pull requests, and reviewing diffs. A web UI outage is the least severe; CLI workflows continue, only the human-facing layer is down.

The four scopes share infrastructure but fail independently. Engineers who treat “GitHub is down” as a single condition miss the distinction: a control-plane-only outage allows clones and pushes while breaking CI; a git-protocol-only outage allows the web UI and API while breaking clones.

What survives a hosting outage

Git is distributed. Every clone is a full backup of the repository, including the reflog, the objects, and the branches that have ever been fetched. The canonical remote is one of many copies, not the only one.

What survives a hosting outage: the repositories (every clone on every laptop and runner), the IaC that provisions the forge itself, the secrets held in the password manager, the documentation that names the recovery endpoints. What does not survive: any commit that was pushed to the canonical remote but not yet fetched into a local clone - the RPO is the gap between the last git fetch and the disaster time.

flowchart LR
    A["regional outage"] --> B["canonical remote lost"]
    A --> C["control plane lost"]
    B --> D["last clone survives"]
    B --> E["mirror survives"]
    C --> F["CI workflows stop"]
    D --> G["push to mirror"]
    E --> G
    G --> H["CI runs against mirror"]

Detection — the four signals

The failover decision is taken when at least two independent signals confirm the regional outage. A single signal is not enough; DNS can fail locally, TLS can be blocked by a corporate proxy, the status page can lag the actual recovery.

The four production signals:

1. Status page. The vendor’s status page is the first signal. The discipline is to bookmark the status page before the incident, not to search for it when the incident starts. A team that has to type github status into a search bar is a team that has not prepared.
2. DNS resolution. dig github.com and dig gitlab.com against a public resolver return the expected IP. A regional outage usually shows as NXDOMAIN or as a CNAME that no longer resolves.
3. TLS handshake. openssl s_client -connect github.com:443 -servername github.com < /dev/null returns a valid certificate. A forge behind a captive proxy or a CDN with a regional failure shows as a certificate error or a handshake timeout.
4. API latency. curl -w "%{time_total}\n" -o /dev/null -s https://api.github.com/zen returns within the expected latency. An API that returns 200 in 30 seconds is a sign of regional degradation, not a clean outage.

The failover moment

The decision is when, not whether. The team that waits for the vendor to recover before pushing to the mirror is the team whose RTO is dictated by the vendor’s recovery time. The team that decides up front - “if the canonical remote is unreachable for more than fifteen minutes, push to the mirror” - has an RTO that is the team’s, not the vendor’s.

The trigger is the elapsed time from the first failed push, not the elapsed time from the first DNS error. A DNS error at the corporate proxy is a local problem; a failed push from a runner in the cloud is a regional problem.

Production discipline

1. Two independent signals before the failover decision. A single signal is unreliable; the decision is taken when status page, DNS, TLS, and API latency agree.
2. The failover trigger is elapsed time from the first failed push, not from the first DNS error. A local DNS error is not a regional outage.
3. The mirror is the recovery path, not the vendor’s status page. The team’s RTO is the team’s, not the vendor’s.
4. Document the failover trigger and the recovery path before the incident. The runbook is rehearsed, not discovered.

Cross-course references

• Git, CI/CD & GitOps — Part XX-05 (Multiple Remotes) covers the origin plus backup pattern this lesson assumes.
• Git, CI/CD & GitOps — Part XCVII-01 (The CI/CD DR Question) names the recovery order this lesson’s failover step slots into.
• Linux for Production Sysadmins — Part XIX (DNSFailover) covers the DNS-side failure modes that produce false-positive outage signals.

Quiz