Skip to main content
RunBook Academy

← All break/fix scenarios in Git, CI/CD & GitOps

intermediategit-remote~30 min

Git hosting outage (cannot clone)

Reported symptoms

  • ●Every CI job fails at the `actions/checkout` step with `fatal: unable to access https://github.com/<org>/<repo>.git/: Could not resolve host: github.com`
  • ●Argo CD repo-server logs show `failed to checkout repo: rpc error: code = Unavailable desc = Get https://github.com/<org>/gitops/info/refs?service=git-upload-pack: dial tcp: lookup github.com on 10.0.0.2:53: no such host`
  • ●`git ls-remote https://github.com/<org>/<repo>.git` from a developer laptop returns `fatal: unable to access ... Could not resolve host`
  • ●`curl -I https://github.com` returns the same `Could not resolve host` or `Connection timed out`; `curl -I https://api.github.com` returns the same
  • ●GitHub status page (status.github.com) shows `All systems operational` or `Investigating`; if degraded, the relevant service is named
  • ●No DNS or firewall change was made by the team; the failure is on the GitHub side
  • ●Artifacts already pushed to the cluster (via previous syncs) are still running; only new clones are affected
  • ●A previously-cloned, locally-cached copy of the repo on a runner or developer laptop is reachable

Evidence

  • · `gh api /repos/<org>/<repo>` from a developer laptop returns `gh: API call failed: dial tcp ...: i/o timeout`
  • · `dig github.com +short` returns NXDOMAIN or SERVFAIL, while `dig api.github.com +short` shows the same — DNS for the GitHub domain is failing at the resolver
  • · `nslookup github.com <resolver>` returns `server can not find github.com: NXDOMAIN` from the runner VPC, while from a different network (e.g. corporate VPN) the same query returns valid A records
  • · GitHub status page (`gh status` or `curl https://www.githubstatus.com/api/v2/status.json`) returns `status: minor: Partial outage` or similar during the incident window
  • · Argo CD Application `Sync Status` shows `Unknown` and `Last Sync` is the timestamp before the outage
  • · A locally-cached clone on a developer laptop (`cd ~/work/<repo> && git fetch`) succeeds once an alternative DNS resolver or a local proxy is used
  • · The runner `/etc/resolv.conf` points at the VPC DNS resolver, which is not resolving `github.com`; the developer laptop resolver succeeds
  • · No firewall or security-group change was made in the outage window; CloudTrail (or equivalent) shows no relevant `AuthorizeSecurityGroupIngress` events
Diagnosis and resolutionclick to reveal

Root cause

Git hosting is a single point of failure for both CI (which must clone the source repo) and the GitOps controller (which must clone the GitOps repo). When GitHub is unreachable from the runner VPC — whether because GitHub itself is degraded, the DNS resolver is failing for the GitHub domain, or the egress path is broken — every clone fails and every deploy halts. The structural failure is the single-source dependency: every runner and every controller must reach GitHub over the public internet with no fallback. The remediation is not "wait for GitHub" — it is to engineer a redundant path so an outage on the primary host or the primary DNS resolver does not block deploys.

Remediation

Triage by isolating the failure layer: DNS, network, or GitHub itself. If DNS (`dig github.com`), check the VPC resolver and consider routing the runner subnet to a secondary resolver (`/etc/resolv.conf` fallback) or to a forwarder (`unbound`/`dnsmasq`) configured with Google `8.8.8.8` or Cloudflare `1.1.1.1`. If network (`curl -I`), check the egress path: a NAT gateway outage, an internet gateway detach, or a security-group rule that blocks port 443 to GitHub IPs. If GitHub itself, the only fix is a redundant Git hosting path: configure the runner `actions/checkout` to use a mirror (`github.com/<org>/<repo>` to `<ghe-mirror>`), and configure the Argo CD repo-server to fall over to a mirror via a `Repositories` CRD entry that points at the mirror. For long-term resilience, deploy a self-hosted Git (Gitea, GitLab, or a GitHub Enterprise mirror) in the same region as the runners, and have the runners prefer the mirror for clones while keeping GitHub as the source of truth for PRs and approvals. The GitOps controller can then clone from the mirror without depending on the public GitHub service.

Verification

`git ls-remote https://github.com/<org>/<repo>.git` from a runner host succeeds after the fix. `actions/checkout` in CI completes without `Could not resolve host`. Argo CD repo-server logs no longer show `failed to checkout repo`. The fallback DNS resolver is configured and tested: `dig github.com @<resolver>` returns valid A records. The Git mirror is reachable from the runner subnet, and a test clone (`git clone <mirror>`) succeeds. The Argo CD `Repositories` CRD includes the mirror entry, and a forced sync against the mirror returns `Healthy`. A drill: simulate a GitHub outage by blocking port 443 to GitHub IPs in a staging environment and verify clones succeed via the mirror.

Prevention

Git hosting is a deployment dependency. Treat it like one: have a redundant path, monitor it, and drill the failure. For CI, configure `actions/checkout` to fall back to a mirror if the primary fails, or use a self-hosted Git (Gitea, GitLab, GHE mirror) in-region and have the runners prefer it. For the GitOps controller, configure Argo CD `Repositories` to fall over to a mirror and `gitRepo` to use `git protocol` with retries. Add a synthetic monitor that pulls a known small file from the primary Git host every minute and alerts on first failure; page on >2 minutes of degraded pull. Have a local bare-repo fallback (a daily `git fetch` into a local bare repo on a runner-accessible volume) that the GitOps controller can clone from if both the primary and the mirror are down. Track RTO for Git hosting and include the Git outage scenario in the DR drill. The principle is that the Git host is the source of truth, but the source of truth is useless if it cannot be reached.

Git hosting is on the deploy path, and a deploy path with one entry point is a deploy path that can stop. A mirror, a local bare repo, and a synthetic pull monitor are the structural controls that turn a 40-minute outage into a non-event. Drill it quarterly — the next outage will not wait for the runbook to be written.