Repository unavailable — what happens when Git is down

The first failure mode is the one that motivates the discipline: the source-of-truth repository becomes unreachable. The controller cannot pull, reconciliation stalls, and the natural assumption - “the cluster is safe because nothing new is being applied” - is wrong. Humans with kubectl, other controllers, and scheduled jobs all continue to write to the cluster, and the GitOps reconciliation has gone blind.

The symptom on the controller

The first read is the application status:

argocd app get guestbook

When the repository is unreachable, the status reports OutOfSync or Unknown and the message column carries a fetch error: rpc error: code = Unavailable for a network failure or failed to fetch: repository not found for a misconfigured URL. The canonical source is the controller log:

kubectl -n argocd logs statefulset/argocd-application-controller

The log shows repeated fetch attempts with exponential backoff. After three failed attempts Argo CD marks the application Degraded with reason ComparisonError.

flowchart LR
    A[Controller poll] --> B{Repo reachable?}
    B -- yes --> C[Fetch manifests]
    B -- no --> D[Backoff and retry]
    D --> E{Retries exhausted?}
    E -- no --> D
    E -- yes --> F["Mark Degraded: ComparisonError"]
    C --> G[Diff against live]
    G --> H[Apply if drift]

Distinguishing transient from sustained

A network blip produces a single failed poll that the next successful attempt masks. The signal to read is the gap between the last successful sync timestamp and the controller’s Degraded annotation. A gap under one minute is a blip; a gap measured in hours with the same error repeating is an outage.

For Flux, the equivalent is the GitRepository resource:

kubectl get gitrepository -A

The status.conditions field reports Ready=False with reason GitOperationFailed and the underlying transport error. Ready=Unknown means the source controller itself is not running - a separate failure mode covered in the next lesson.

What the controller does during the outage

The reconciliation loop does not stop on a fetch failure. It continues polling at its configured interval (default three minutes for Argo CD, one minute for Flux) and increments the backoff on each failure. The controller does not roll back to the last known good state, because the cluster already reflects that state and there is nothing to revert to.

A controller that reverted on a fetch failure would revert correctly-applied changes every time the network blinked. The trade-off is operator visibility, which must be recovered through alerts, not through automatic rollback.

Production discipline

1. Do not sync against a stale cache. Argo CD supports --revision to pin a commit - the correct escape hatch when the repository is down but a known-good commit exists.
2. Treat the controller as degraded, not as correct. The cluster can drift during the outage. When the repository returns, the next diff surfaces real drift.
3. Page on Degraded, not on OutOfSync. OutOfSync means “live differs from desired” and triggers a sync. Degraded means “controller cannot fulfil its contract” and is the alert.

Cross-course references

• Linux for Production Sysadmins - Parts XII (RepoSecurity) and XXXIV (ConfigMgmt) cover the apt/dnf mirror-down analogue: a stalled install, not a frozen filesystem.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover state-store unavailability, with the same shape.

Quiz