Skip to main content
RunBook Academy

← All break/fix scenarios in Git, CI/CD & GitOps

intermediategitops-repo~30 min

GitOps repository authentication failure (token rotated)

Reported symptoms

  • ●Every Argo CD `Application` shows `Sync Status: Unknown` and `Last Sync: <stale timestamp>`
  • ●The Argo CD controller (or repo-server) log shows `Failed to get client for repo: rpc error: code = Unauthenticated desc = Authentication required`
  • ●`argocd app list` reports `repo: https://github.com/&lt;org&gt;/gitops.git` and `status: Unknown` for every Application
  • ●The git repository itself is healthy — `git ls-remote https://github.com/&lt;org&gt;/gitops.git` from a developer laptop works
  • ●The old SSH key was removed from the GitHub repo's deploy keys page; the new key was added but never given to Argo CD
  • ●`kubectl get secret -n argocd repo-creds-&lt;id&gt; -o yaml` shows base64 for the old key, not the new one
  • ●The rotation was performed under a credential-rotation Jira ticket; the rotation procedure listed "rotate key" but not "rotate key in Argo CD"
  • ●A test sync from the Argo CD UI fails with the same `Unauthenticated` error

Evidence

  • · `argocd repo list` returns `https://github.com/&lt;org&gt;/gitops.git Unknown Failed: rpc error: code = Unauthenticated desc = Authentication required`
  • · `kubectl logs -n argocd deploy/argocd-repo-server --since 30m` shows multiple `Authentication required` lines with the GitHub host and the now-deleted key fingerprint
  • · `kubectl get applications -n argocd -o custom-columns=NAME:.metadata.name,SOURCE:.spec.source.repoURL,STATUS:.status.sync.status` shows every Application in `Unknown`
  • · `curl -sS https://api.github.com/repos/&lt;org&gt;/gitops/keys` shows one key, with a fingerprint that does not match `ssh-keygen -lf <(kubectl get secret ...)`
  • · `kubectl get secret -n argocd argocd-repo-server-tls -o yaml` shows the secret was last modified before the rotation window
  • · GitHub's audit log shows a `public_key.remove` event for the old key fingerprint, and a `public_key.create` for the new fingerprint, but no secret update in the argocd namespace
  • · `kubectl exec -n argocd deploy/argocd-repo-server -- ssh -o StrictHostKeyChecking=accept-new -T git@github.com` exits with `Permission denied (publickey)`
Diagnosis and resolutionclick to reveal

Root cause

Argo CD''s repo-server holds the credential it uses to clone the GitOps repository as a Kubernetes Secret in the `argocd` namespace (`argocd-repo-server` mounts it, or the `Repositories` CRD references it). The credential is not auto-discovered from GitHub; it is a static secret the operator configured. When the GitHub deploy key was rotated, the new public half was added to the GitHub repo and the old public half was removed, but the corresponding private half in the Argo CD namespace was never updated. The result is that Argo CD presents the old private key, GitHub no longer recognises the old public key, every clone attempt fails authentication, and every Application that depends on this repo goes to `Unknown`. The structural failure is the absence of a rotation procedure that covers both halves of the keypair and a verifier that proves the controller can still clone after rotation.

Remediation

Re-establish authentication immediately. Generate the new keypair on a trusted workstation, add the new public key to the GitHub repo''s deploy keys, and replace the private key in the Argo CD namespace: `kubectl create secret generic repo-creds-&lt;id&gt; -n argocd --from-file=sshPrivateKey=&lt;new-key&gt; --from-file=known_hosts=&lt;github-hosts&gt; --dry-run=client -o yaml | kubectl apply -f -` (or update the `Repositories` CRD''s `password` / `sshPrivateKey` field). Restart the repo-server to pick up the rotated secret: `kubectl rollout restart deploy/argocd-repo-server -n argocd`. Then verify end-to-end: `argocd repo list` should show the repo with an empty error column, and `argocd app sync &lt;app&gt; --prune` should reach `Healthy`. Add the rotation procedure to the runbook so the next rotation covers both halves, and add a CI check that runs `argocd repo list` after every declared rotation and fails if any repo reports `Failed` or `Unknown`.

Verification

`argocd repo list` reports no `Failed` entries for the GitOps repo. `argocd app list` shows every Application in `Synced` or `OutOfSync` (not `Unknown`). A forced sync on a single Application (`argocd app sync &lt;app&gt; --force`) succeeds. The repo-server logs no longer contain `Authentication required`. The `repo-creds-&lt;id&gt;` secret in the `argocd` namespace has a `metadata.annotations.rotation-time` field that is current. The CI verifier `argocd repo list | grep -v Unknown | wc -l` equals the number of configured repos.

Prevention

Treat GitOps controller credentials as a rotating, owned asset. Use a Secrets Manager (Vault, AWS Secrets Manager, External Secrets Operator) to hold the private key, and have the Argo CD repo-server mount the secret via ESO so rotation in the source of truth propagates to the namespace automatically. Add a `Reloader` (or Argo CD''s `argocd-repo-server --refresh` webhook) so secret changes trigger a re-read without a manual restart. Add a CI verifier that runs every 15 minutes: `argocd repo list` must report no `Failed` or `Unknown`, and `argocd app list` must report no `Unknown`. Alert on either. The rotation runbook must include "rotate in source of truth, verify `argocd repo list`, verify a test sync" as three steps, not one — rotation without verification is rotation that has not happened.

A credential rotation that updates only one half of the pair is a rotation that has not happened. The controller”s view of the world is the secret in its namespace; the source of truth in GitHub is the public half. Both must move together, and the controller must prove it can still clone before the rotation is considered done.