Skip to main content
RunBook Academy

← All runbooks in Git, CI/CD & GitOps

high riskservice affecting~45 min

Runbook: Troubleshoot GitOps Repository Authentication

1 · Prerequisites

Confirm every item is in place before any state change.

  • git-cicd-gitops-rb-08-rotate-git-credentials
  • kubectl configured against the GitOps controller's namespace (typically argocd or flux-system)
  • Access to the GitOps controller UI/CLI (argocd, flux)
  • Access to the Git hosting provider's settings (GitHub/GitLab/Bitbucket admin)
  • For SSH: the controller's SSH key fingerprint (ssh-keygen -lf)

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · Identify the GitOps controller in use (Argo CD, Flux, Argo CD Application Set, custom). The controller determines where credentials are stored (Argo CD argocd-secret, Flux flux-system secret, sealed-secret, external secret)
  • · Capture the error message. Argo CD: kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=200 | grep -i -E "auth|credential|permission|denied|fingerprint". Flux: kubectl logs -n flux-system -l app=source-controller --tail=200 | grep -i -E "auth|credential|permission|denied". The error message identifies the failure class (key, token, host key)
  • · Capture the current sync status: argocd app get <app> --refresh or flux get kustomization <name>. Compare to the expected state. OutOfSync with no progress for >5 minutes indicates auth failure rather than a manifest diff
  • · Verify the Git repository is reachable from the controller network. kubectl exec -n argocd deploy/argocd-repo-server -- git ls-remote <repo-url> (Argo CD) or kubectl exec -n flux-system deploy/source-controller -- git ls-remote <repo-url> (Flux). A "could not resolve host" or connection refused is a network issue, not auth
  • · Confirm the credential has not been rotated or revoked upstream. GitHub: Settings → Developer settings → Personal access tokens → confirm the token still exists and is not expired. GitLab: Settings → Access Tokens → confirm. Bitbucket: Settings → App passwords → confirm

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1STEP 1 - Classify the error. Three classes: (a) "could not read Username for" / "authentication required" → credential missing or wrong; (b) "Permission denied (publickey)" / "fingerprint" → SSH key issue; (c) "The repository does not exist or you do not have access" → permission scope issue. The class determines the fix
  2. 2STEP 2 - For class (a) HTTPS token: locate the secret containing the token. Argo CD: kubectl get secret -n argocd argocd-repo-credentials -o yaml (decoded) or argocd repocreds list. Flux: kubectl get secret -n flux-system and inspect the GitRepository CR. Confirm the secret contains the expected token and the token has the required scopes (GitHub: repo for private repos; GitLab: read_repository; Bitbucket: repository:read)
  3. 3STEP 3 - Test the credential outside the controller. Clone the repo using the same credential: GIT_ASKPASS=echo git clone https://x-access-token:<token>@github.com/<org>/<repo>.git /tmp/test-clone && rm -rf /tmp/test-clone. A successful clone proves the credential works; a failure (401, 403) proves the credential is the problem
  4. 4STEP 4 - Rotate the credential if it has expired or been revoked. Generate a new PAT with the required scopes. Update the controller's secret: Argo CD via UI (Repositories → <repo> → Edit) or argocd repocreds add --upsert. Flux: kubectl create secret generic flux-system -n flux-system --from-literal=username=<user> --from-literal=password=<token> --dry-run=client -o yaml | kubectl apply -f -. The upsert pattern is critical — a partial write can leave the controller unable to authenticate at all
  5. 5STEP 5 - For class (b) SSH key: locate the key. Argo CD stores repository credentials in Secrets labeled argocd.argoproj.io/secret-type=repository: kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=repository -o yaml and read the sshPrivateKey field of the Secret whose url matches the repo (argocd repo get <repo> --refresh shows the controller's view). The argocd-ssh-known-hosts-cm ConfigMap holds only SSH host keys, never the credential. Flux: check the GitRepository CR's spec.secretRef. Confirm the key is registered in the Git provider (GitHub: Settings → Deploy keys; GitLab: Settings → Repository → Deploy Keys) and the box "Allow write access" is set if push is required (it is NOT required for read-only GitOps)
  6. 6STEP 6 - Test the SSH key outside the controller. Copy the key from the secret, write to /tmp/gitops-key, chmod 600, and GIT_SSH_COMMAND="ssh -i /tmp/gitops-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git ls-remote git@github.com:<org>/<repo>.git. A successful remote listing proves the key works; a failure (Permission denied) proves the key is the problem
  7. 7STEP 7 - Re-add the known_hosts entry. SSH failures due to host key changes require a known_hosts update. Argo CD stores this in the argocd-ssh-known-hosts-cm ConfigMap. Update with kubectl get cm -n argocd argocd-ssh-known-hosts-cm -o yaml and add the entry from ssh-keyscan github.com 2>/dev/null. Restart the repo server: kubectl rollout restart deploy/argocd-repo-server -n argocd
  8. 8STEP 8 - For class (c) permission scope: the credential works for clone but the controller cannot pull. Check the Git provider's audit log for the specific error. GitHub: Settings → Audit log → filter by user/token. GitLab: Admin → Audit Events. Common cause: the token has access to the repo but the GitOps app/team does not. Add the controller's identity (the user/service account that owns the token) to the repo or team
  9. 9STEP 9 - Trigger a sync to test. argocd app get <app> --refresh && argocd app sync <app> (Argo CD) or flux reconcile source git <name> (Flux). The sync must succeed; a failed sync returns to STEP 1 with the new error message
  10. 10STEP 10 - Verify multi-repo setups. If the controller manages many repos, the auth fix must be applied to every repo using the same credential. argocd repo list and kubectl get gitrepositories -A enumerate all repos. A single failed repo causes OutOfSync for every Application sourced from it

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓kubectl exec -n argocd deploy/argocd-repo-server -- git ls-remote https://github.com/<org>/<repo>.git returns the expected refs without prompting for credentials
  • ✓argocd app list shows all Applications as Synced and Healthy
  • ✓flux get kustomization -A shows all kustomizations as Ready=True
  • ✓A forced refresh (argocd app get <app> --refresh / flux reconcile source git <name>) succeeds and produces a new sync attempt
  • ✓The controller logs (kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=50) no longer contain auth-related errors

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the new credential does not work and the old credential was overwritten: the controller is now in a worse state. Restore the previous secret from a backup: Argo CD via UI (Repositories → &lt;repo&gt; → Edit → use the previous token), Flux via kubectl apply -f <previous-secret-backup.yaml>. The backup was created by the secrets-management workflow; without one, the previous token must be regenerated or recovered from the provider's audit log
  • ↶If the SSH key rotation broke write access but the GitOps controller only needs read access: confirm the deploy key is registered as read-only in the Git provider. The controller should never have write access — it only reads
  • ↶If the fix is stuck on "unknown host key": the controller's known_hosts is empty. The ssh-keyscan approach in STEP 7 is correct; if it does not work, the issue is DNS or egress (the controller cannot reach the Git provider at all). See the network runbook
  • ↶If the controller's secret store is sealed (Sealed Secrets, External Secrets, SOPS): the rotation must go through the same store. A direct kubectl apply will not persist across controller restarts. Re-encrypt and re-apply through the sealed-secret workflow
  • ↶If multiple Applications are stuck due to a single credential: the Applications will recover together once the credential is fixed. Do not try to "fix" individual Applications — they all share the same broken credential

6 · Escalation

When the runbook isn't enough, contact:

  • · The credential works from a developer laptop but not from the controller: this is a network egress issue (egress proxy, NetworkPolicy, firewall). Engage the platform/network team; the controller cannot reach the Git provider
  • · The credential has been revoked because of an incident (e.g., the GitOps app is on a compromised account): see git-cicd-gitops-rb-08-rotate-git-credentials and git-cicd-gitops-rb-28-respond-to-supply-chain-compromise. The auth issue is a symptom of a larger security event
  • · The token has the required scopes but the Git provider's audit log shows repeated 401s: the token may have been cached at the wrong scope by the provider. Rotate the token with a fresh generation; the cache will expire
  • · Argo CD or Flux is running with the controller's secret mounted from an external secrets manager (Vault, AWS Secrets Manager): the rotation must go through the secrets manager. A direct edit will be reverted within the sync interval. Engage the secrets-management team
  • · The fix requires a controller restart (e.g., to pick up a new entry in argocd-ssh-known-hosts-cm): in a production GitOps cluster, restarting the repo server pauses manifest generation. Schedule the restart in a change window. Repository credential Secrets are read on each Git operation, so a rotated key or token takes effect without a restart — only known-hosts and TLS configuration changes need one
  • · The repository is large (>1 GB) and git clone times out: the auth is correct but the operation does not finish. Increase the controller's timeout; consider a shallow clone (--depth 1) or a webhook-driven reconciliation instead of polling

GitOps controllers pull manifests from Git on every reconcile. When authentication fails, the controller stops producing new manifests — applications go OutOfSync and stay there until the auth is fixed. The fix is not to “push a manifest anyway”; the fix is to repair the credential and let the controller reconcile.

1. Capture the auth error from the controller logs

Read-only / Safe
$ echo "--- Argo CD application controller ---"
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=300 --since=10m | grep -iE 'auth|credential|denied|forbidden|fingerprint|repository|known_hosts'
echo "--- Argo CD repo server ---"
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=300 --since=10m | grep -iE 'auth|credential|denied|fingerprint|known_hosts'
echo "--- Flux source controller ---"
kubectl logs -n flux-system -l app=source-controller --tail=300 --since=10m | grep -iE 'auth|credential|denied|fingerprint|repository'

The error message is the source of truth for the failure class. “Could not read Username for” → missing credential. “Permission denied (publickey)” → SSH key issue. “Repository not found” → permission scope issue.

2. Test the credential outside the controller

Read-only / Safe
$ REPO_URL="https://github.com/myorg/gitops-prod"
TOKEN="ghp_xxxxxxxxxxxxxxxxxxxx"
echo "--- HTTPS clone test ---"
GIT_ASKPASS=echo GIT_TERMINAL_PROMPT=0 git ls-remote "$REPO_URL" || echo "AUTH FAILED"
echo "--- with token in URL ---"
git ls-remote "https://x-access-token:$TOKEN@${REPO_URL#https://}"

A successful git ls-remote outside the controller proves the credential is functional. A failure (401, 403) proves the credential is the problem and the controller is just the messenger.

3. Update the credential in Argo CD

Read-only / Safe
$ REPO_URL="https://github.com/myorg/gitops-prod"
USERNAME="myorg-gitops-bot"
TOKEN="ghp_newtokenxxxxxxxxxxxxxxxxx"
argocd repocreds add --upsert --repo "$REPO_URL" --username "$USERNAME" --password "$TOKEN"
echo "--- verify ---"
argocd repocreds list
echo "--- trigger refresh ---"
argocd repo get "$REPO_URL" --refresh

The --upsert flag is critical: it overwrites any existing credential for the repo URL. Without --upsert, a partial write can leave the controller unable to authenticate at all.

4. Update the credential in Flux

Read-only / Safe
$ NS="flux-system"
SECRET="flux-system"
USERNAME="myorg-gitops-bot"
TOKEN="ghp_newtokenxxxxxxxxxxxxxxxxx"
kubectl create secret generic "$SECRET" -n "$NS" \
--from-literal=username="$USERNAME" \
--from-literal=password="$TOKEN" \
--dry-run=client -o yaml | kubectl apply -f -
echo "--- force reconciliation ---"
flux reconcile source git flux-system
flux get kustomization -A

Flux reads the secret on every reconcile. A direct kubectl apply of the new secret is sufficient if the secret is not sealed; otherwise re-encrypt through the sealed-secret workflow.

5. Fix SSH key auth (class b)

Read-only / Safe
$ REPO_URL="git@github.com:myorg/gitops-prod.git"
NS="argocd"
echo "--- locate the repository credential secret for this repo URL ---"
for s in $(kubectl get secret -n "$NS" -l argocd.argoproj.io/secret-type=repository -o name); do
echo "$s -> $(kubectl get -n "$NS" "$s" -o jsonpath='{.data.url}' | base64 -d)"
done
echo "--- extract the SSH key from the matching secret ---"
kubectl get secret -n "$NS" REPLACE_WITH_REPO_SECRET -o jsonpath='{.data.sshPrivateKey}' | base64 -d > /tmp/gitops-key
chmod 600 /tmp/gitops-key
echo "--- test outside the controller ---"
GIT_SSH_COMMAND="ssh -i /tmp/gitops-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git ls-remote "$REPO_URL"
echo "--- update known_hosts ---"
ssh-keyscan github.com 2>/dev/null >> /tmp/known_hosts
kubectl create configmap argocd-ssh-known-hosts-cm -n "$NS" --from-file=ssh_known_hosts=/tmp/known_hosts --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deploy/argocd-repo-server -n "$NS"

The SSH key is the controller”s identity to GitHub. The known_hosts map prevents man-in-the-middle attacks. Both must be correct; either alone will fail.

6. Verify all applications recover

Read-only / Safe
$ echo "--- Argo CD applications ---"
argocd app list -o wide | head -30
echo "--- force refresh of all apps ---"
for app in $(argocd app list -o name); do argocd app get "$app" --refresh; done
echo "--- Flux kustomizations ---"
flux get kustomization -A
echo "--- controller logs (post-fix) ---"
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100 --since=2m | grep -iE 'auth|denied|fingerprint' || echo "no auth errors"

The fix is complete when every application is Synced and Healthy and the controller logs no longer contain auth errors. A forced refresh confirms the controller can read the repo.

Verification

kubectl exec against the controller”s repo server returns the expected refs without prompting. argocd app list shows all applications as Synced and Healthy. flux get kustomization -A shows all kustomizations as Ready. A forced refresh succeeds and produces a new sync. The controller logs no longer contain auth errors.

Rollback

If the new credential does not work and the old credential was overwritten, restore the previous secret from a backup via the secrets-management workflow. If the SSH key rotation broke write access but the controller only needs read access, confirm the deploy key is registered as read-only in the Git provider. If the fix is stuck on “unknown host key,” the controller”s known_hosts is empty and must be populated via ssh-keyscan. If the secret store is sealed, re-encrypt and re-apply through the sealed-secret workflow — a direct kubectl apply will not persist. If multiple Applications are stuck on a single credential, they recover together once the credential is fixed; do not try to fix individual Applications.

References

  1. Argo CD — Repositories
  2. Argo CD — Private Repositories
  3. Flux — Git Repositories
  4. Flux — GitHub Authentication
  5. GitHub — Personal Access Tokens
  6. GitLab — Personal Access Tokens
  7. Bitbucket — App Passwords