Authentication failure — token expired, key rotated

The second failure mode looks identical to the first until you read the log line: the repository is reachable, but the controller cannot authenticate against it. The symptoms - Degraded, ComparisonError, stalled reconciliation - match the unreachable repository failure exactly. The cause is different, the remediation is different, and the blast radius is wider because the credential is shared across every application the controller manages.

The symptom in the controller log

The first read is the application status:

argocd app get guestbook

A network failure shows rpc error: code = Unavailable. An authentication failure shows:

failed to fetch source: rpc error: code = Unauthenticated
desc = Authentication failed

For SSH, the equivalent is git fetch: Permission denied (publickey). The controller log carries the canonical signal:

kubectl -n argocd logs statefulset/argocd-application-controller

Look for status code 401, status code 403, or Permission denied. The HTTP status distinguishes them: 401 is an expired or revoked token; 403 is a valid token that lacks scope, or a key not registered with the remote.

flowchart LR
    A[Controller fetch] --> B{Repo responds?}
    B -- connection refused --> C["Unavailable - network"]
    B -- HTTP 401 --> D["Unauthenticated - token expired or revoked"]
    B -- HTTP 403 --> E["Forbidden - scope or key not registered"]
    B -- HTTP 404 --> F["NotFound - URL or path wrong"]
    D --> G[All applications Degraded]
    E --> G
    F --> H[Single application Degraded]

Why this failure cascades

A repository outage usually affects one repository. An authentication failure affects every application that uses the same credential. In a typical Argo CD installation there is one Repository resource per remote, and every Application that points at that remote resolves through it. When the credential on that resource stops working, every dependent application transitions to Degraded simultaneously.

For Flux, the equivalent is the GitRepository and its referencing Kustomization and HelmRelease resources. One expired credential takes the entire source tree down.

The operator’s first diagnostic move is to count Degraded applications. One is a misconfiguration; many is a shared credential problem.

The rotation procedure

1. Generate the new credential in the source system (GitHub PAT, GitLab deploy token, SSH key pair). Note the expiry.
2. Update the secret in the cluster. For Argo CD this is the secret referenced by the Repository resource, typically in the argocd namespace. For Flux this is the secret referenced by the GitRepository secretRef.
3. Restart the application controller so it picks up the new secret. Argo CD’s controller reads repository credentials at startup; an in-flight controller will use the cached credential until restart.
4. Verify with a single application. argocd app get <app> should show Synced and Healthy within the next cycle.
5. Revoke the old credential in the source system. Only after the cluster-side rotation has been verified.

The window between steps 2 and 5 is the at-risk window. An operator who revokes first produces an outage; one who creates the new credential without testing produces an outage when the cluster-side secret references the new credential but the source system has not accepted it.

The SSH variant

SSH key rotations are slower because the public key must be registered with the remote. The order: add the new public key, update the cluster-side secret, restart the controller, verify, then remove the old key. Removing the old key first takes every fetch down for the duration.

Production discipline

• Treat credential expiry as an SLO with a pre-expiry window. Tokens that expire in 90 days should be rotated at day 60.
• One credential per remote, not per application. Per-application credentials multiply the rotation surface.
• Test against a non-production application first. The cost of a failed rotation is the same as a repository outage.

Cross-course references

• Linux for Production Sysadmins - Parts XII (RepoSecurity) and XXXIV (ConfigMgmt) cover apt key rotation, with the same add-then-remove ordering.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover AWS access key rotation, with the same new-then-old ordering.

Quiz