GitOps controller credential rotation — the controller's repository and cluster access

A GitOps controller is the agent that watches the source repository and reconciles the cluster to match. It holds two distinct credentials: one to read the source repository and one to apply manifests to the target cluster. Both have rotation lifecycles; the rotation is harder than for a CI job because the controller is a long-running process whose interruption is a reconciliation outage. This lesson walks both rotations and the discipline of doing them without downtime.

The controller’s two credentials

The controller sits between two systems and authenticates to both:

flowchart LR
    A["Source repository"] -->|source credential| B["Controller"]
    B -->|cluster credential| C["Target cluster"]
    B --> D["Apply manifests"]

• Source credential. Authenticates the controller to the Git repository. A deploy key for SSH, a PAT or GitHub App token for HTTPS. Used to clone or fetch the repository contents and to poll for new commits.
• Cluster credential. Authenticates the controller to the target cluster. A Kubernetes ServiceAccount token (bound, projected, or via Workload Identity). Used to apply manifests, create resources, and report status.

Each credential has its own rotation lifecycle and its own blast radius. A leak of the source credential lets an attacker read the source repository; a leak of the cluster credential lets an attacker apply arbitrary manifests to the cluster.

Rotating the source credential

The source credential is held by the controller as a Kubernetes secret referenced by a GitRepository or Source custom resource. The rotation is a secret replacement:

kubectl create secret generic flux-system-git-auth \
  --namespace flux-system \
  --from-literal=username="$GIT_USER" \
  --from-literal=password="$GIT_TOKEN"

flux reconcile source git flux-system

The first command creates the new secret in the controller’s namespace. The second forces an immediate reconcile; the controller re-reads the secret and re-authenticates against the source repository using the new credential. If the new credential is valid, the source continues to update. If it is invalid, the reconcile fails with an authentication error, and the old credential remains in place.

Rotating the cluster credential

The cluster credential is held by the controller as a ServiceAccount token. Kubernetes supports two kinds:

• Long-lived legacy token. A bearer token stored in a secret. Rotation creates a new secret, replaces the ServiceAccount’s reference, and deletes the old secret.
• Bound projected token. A token bound to the pod, the audience, and the lifetime, projected by the kubelet at pod start. Rotation is automatic when the bound token’s TTL elapses; the controller picks up the new token without intervention.

The discipline is to use bound projected tokens exclusively. A controller configured with a long-lived legacy token requires the same two-key rotation discipline as any other long-lived credential: issue a new token, distribute, cut over, retire. A controller configured with a bound projected token has no rotation cadence because the token expires by design.

kubectl set serviceaccount \
  --namespace flux-system \
  --field-selector metadata.name=kustomize-controller \
  flux-system

The command above is illustrative; the precise rotation depends on the controller (Flux, Argo CD, or custom) and the credential source. The pattern is the same: identify the token holder, replace the token, verify the controller can still authenticate, retire the old token.

Argo CD credential rotation

Argo CD stores repository credentials in a secrets resource referenced by the argocd-cm ConfigMap. Rotation is performed via the Argo CD CLI:

argocd repo add "$REPO_URL" \
  --username "$GIT_USER" \
  --password "$GIT_TOKEN" \
  --type git

argocd repo list

The first command adds the new credential to Argo CD’s repository credentials store. The second lists the configured repositories. The old credential is removed by running argocd repo rm "$REPO_URL", which removes the credential from the store. A reconcile triggered after the swap uses the new credential; a reconcile triggered before uses the old one.

Production discipline

1. The controller has two credentials, not one. Rotation procedures must address each independently.
2. Prefer bound projected tokens over legacy tokens. Bound tokens expire by design; legacy tokens require rotation.
3. Reconcile after rotation, do not restart the controller. Restarting creates a reconciliation outage.
4. Validate the source status after rotation. A failed reconcile is the desired signal that the new credential is invalid.
5. Rotation of the cluster credential requires a kubeconfig update if the controller is deployed outside the cluster. A controller that authenticates with a static kubeconfig must have that kubeconfig rotated as part of the credential rotation.

Cross-course references

• This course, Part LXXIV-03 (Source reconciliation loops) covers the controller’s reconciliation model that rotation must not interrupt.
• This course, Part XCIII-02 (Deploy keys and SSH key rotation) covers the source-credential rotation in depth.
• Kubernetes for Production Sysadmins, Part XXVIII (ServiceAccount and projected tokens) covers the cluster-credential mechanics.

Quiz