Registry credential rotation — push and pull credentials

A container registry holds the images the cluster runs. The registry credential authenticates two distinct actions: the CI push that publishes the image, and the cluster pull that fetches the image at pod start. Each action has its own credential, its own rotation lifecycle, and its own blast radius. This lesson walks both lifecycles and the CLI commands that execute them.

Two consumers, two credentials

The same registry may require two different credentials for the same image:

• Push credential. Held by CI. Grants the push (and typically the pull and delete) scope on the repository. Used during the build pipeline to publish a new image tag.
• Pull credential. Held by the cluster. Grants the pull scope on the repository. Used by the kubelet to fetch the image when a pod referencing it is scheduled.

flowchart LR
    A["CI build"] -->|push credential| B["Registry"]
    C["Kubelet"] -->|pull credential| B
    B --> D["Image stored"]

The push credential is the higher-privilege credential: it can overwrite tags and delete images. The pull credential is the lower-privilege credential: it can only read. A team that uses the same credential for both has coupled two rotation cadences and given the cluster’s pull path the ability to mutate the registry.

Push credential rotation

The push credential lives in the CI runner’s ~/.docker/config.json or in the platform’s secret store. The rotation is a login/logout pair:

echo "$REGISTRY_PASSWORD" | docker login "$REGISTRY" \
  -u "$REGISTRY_USER" --password-stdin

docker logout "$REGISTRY"

The first command authenticates the new credential; the --password-stdin flag reads the password from standard input to avoid the password appearing in shell history or process listings. The second command removes the credential from the runner’s config file. After logout, an attempt to push fails with an authentication error, which confirms the old credential is no longer cached.

Pull credential rotation in Kubernetes

The pull credential lives in the cluster as a kubernetes.io/dockerconfigjson secret. The rotation creates a new secret, references it from the namespace’s default service account, and removes the old secret:

kubectl create secret docker-registry "$NEW_PULL_SECRET" \
  --namespace "$NAMESPACE" \
  --docker-server="$REGISTRY" \
  --docker-username="$REGISTRY_USER" \
  --docker-password="$REGISTRY_PASSWORD"

kubectl patch serviceaccount default \
  --namespace "$NAMESPACE" \
  -p "{\"imagePullSecrets\": [{\"name\": \"$NEW_PULL_SECRET\"}]}"

kubectl delete secret "$OLD_PULL_SECRET" --namespace "$NAMESPACE"

The first command creates a new docker-registry secret in the namespace. The second patches the default service account in the namespace to reference the new secret for image pulls. The third removes the old secret. Pods created after the patch pick up the new secret automatically; pods already running continue to use the image they have already pulled.

Per-namespace pull secrets

The pull secret is per-namespace, not cluster-wide. A production namespace and a development namespace can hold different credentials, with different rotation cadences, and the leak of a development credential does not give an attacker access to production images.

flowchart TD
    A["Cluster"] --> B["Namespace: production"]
    A --> C["Namespace: development"]
    A --> D["Namespace: ci-build"]
    B --> E["pull-secret-prod"]
    C --> F["pull-secret-dev"]
    D --> G["pull-secret-ci"]

The discipline is to never copy a pull secret across namespaces. Each namespace holds the credential it needs; rotation is per-namespace; a leak in one namespace does not propagate. A team that uses a single cluster-wide secret for image pulls has collapsed this isolation and made every namespace’s blast radius equal to the cluster’s.

ECR and short-lived pull

ECR (and the equivalent on other cloud providers) supports a pull pattern that does not require a long-lived credential: the kubelet exchanges its node’s IAM role for a short-lived ECR login token. The token is valid for twelve hours; no static credential exists.

kubectl patch serviceaccount default \
  --namespace "$NAMESPACE" \
  -p "{\"metadata\": {\"annotations\": {\"eks.amazonaws.com/role-arn\": \"$ECR_PULL_ROLE_ARN\"}}"

The patch annotates the service account with the IAM role that grants pull access. The cluster’s IAM integration performs the credential exchange at pod start. The discipline is that no pull secret exists for ECR-backed images; the static credential category is eliminated entirely for that registry.

Production discipline

1. Use different credentials for push and pull. Coupling the two means coupling two blast radii.
2. Pull secrets are per-namespace. Never copy across namespaces.
3. Local cleanup and registry revocation are separate steps. Docker logout is local; registry revocation is remote.
4. Prefer IAM- or OIDC-based pull. No static credential exists in the cluster.
5. Pull credential rotation includes a node rolling step. Nodes that miss the rotation keep the old credential.

Cross-course references

• This course, Part LIII-06 (Image signing in CI) covers the signing workflow that the push credential enables.
• This course, Part LXII-04 (Per-namespace image policies) covers the policy layer that constrains which registries a namespace may pull from.
• Kubernetes for Production Sysadmins, Part XXII (Image pull secrets) covers the kubelet-side mechanics in depth.

Quiz