high riskservice affecting~60 min

Runbook: Rotate registry credentials with no failed pulls

1 · Prerequisites

Confirm every item is in place before any state change.

You can issue a new credential at the registry without revoking the current one first
The registry supports more than one active credential per identity, or you have accepted a brief cutover instead of an overlap
You have an inventory of every consumer: hosts, CI jobs, build agents, Kubernetes or Swarm clusters, and any human who logs in
You can edit the CI secret store and re-run a job to pick up the new value
You have shell access to every host that holds a credential on disk
The secret manager is reachable and you can write the new value to it
A rollback path exists: the old credential stays valid until the overlap window closes

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· REG=registry.example.com; jq -r "keys[]" ~/.docker/config.json shows whether credsStore, credHelpers or auths is in use on this host
· jq -r ".auths | keys[]" ~/.docker/config.json lists every registry this host holds a credential for
· If credsStore is set, command -v docker-credential-"$(jq -r .credsStore ~/.docker/config.json)" resolves to a helper binary on PATH
· sudo ls -l /root/.docker/config.json checks whether root holds a separate credential from your user
· systemctl show docker --property=Environment and systemctl cat docker check for a credential passed through a unit or drop-in
· grep -rIl "$REG" /etc/systemd/system /srv/infra 2>/dev/null lists configuration files that name the registry
· The CI secret store lists which pipelines reference the registry credential
· curl -fsS -o /dev/null -w "%{http_code}\n" https://"$REG"/v2/ confirms the registry endpoint is reachable before you change anything

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Set REG=registry.example.com and REG_USER=svc-deploy so every step below refers to the same identity
2Build the consumer inventory and write it down: every host, CI pipeline, build agent, orchestrator and human account that authenticates as "$REG_USER"
3Issue the NEW credential at the registry without revoking the old one; both must be valid at the same time
4Write the new credential into the secret manager under a new version, leaving the old version readable
5Confirm the new credential works in isolation before touching any consumer: DOCKER_CONFIG=$(mktemp -d) docker login "$REG" --username "$REG_USER" --password-stdin then pull one private image
6Update the CI secret store to the new value, then re-run one pipeline and confirm its pull step succeeds
7Update each host in turn: get-secret registry/svc-deploy-token | docker login "$REG" --username "$REG_USER" --password-stdin
8On hosts where root pulls images independently of your user, repeat the login as root: sudo -H docker login "$REG" --username "$REG_USER" --password-stdin
9Update any systemd unit or drop-in that carries the credential in Environment or EnvironmentFile, then systemctl daemon-reload and restart the unit
10Update orchestrator registry secrets and redeploy the workloads that reference them so the new value is actually in use
11Force a real pull on each updated consumer to confirm the new credential is the one being used: docker pull "$REG"/team/app:1.4.2
12Wait out the agreed overlap window with monitoring on registry authentication failures
13Revoke the old credential at the registry only after every consumer has completed a successful pull with the new one
14Prove the old credential no longer works, from an isolated config directory (see Verification)
15Remove the superseded version from the secret manager and record the rotation date and the next due date

4 · Verification

Confirm the procedure actually fixed the problem.

✓A pull succeeds with the NEW credential from an isolated config: DOCKER_CONFIG=$(mktemp -d) docker login "$REG" --username "$REG_USER" --password-stdin then docker pull "$REG"/team/app:1.4.2 completes
✓A login FAILS with the OLD credential from an isolated config: docker login returns a non-zero exit and an unauthorized error. Both directions are required; a successful pull with the new credential alone proves nothing about whether the old one is dead.
✓Every host in the inventory completed a fresh authenticated pull after its update, evidenced by the pull output rather than by the presence of a config file
✓A CI pipeline that pulls a private image ran to completion after the secret store was updated
✓Registry authentication-failure metrics show no increase during or after the overlap window
✓No consumer in the inventory remains unaccounted for: the written list has a tick against every entry
✓grep -c "$REG" on each updated config file shows exactly one entry per registry, with no stale duplicate host entry
✓Orchestrator workloads were redeployed, not merely reconfigured: a pod or task that starts fresh pulls successfully

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶During the overlap window the rollback is trivial: the old credential is still valid, so re-run docker login with the old value on any consumer that broke
↶Restore the previous version from the secret manager and re-run the affected CI pipeline
↶Revert any systemd unit or drop-in change, then systemctl daemon-reload and restart the unit
↶After the old credential is revoked there is no rollback to it: revocation is not reversible at the registry. Issue a third credential and treat it as a fresh rotation.
↶If pulls are failing in production and the cause is unclear, issue a new credential immediately and push it to the failing consumers rather than trying to restore the revoked one

6 · Escalation

When the runbook isn't enough, contact:

· The registry does not permit two active credentials for one identity: escalate to the registry owner to agree a short cutover window, since no overlap is possible
· A consumer cannot be identified from registry logs and pulls keep failing after revocation: escalate to the platform team with the source addresses from the registry access log
· The credential is held by a third party or a partner system: escalate to the service owner before revoking, since revocation breaks somebody you cannot page
· The rotation was triggered by a suspected leak rather than by schedule: stop following this runbook and follow the leaked-secret response instead, which revokes first
· Rotation cannot complete inside the overlap window: extend the window with the registry owner rather than revoking on schedule with consumers still on the old credential

Rotating a registry credential is easy. Rotating it without a single failed pull is the actual job, and it fails for one reason: somebody revoked the old credential before finding every consumer of it.

The consumers you forget are never the obvious ones. They are the root user on a host where a systemd unit pulls at boot, a CI runner whose secret was set two years ago by someone who has left, and a staging cluster nobody counts as production until it stops working.

Where registry credentials actually live

Six places to look, every time:

~/.docker/config.json for your user — the auths entry.
/root/.docker/config.json — separate, and easy to miss. Anything run with sudo docker without -H uses root’s config.
A credential helper. If credsStore or credHelpers is set, the credential is not in config.json; it is in an external store reached through a docker-credential- binary. Rotating the file achieves nothing in that case.
CI secret stores. Usually the largest number of consumers and the least visible.
systemd unit environment. An Environment= line or an EnvironmentFile= in a unit or drop-in, feeding a wrapper script that logs in at boot.
DOCKER_CONFIG. A process pointed at a different config directory entirely. Automation does this more often than people.

Read-only / Safefind the credential— Reads configuration files and unit definitions. No changes.

REG=registry.example.com

# Which mechanism is in use on this host?
grep -o '"credsStore"\|"credHelpers"\|"auths"' ~/.docker/config.json

# Which registries does this host hold credentials for?
jq -r '.auths | keys[]' ~/.docker/config.json

# Is a helper configured, and does it exist?
HELPER=$(jq -r '.credsStore // empty' ~/.docker/config.json)
[ -n "$HELPER" ] && command -v "docker-credential-$HELPER"

# Root has its own config
sudo ls -l /root/.docker/config.json

# Units and drop-ins that carry a credential
systemctl cat docker
grep -rIl "$REG" /etc/systemd/system /srv/infra 2>/dev/null

Step 1: Build the inventory before you issue anything

Write it down. A list in your head is a list you will revoke against.

For each consumer record: what it is, who owns it, how the credential reaches it, and how you will confirm it has cut over. The last column is the one that matters — “updated the secret” is not confirmation, “the pipeline ran and pulled” is.

If the registry exposes an access log, use it. Source addresses over the last thirty days for the identity you are rotating will surface consumers that are on nobody’s list.

Step 2: Issue the new credential, keep the old one alive

On Docker Hub this means creating a second personal access token rather than regenerating the first. Tokens are managed under Account settings → Personal access tokens, where an existing token can be deactivated or deleted independently of any other.

Step 3: Test the new credential in isolation first

Before you touch a single consumer, prove the new credential actually works. DOCKER_CONFIG points the CLI at a throwaway config directory, so this test cannot disturb the credential currently in use.

Read-only / Safeisolated login test— Authenticates into a temporary config directory. Does not modify your real config.json.

REG=registry.example.com
REG_USER=svc-deploy

TESTDIR=$(mktemp -d)
DOCKER_CONFIG="$TESTDIR" sh -c '
get-secret registry/svc-deploy-token-new |
  docker login "$0" --username "$1" --password-stdin
' "$REG" "$REG_USER"

DOCKER_CONFIG="$TESTDIR" docker pull "$REG"/team/app:1.4.2
echo "pull exit: $?"

rm -rf "$TESTDIR"

Note the credential is piped into --password-stdin, never passed as --password. A credential on the command line lands in shell history, in the process list for anyone running ps, and in any audit log that records command lines.

Step 4: Cut consumers over, one at a time

Configuration changeupdate a host— Replaces the stored credential on one host. The old credential remains valid at the registry.

REG=registry.example.com
REG_USER=svc-deploy

# Your user
get-secret registry/svc-deploy-token-new \
| docker login "$REG" --username "$REG_USER" --password-stdin

# Root, if root pulls independently
get-secret registry/svc-deploy-token-new \
| sudo -H docker login "$REG" --username "$REG_USER" --password-stdin

# Confirm by PULLING, not by reading the config file
docker pull "$REG"/team/app:1.4.2
sudo docker pull "$REG"/team/app:1.4.2

A config file containing the new value is not evidence. A completed pull is. If a credential helper is in use, the config file will not change at all, which is precisely why the file is the wrong thing to check.

For systemd-managed logins, edit the unit or drop-in, then:

Service impact possibleunit reload— Reloads unit definitions and restarts the unit. The unit is briefly unavailable.

sudo systemctl daemon-reload
sudo systemctl restart docker-login-helper.service
systemctl status docker-login-helper.service --no-pager

Step 5: Revoke, and then prove revocation

Only after every consumer in the written inventory has a tick against it.

Destructiverevoke and prove— Revocation at the registry is not reversible. Any consumer still on the old credential fails immediately.

REG=registry.example.com
REG_USER=svc-deploy

# 1. Revoke the old credential at the registry
#    (Docker Hub: Account settings -> Personal access tokens -> Delete)

# 2. The NEW credential still works
NEWDIR=$(mktemp -d)
DOCKER_CONFIG="$NEWDIR" sh -c '
get-secret registry/svc-deploy-token-new |
  docker login "$0" --username "$1" --password-stdin
' "$REG" "$REG_USER"
DOCKER_CONFIG="$NEWDIR" docker pull "$REG"/team/app:1.4.2
echo "new credential pull exit: $? (expect 0)"

# 3. The OLD credential must now FAIL
OLDDIR=$(mktemp -d)
DOCKER_CONFIG="$OLDDIR" sh -c '
get-secret registry/svc-deploy-token-old |
  docker login "$0" --username "$1" --password-stdin
' "$REG" "$REG_USER"
echo "old credential login exit: $? (expect non-zero, unauthorized)"

rm -rf "$NEWDIR" "$OLDDIR"

Step 6: Clean up

Configuration changetidy— Removes superseded credential material. Confirm the rotation is complete first.

REG=registry.example.com

# Remove a stale entry for a registry this host no longer uses
docker logout "$REG"

# Confirm what remains
jq -r '.auths | keys[]' ~/.docker/config.json

# Retire the superseded secret version and record the next due date

Common failure patterns

Symptom	Likely cause	Resolution
Rotation had no effect at all	`credsStore` is set, so the credential was never in `config.json`	Rotate through the helper; `docker login` handles this for you
Your pulls work, automated pulls fail	Root holds a separate `config.json`	Repeat the login with `sudo -H`
CI pulls fail, hosts are fine	CI secret store still holds the old value	Update the secret, then re-run a pipeline to confirm
Pulls fail only at boot	A unit logs in from `Environment=` or `EnvironmentFile=`	Update the drop-in, `daemon-reload`, restart
Orchestrator workloads fail after a delay	Registry secret updated but workloads never redeployed	Redeploy so new tasks pull with the new secret
Old credential still authenticates	Deactivated rather than deleted, or a second identity exists	Delete it, then re-run the failure test