critical riskservice affecting~60 min

Runbook: Recover from an Artifact Registry Outage

1 · Prerequisites

Confirm every item is in place before any state change.

git-cicd-gitops-rb-15-validate-production-artifact
crane, skopeo, or docker CLI installed for registry operations
kubectl access to affected clusters
A documented mirror/backup registry with the required images pre-mirrored (or a warm-standby)
Credentials for the mirror registry (pull secret, IAM role)

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the registry outage is real, not a credential or network issue. crane manifest <registry>/<image>:<tag> (or skopeo inspect --raw docker://<registry>/<image>:<tag>) returns the registry error. Compare against the registry's status page: GHCR https://www.githubstatus.com/, ECR is part of AWS, GCR is part of GCP, Harbor self-hosted has its own health endpoint
· Identify every image that needs to be served from the mirror. kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\\n' | sort -u > /tmp/in-use-images.txt. Every image in this list must be available in the mirror, or every workload that depends on it cannot start
· Verify the mirror registry has the required images. For each image: crane manifest <mirror>/<image>:<tag>. A missing image in the mirror means the corresponding workload cannot start during the failover
· If the mirror does not have the images: pre-mirror them now. crane copy <registry>/<image>:<tag> <mirror>/<image>:<tag> --all-tags for each image. The copy may fail if the registry is down — fall back to a recent backup or to image promotion from the build pipeline (if the build pipeline produces to both registries)
· Confirm Kubernetes image pull secrets are configured for the mirror. kubectl get secret -A -o jsonpath='{.items[?(@.type=="kubernetes.io/dockerconfigjson")].metadata.namespace}/{.metadata.name}'. Every namespace that pulls from the failed registry must have a pull secret for the mirror
· Identify the GitOps controller's image-pull secret. The controller pulls manifests, not images directly — but the resources it applies reference images, and the kubelet pulls those images. The fix is at the kubelet level, not the controller level
· For ECR/GCR/ACR: verify the registry endpoint is reachable from cluster nodes. kubectl debug node/<node> -it --image=alpine -- nslookup <registry-endpoint>. If the DNS resolves but the connection fails, it is a registry outage, not a cluster DNS issue

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Declare the registry failover and notify consumers. Post to the on-call channel: "Registry failover initiated from <primary> to <mirror>. Provider status: <url>. Reason: <outage>. ETA: <minutes>. Workloads affected: <list>". The notification is the audit trail
2STEP 2 - Pre-mirror all in-use images to the failover registry. For every image in /tmp/in-use-images.txt: crane copy <primary>/<image>:<tag> <mirror>/<image>:<tag>. The --all-tags flag copies all tags for that image; use it for libraries that have many tags. Skip images that fail — they cannot be served during the failover
3STEP 3 - Re-sign the mirrored images (if using cosign/Sigstore). crane copy re-tags but does not re-sign. cosign copy preserves the signature and attestation; crane copy does not. Use cosign copy <primary>/<image>:<tag> <mirror>/<image>:<tag> to preserve the signature. After copy, verify with cosign verify --key <pubkey> <mirror>/<image>@<digest>
4STEP 4 - Update the image pull secret in every namespace. Create or update the docker-registry secret for the mirror: kubectl create secret docker-registry <pull-secret-name> --docker-server=<mirror> --docker-username=<user> --docker-password=<token> --docker-email=<email> --namespace=<ns> --dry-run=client -o yaml | kubectl apply -f -. The --dry-run=client -o yaml produces a manifest that can be applied without leaking the secret to stdout
5STEP 5 - Patch Kubernetes ServiceAccounts to reference the new pull secret. kubectl patch serviceaccount default -n <ns> -p '{"imagePullSecrets":[{"name":"<pull-secret-name>"}]}'. Repeat for every namespace that pulls from the failed registry. The default ServiceAccount is the most common; custom ServiceAccounts must also be patched
6STEP 6 - For workloads already running: they keep running (their images are already pulled). The outage only affects new pods (restarts, scaling, new deployments). Workloads whose manifests are repointed to the mirror in STEP 8 roll automatically once the controller syncs the changed image reference — any pod-template change triggers the rolling update. kubectl rollout restart deploy/<name> -n <ns> is needed only when the mirror is reached without a template change (for example, a node-level containerd mirror config). Verify the new pods pull from the mirror: kubectl get pods -n <ns> -l app=<name> -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'
7STEP 7 - Update CI/CD pipelines to push to the mirror (or both registries). The pipeline should produce to <mirror> instead of <primary> until the primary returns. For GitHub Actions: update the docker push step. For GitLab CI: update the docker build --push step. For multi-registry pipelines: push to both registries (the cost is acceptable insurance against an outage)
8STEP 8 - Update GitOps manifests if they hardcode the registry hostname. Argo CD/Flux does not transform image names — the manifest references the registry directly. Patching the manifest in Git is the source-of-truth fix. Rewrite every affected manifest: git grep -l "<primary>" -- "*.yaml" | xargs sed -i "s|<primary>|<mirror>|g", then commit and push. The controller reconciles to the new image names
9STEP 9 - Verify workloads start with the mirrored image. kubectl describe pod <pod> -n <ns> | grep -A5 "Events:". The Pulling event should reference the mirror. The Started event confirms the container started. kubectl get pods -n <ns> shows all pods Running
10STEP 10 - When the primary returns: re-push the images that were built during the failover from the mirror to the primary. crane copy <mirror>/<image>:<new-tag> <primary>/<image>:<new-tag>. Re-sign with cosign copy. Re-deploy from the primary to restore the canonical state
11STEP 11 - Repoint GitOps manifests back to the primary. git revert <failover-commit> (the commit that changed the registry hostname). Push. The controller reconciles back to the primary registry. Verify with kubectl get pods -n <ns> -o jsonpath='{.items[*].status.containerStatuses[*].image}'
12STEP 12 - Repoint CI/CD pipelines back to the primary. Update the docker push step back to the primary. Verify with a test push. The failover is complete; the primary is the source of truth again
13STEP 13 - Record the failover. Open an incident ticket: registry, time of failover, time of recovery, workloads affected, images re-pushed, who authorized the failover, who executed it, the reconciliation performed after recovery. The record is the audit trail

4 · Verification

Confirm the procedure actually fixed the problem.

✓crane manifest <mirror>/<image>:<tag> returns a valid manifest for every image in use
✓cosign verify --key <pubkey> <mirror>/<image>@<digest> returns "Verified OK" for every signed image in use
✓kubectl get pods -A -o jsonpath='{.items[*].status.containerStatuses[*].state}' | grep -i running | wc -l equals the expected pod count
✓No pods in ImagePullBackOff or ErrImagePull: kubectl get pods -A | grep -E "ImagePull|ErrImage" | wc -l returns 0
✓The application's health endpoint returns ok: curl -fsS https://<service>/healthz | jq .status
✓After primary recovery: crane manifest <primary>/<image>:<tag> returns the expected manifest (including images built during the failover)
✓After repointing GitOps manifests back to the primary: kubectl get pods -A -o jsonpath='{.items[*].status.containerStatuses[*].image}' shows the primary registry hostname

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If pre-mirroring fails because the primary is unreachable: the failover cannot start. Escalate per the incident response process. Workloads continue to run on their already-pulled images but cannot be restarted, scaled, or redeployed until the registry returns
↶If the mirror does not have the required images and pre-mirroring fails: some workloads will be unable to start. Identify them and engage the application owner; the workloads may need to be paused until the primary returns
↶If re-signing with cosign copy fails because the original signature is not on the mirror: the image on the mirror is unsigned. Do NOT deploy unsigned images to production. Either re-sign manually with cosign sign or wait for the primary to return
↶If the GitOps manifest change cannot be pushed (the Git provider is also down): see git-cicd-gitops-rb-21-recover-git-hosting-dependency. The registry failover and the Git hosting failover may happen simultaneously; coordinate the two
↶If the failover causes an outage longer than the workload's restart tolerance (some workloads will not recover automatically): engage the platform team. The failover may need to be partial — only workloads with mirror coverage failover, the rest wait for the primary
↶If the mirror registry is on the same provider as the primary (and that provider is down): the failover has no failover target. Escalate per the incident response process. The team is offline
↶If the GitOps manifest revert cannot be applied because the manifest path was deleted during the failover: the revert must re-create the manifest. git revert -m 1 <failover-merge> restores the deleted file as part of the revert

6 · Escalation

When the runbook isn't enough, contact:

· The registry outage also affects the team's image-signing infrastructure (cosign, Fulcio, Rekor): the failover cannot re-sign images. Engage the security team. Deploying unsigned images to production may be acceptable as an emergency but must be documented as a security exception
· The mirror registry is shared with other teams and is being overwhelmed: the failover may not work for all images. Prioritize by business impact (revenue-generating services first, internal tools last). Engage the platform team and the application owners for prioritization
· The outage is caused by a security incident at the registry (ransomware, supply-chain attack on the registry itself): the failover must include scanning every image for compromise. Do not deploy any image that has not been scanned. Engage the security team and see git-cicd-gitops-rb-28-respond-to-supply-chain-compromise
· The failover reveals that some images are not reproducible (the build pipeline cannot produce them again): the failover works for existing images but new builds of those images fail. Engage the application owner to fix the build; the failover is a temporary state
· The outage is on a managed registry (ECR/GCR/ACR) and the cloud provider is also down: the failover target must be a different provider. Multi-cloud mirror strategies are needed; engage the platform team
· The outage lasts longer than the team's recovery time objective (RTO) and the team cannot deploy for an extended period: engage the leadership team; the business impact must be communicated. The failover is partial — operations continue in degraded mode, but new feature deployment is paused

When the artifact registry is down, new pods cannot pull images. Workloads already running keep running (their images are already-pulled), but restarts, scaling, and new deployments fail. The failover is to a mirror registry — every image must be pre-mirrored, image pull secrets must be updated, and the GitOps manifests (which reference the registry hostname) must be patched.

1. Identify in-use images and the failover target

Read-only / Safe

$ echo "--- enumerate in-use images ---"
kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u > /tmp/in-use-images.txt
wc -l /tmp/in-use-images.txt
head /tmp/in-use-images.txt
echo "--- mirror registry check ---"
MIRROR="myteam-mirror.example.com"
while read -r IMAGE; do
RESULT=$(crane manifest "$MIRROR/$IMAGE" 2>&1 | head -1)
echo "$IMAGE: $RESULT"
done < /tmp/in-use-images.txt | head -20

The in-use image list is the failover scope. The mirror registry must have every image — gaps mean some workloads cannot start during the failover.

2. Pre-mirror all in-use images to the failover registry

Read-only / Safe

$ PRIMARY="ghcr.io"
MIRROR="myteam-mirror.example.com"
echo "--- mirror all images ---"
while read -r IMAGE; do
echo "mirroring: $IMAGE"
crane copy "$PRIMARY/$IMAGE" "$MIRROR/$IMAGE" 2>&1 | tail -3
done < /tmp/in-use-images.txt

The copy may be slow for large images. Run in parallel with xargs or a background loop if the list is long. Skip images that fail — they cannot be served during the failover.

3. Preserve signatures during the mirror copy

Read-only / Safe

$ PRIMARY="ghcr.io"
MIRROR="myteam-mirror.example.com"
PUBKEY="https://accounts.google.com/.well-known/key-signing.pub"
IMAGE="myorg/myimage:v1.2.3"
echo "--- copy with cosign (preserves signature and attestation) ---"
cosign copy "$PRIMARY/$IMAGE" "$MIRROR/$IMAGE"
echo "--- verify the signature on the mirror ---"
DIGEST=$(crane digest "$MIRROR/$IMAGE")
cosign verify --key "$PUBKEY" "$MIRROR/$IMAGE@$DIGEST"

cosign copy preserves the signature, attestation, and SBOM. crane copy does not. For production images, cosign copy is the correct tool.

4. Update image pull secrets

Read-only / Safe

$ MIRROR="myteam-mirror.example.com"
USERNAME="myorg-mirror-bot"
TOKEN="REPLACE_WITH_MIRROR_TOKEN"
NS="prod"
echo "--- create the pull secret ---"
kubectl create secret docker-registry regcred \
--docker-server="$MIRROR" \
--docker-username="$USERNAME" \
--docker-password="$TOKEN" \
--docker-email=ops@example.com \
--namespace="$NS" \
--dry-run=client -o yaml | kubectl apply -f -
echo "--- patch the default ServiceAccount ---"
kubectl patch serviceaccount default -n "$NS" -p '{"imagePullSecrets":[{"name":"regcred"}]}'

The --dry-run=client -o yaml pattern produces a manifest that can be applied without leaking the secret. The default ServiceAccount patch is required for pods that do not specify their own imagePullSecrets.

5. Patch GitOps manifests to reference the mirror

Read-only / Safe

$ GITOPS_REPO="/path/to/gitops-prod"
PRIMARY="ghcr.io"
MIRROR="myteam-mirror.example.com"
cd "$GITOPS_REPO"
echo "--- enumerate manifests that reference the primary ---"
grep -rl "$PRIMARY" --include='*.yaml' .
echo "--- sed in place ---"
git grep -l "$PRIMARY" -- '*.yaml' | xargs sed -i "s|$PRIMARY|$MIRROR|g"
git diff --stat
git add -A
git commit -m "failover(registry): mirror to $MIRROR during primary outage"
git push

The Git commit is the source of truth. The GitOps controller will apply the new image references to the cluster. The reverse commit restores the primary when it returns.

6. Roll workloads onto the mirror

Read-only / Safe

$ NS="prod"
SERVICE="checkout-api"
echo "--- template changed in Git (step 5): the sync triggers the rollout ---"
kubectl rollout status deploy/"$SERVICE" -n "$NS" --timeout=300s
echo "--- template unchanged (runtime-level mirror config only): restart manually ---"
kubectl rollout restart deploy/"$SERVICE" -n "$NS"
kubectl rollout status deploy/"$SERVICE" -n "$NS" --timeout=300s
echo "--- check the image ---"
kubectl get pods -n "$NS" -l app="$SERVICE" -o jsonpath='{.items[*].status.containerStatuses[*].image}{"\n"}'
echo "--- check for ImagePullBackOff ---"
kubectl get pods -n "$NS" | grep -E "ImagePull|ErrImage" || echo "no pull errors"

If step 5 changed the image references in Git, no restart is needed: the controller’s sync updates the Deployment’s pod template, and any pod-template change triggers the rolling update automatically. A kubectl rollout restart is required only when the mirror is reached without a template change (for example, a node-level containerd mirror config) — nothing then triggers a rollout, and running pods keep their old images until restarted.

7. When the primary returns: re-push, re-sign, repoint

Read-only / Safe

$ PRIMARY="ghcr.io"
MIRROR="myteam-mirror.example.com"
PUBKEY="https://accounts.google.com/.well-known/key-signing.pub"
echo "--- re-push images built during the failover ---"
crane copy "$MIRROR/myorg/myimage:v1.2.4" "$PRIMARY/myorg/myimage:v1.2.4"
echo "--- re-sign ---"
cosign copy "$MIRROR/myorg/myimage:v1.2.4" "$PRIMARY/myorg/myimage:v1.2.4"
echo "--- revert the GitOps commit ---"
GITOPS_REPO="/path/to/gitops-prod"
cd "$GITOPS_REPO"
git revert --no-edit HEAD
git push

The cosign copy re-pushes the signature and attestation. The git revert rolls back the manifest change. The controller reconciles back to the primary registry.

Verification

crane manifest returns valid manifests for every in-use image in the mirror. cosign verify returns “Verified OK” for every signed image in the mirror. All expected pods are Running. No pods in ImagePullBackOff or ErrImagePull. The application”s health endpoint returns ok. After primary recovery, crane manifest returns the expected manifest for images built during the failover. After repointing, pods show the primary registry hostname.

Rollback

If pre-mirroring fails because the primary is unreachable, escalate — workloads continue on already-pulled images but cannot be restarted until the registry returns. If the mirror is missing required images, identify and engage the application owner; some workloads will be unable to start. If re-signing fails, do not deploy unsigned images — wait for the primary or re-sign manually. If the GitOps manifest change cannot be pushed because the Git provider is also down, coordinate the two failovers. If the failover outage is longer than the workload”s restart tolerance, engage the platform team. If the mirror registry is on the same provider as the primary (and that provider is down), escalate — no failover target exists.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Identify in-use images and the failover target

2. Pre-mirror all in-use images to the failover registry

3. Preserve signatures during the mirror copy

4. Update image pull secrets

5. Patch GitOps manifests to reference the mirror

6. Roll workloads onto the mirror

7. When the primary returns: re-push, re-sign, repoint

Verification

Rollback

References