Objective

By the end of this lab you will have authored the artefacts that recover from a regional OCI registry outage: the multi-region topology, the cross-region replication configuration, the failover runbook, the synthetic check, the failover script, the cache restore script, the recovery evidence bundle, and the audit trail.

The point of this lab is not the ECR replication configuration — the AWS docs cover that. The point is the operationalisation: the topology that minimises the blast radius, the synthetic checks that detect the outage early, the failover runbook that the on-call engineer follows under pressure, the cache restoration procedure for when the replica is also unavailable, and the audit trail.

Architecture

The team’s OCI registry fleet is multi-region: us-east-1 (primary), us-west-2 (replica), and an immutable cache (S3 with versioning) for cold storage. Cross-region replication is configured from us-east-1 to us-west-2; the replication is asynchronous with a typical lag of 5-15 minutes. The immutable cache stores every image with its digest; the cache is the last resort when both regions are unavailable.

flowchart LR
    A["CI build\nus-east-1"] -- "push" --> B["ECR primary\nus-east-1"]
    B -- "replication\n(5-15 min)" --> C["ECR replica\nus-west-2"]
    B -- "push" --> D["S3 immutable cache\nus-east-1"]
    D -- "cross-region\nreplication" --> E["S3 immutable cache\nus-west-2"]
    F["EKS cluster\nus-east-1"] -- "pull" --> B
    G["EKS cluster\nus-west-2"] -- "pull" --> C
    F -- "fallback pull" --> D
    G -- "fallback pull" --> E

The on-call engineer fails over from us-east-1 to us-west-2 when the primary is unreachable. The failover updates the cluster’s imagePullSecret and the image: references in the manifests (via a Git PR). The cluster pulls from the replica; the deploys continue to work.

Requirements

An OCI registry with cross-region replication (ECR, GCP Artifact Registry, GHCR with replication, or a self-hosted Harbor).
An immutable cache (S3 with versioning, GCS with object versioning, or a self-hosted blob store).
A kind cluster with Argo CD installed (Lab 19).
crane or skopeo on the workstation.
kubectl 1.36.x teaching target (1.29+ minimum) and the argocd CLI 3.5.x teaching target (3.0+ minimum).

Scenario

A platform team uses ECR in us-east-1 as the primary registry. On 2026-08-22 at 14:15 UTC, the primary registry becomes unreachable due to an AWS regional incident. The team’s production deploys fail with ImagePullBackOff. The on-call engineer detects the outage via the synthetic check, fails over to the us-west-2 replica, and restores the cluster to a working state.

Tasks

Task 1 — Build the multi-region topology

# check-shell-blocks: allow-invalid
LAB="$HOME/registry-outage-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

cat > multi-region-topology.md <<'EOF'
# Multi-region registry topology

This document is the canonical description of the
team's OCI registry fleet. The topology is designed
for high availability: a regional outage of the
primary registry does not block production deploys.

## Components

| Component | Region | Purpose | RPO | RTO |
|-----------|--------|---------|-----|-----|
| ECR primary | us-east-1 | Active registry; all new pushes go here | 0 (write) | (target) |
| ECR replica | us-west-2 | Replicated registry; read-only on the team's path | 15 min (replication lag) | 30 min (failover) |
| S3 immutable cache (primary) | us-east-1 | Versioned object storage; every image archived | 0 (write) | 60 min (cache restore) |
| S3 immutable cache (replica) | us-west-2 | Replicated S3; cross-region disaster recovery | 15 min (replication lag) | 60 min (cache restore) |
| Cluster pull secret (primary) | us-east-1 | EKS production cluster | n/a | 0 (already configured) |
| Cluster pull secret (replica) | us-west-2 | EKS standby cluster (if multi-region) | n/a | 0 (already configured) |

## Replication

- ECR primary → ECR replica: cross-region
  replication, asynchronous, 5-15 min lag.
- S3 primary → S3 replica: cross-region replication,
  asynchronous, 15 min lag.
- ECR primary → S3 primary: image archiving via
  Lambda on push, synchronous within the region.

## Pull paths

The cluster's `imagePullSecrets` reference the
primary ECR. The team's CI pipeline pushes to the
primary ECR. The cluster's manifests reference the
primary ECR (`<acct>.dkr.ecr.us-east-1.amazonaws.com/web:1.2.3`).

The failover runbook updates the cluster's
`imagePullSecrets` and the manifests' `image:`
references to the replica ECR
(`<acct>.dkr.ecr.us-west-2.amazonaws.com/web:1.2.3`).

## Cache path

If both ECR regions are unavailable, the on-call
engineer restores images from the S3 immutable cache.
The cache is queried by digest; the restored image is
pushed to whichever registry is available.

## Verification

- `crane catalog <primary>` returns the repository
  list.
- `crane catalog <replica>` returns the same
  repository list (after replication lag).
- `aws s3 ls s3://<cache-bucket>/web/` returns the
  archived images.

EOF

git add multi-region-topology.md
git commit -m 'registry: multi-region topology'

The topology document is the canonical reference. The table is the RPO/RTO for each component; the replication section is the data flow; the pull paths are the cluster’s access pattern.

Task 2 — Build the cross-region replication configuration

# check-shell-blocks: allow-invalid
cd "$HOME/registry-outage-lab"

cat > cross-region-replication.md <<'EOF'
# Cross-region replication configuration

This document is the canonical reference for the
cross-region replication between the primary and
replica registries. The replication is the
defence-in-depth that makes the failover possible.

## ECR replication

The team's ECR registry in `us-east-1` is configured
with cross-region replication rules:

```json
{
  "rules": [
    {
      "destinations": [
        {
          "region": "us-west-2",
          "registryId": "123456789012"
        }
      ],
      "repositoryFilters": [
        {
          "filter": "web",
          "filterType": "PREFIX_MATCH"
        },
        {
          "filter": "api",
          "filterType": "PREFIX_MATCH"
        }
      ]
    }
  ]
}

The rule replicates the web and api repositories to us-west-2. The replication is asynchronous; the typical lag is 5-15 minutes.

S3 replication

The team’s S3 bucket (s3://runbook-deploy-artifacts) is configured with cross-region replication:

{
  "Role": "arn:aws:iam::123456789012:role/s3-replication-role",
  "Rules": [
    {
      "ID": "replicate-to-us-west-2",
      "Status": "Enabled",
      "Prefix": "",
      "Destination": {
        "Bucket": "arn:aws:s3:::runbook-deploy-artifacts-replica",
        "StorageClass": "STANDARD"
      }
    }
  ]
}

The rule replicates every object to the replica bucket in us-west-2. S3 versioning is enabled on both buckets; the immutable cache is the versioned object history.

ECR → S3 archiving

The team has a Lambda function that archives every ECR push to the S3 immutable cache. The Lambda is triggered by ECR event notifications; the function:

Receives the ECR push event (image digest, repo, tag).
Pulls the image from ECR via crane.
Pushes the image to S3 with the digest as the key.
Logs the archive in CloudWatch.

The function is deployed via Terraform; the configuration is in the team’s IaC repository.

Verification

ECR replication: aws ecr describe-replication-configuration returns the rule.
S3 replication: aws s3api get-bucket-replication returns the rule.
Lambda: aws lambda get-function --function-name ecr-archive returns the function.

EOF

git add cross-region-replication.md git commit -m ‘registry: cross-region replication’


The replication configuration is the data flow. The
ECR replication is the registry-level copy; the S3
replication is the immutable cache; the Lambda is
the bridge between the two.

### Task 3 — Build the synthetic check

```bash
# check-shell-blocks: allow-invalid
cd "$HOME/registry-outage-lab"

cat > synthetic-check.yaml <<'EOF'
# synthetic-check.yaml — the synthetic check for the
# ECR primary registry. The check runs every minute
# from the team's monitoring cluster in us-east-1.
#
# The check performs four operations:
#   1. GET /v2/ (the OCI Distribution API root)
#   2. GET /v2/web/tags/list (the repository's tag list)
#   3. HEAD /v2/web/manifests/latest (the latest manifest)
#   4. POST a token request (the auth endpoint)
#
# If any operation fails or returns 5xx, the check
# fails and PagerDuty is paged.

apiVersion: v1
kind: ConfigMap
metadata:
  name: registry-synthetic-check
  namespace: monitoring
data:
  check.sh: |
    #!/usr/bin/env bash
    set -euo pipefail

    REGISTRY="\${REGISTRY:-123456789012.dkr.ecr.us-east-1.amazonaws.com}"
    REPO="\${REPO:-web}"
    TAG="\${TAG:-latest}"

    # 1. GET /v2/
    curl -fsS "https://${REGISTRY}/v2/" >/dev/null

    # 2. GET /v2/<repo>/tags/list
    AUTH="$(aws ecr get-login-password --region us-east-1)"
    curl -fsS -H "Authorization: Bearer ${AUTH}" \
      "https://${REGISTRY}/v2/${REPO}/tags/list" >/dev/null

    # 3. HEAD /v2/<repo>/manifests/<tag>
    curl -fsS -I -H "Authorization: Bearer ${AUTH}" \
      -H "Accept: application/vnd.oci.image.manifest.v1+json" \
      "https://${REGISTRY}/v2/${REPO}/manifests/${TAG}" >/dev/null

    echo "OK: registry is reachable"
EOF

# The check is run by the team's monitoring stack
# (Prometheus blackbox exporter or a Kubernetes CronJob).
# The CronJob runs every minute.
cat > synthetic-check-cronjob.yaml <<'EOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: registry-synthetic-check
  namespace: monitoring
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: check
            image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/curl:8.10
            command: ["/check.sh"]
            envFrom:
            - configMapRef:
                name: registry-synthetic-check
          restartPolicy: OnFailure
EOF

git add synthetic-check.yaml synthetic-check-cronjob.yaml
git commit -m 'monitoring: synthetic check for ECR primary'

The synthetic check is the early-warning system. The CronJob runs every minute; the check exercises the four critical operations; a failure pages the on-call engineer.

Task 4 — Build the failover runbook

# check-shell-blocks: allow-invalid
cd "$HOME/registry-outage-lab"

cat > failover-runbook.md <<'EOF'
# Failover runbook: OCI registry

This runbook is the on-call engineer's reference for
failing over from the primary registry to the
replica when the primary is unreachable. The runbook
covers the staged failover: detect, validate, update,
verify.

## When to use this runbook

Use this runbook when:

- The primary OCI registry is unreachable (HTTP 5xx,
  timeout, or DNS failure).
- Production deploys are failing with
  `ImagePullBackOff`.
- The synthetic check (Task 3) is failing for more
  than 5 minutes.

Do not use this runbook when:

- The replica is also unavailable. In that case,
  follow the cache restore procedure (Task 6).
- The failure is a single repository. In that case,
  investigate the repository (corrupt manifest,
  missing tag, etc.).
- The failure is a transient error (network blip,
  certificate renewal). Wait 5 minutes and re-check.

## Phase 1: detect and validate

The on-call engineer confirms the primary is
unreachable:

crane catalog 123456789012.dkr.ecr.us-east-1.amazonaws.com


If the command times out or returns an error, the
primary is unreachable. The engineer confirms the
replica is reachable:

crane catalog 123456789012.dkr.ecr.us-west-2.amazonaws.com


If the replica is reachable, the failover proceeds.

## Phase 2: check replication lag

The on-call engineer checks the replication lag. The
replica may not have the latest images; the engineer
identifies the most recent image that is in both
regions:

Compare the manifest lists.

PRIMARY=”$(crane manifest 123456789012.dkr.ecr.us-east-1.amazonaws.com/web:1.2.3)” REPLICA=”$(crane manifest 123456789012.dkr.ecr.us-west-2.amazonaws.com/web:1.2.3)” if [ “$PRIMARY” = “$REPLICA” ]; then echo “replica is up to date” else echo “replica is behind; check the replication lag” fi


If the replica is behind, the engineer checks the
replication lag:

aws ecr describe-replication-configuration


The lag is typically 5-15 minutes; the engineer
waits for the replica to catch up.

## Phase 3: update the cluster

The on-call engineer updates the cluster's pull
secrets and the manifests' `image:` references.

For the pull secrets:

kubectl create secret docker-registry registry-credentials
—docker-server=123456789012.dkr.ecr.us-west-2.amazonaws.com
—docker-username=AWS
—docker-password=”$(aws ecr get-login-password —region us-west-2)”
—namespace=production
—dry-run=client -o yaml | kubectl apply -f -


For the manifests (via Git PR):

before

image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/web:1.2.3

after

image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/web:1.2.3


The PR is opened, reviewed, and merged. Argo CD
detects the change and syncs.

## Phase 4: verify

The on-call engineer verifies the failover:

Check the Pods.

kubectl get pods -n production

All Pods should be Running.

Check the image pull.

kubectl describe pod <pod-name> -n production | grep “Pulled”

The image should be pulled from us-west-2.

Check the synthetic check.

crane manifest 123456789012.dkr.ecr.us-west-2.amazonaws.com/web:1.2.3

Should return the manifest.


## Phase 5: monitor

The on-call engineer monitors the cluster for 30
minutes after the failover:

- Pod restarts: should be zero.
- Image pull errors: should be zero.
- Application health: should be `Healthy`.

If any metric is anomalous, the engineer investigates.

## Post-incident: fail back

When the primary is restored (the AWS incident is
resolved), the engineer fails back:

1. Verify the primary is reachable.
2. Wait for the replica to replicate the primary's
   new images.
3. Update the manifests to reference the primary.
4. Open a PR; merge; let Argo CD sync.

EOF

git add failover-runbook.md
git commit -m 'registry: failover runbook'

The failover runbook is the on-call reference. The five phases (detect, validate, update, verify, monitor) are the spine; the crane and kubectl commands are the verbs.

Task 5 — Build the failover script

# check-shell-blocks: allow-invalid
cd "$HOME/registry-outage-lab"

cat > failover-script.sh <<'EOF'
#!/usr/bin/env bash
#
# failover-script.sh — fail over from the primary
# registry to the replica.
#
# Required: kubectl; aws CLI; jq; crane.
#
# Usage: PRIMARY_REGISTRY=... REPLICA_REGISTRY=... \
#        REPLICA_REGION=us-west-2 NAMESPACE=production \
#        ./failover-script.sh

set -euo pipefail

: "${PRIMARY_REGISTRY:?PRIMARY_REGISTRY is required}"
: "${REPLICA_REGISTRY:?REPLICA_REGISTRY is required}"
: "${REPLICA_REGION:?REPLICA_REGION is required}"
NAMESPACE="\${NAMESPACE:-production}"
OPERATOR="\${OPERATOR:-$(whoami)}"

NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

echo "=== failover from ${PRIMARY_REGISTRY} to ${REPLICA_REGISTRY} ==="
echo "  now: ${NOW}"
echo "  operator: ${OPERATOR}"

# Step 1: confirm the primary is unreachable.
if crane catalog "$PRIMARY_REGISTRY" >/dev/null 2>&1; then
  echo "ERROR: primary is reachable; failover not needed" >&2
  exit 1
fi
echo "primary is unreachable"

# Step 2: confirm the replica is reachable.
if ! crane catalog "$REPLICA_REGISTRY" >/dev/null 2>&1; then
  echo "ERROR: replica is unreachable; follow the cache restore procedure" >&2
  exit 1
fi
echo "replica is reachable"

# Step 3: update the pull secret.
PASSWORD="$(aws ecr get-login-password --region "$REPLICA_REGION")"
kubectl create secret docker-registry registry-credentials \
  --docker-server="$REPLICA_REGISTRY" \
  --docker-username=AWS \
  --docker-password="$PASSWORD" \
  --namespace="$NAMESPACE" \
  --dry-run=client -o yaml | kubectl apply -f -
echo "pull secret updated"

# Step 4: restart the Deployments to pick up the new
# pull secret.
for dep in $(kubectl get deployment -n "$NAMESPACE" \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "restarting deployment $dep"
  kubectl rollout restart deployment "$dep" -n "$NAMESPACE"
done

# Step 5: wait for the Pods to be ready.
for dep in $(kubectl get deployment -n "$NAMESPACE" \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "waiting for $dep to be ready"
  kubectl rollout status deployment "$dep" -n "$NAMESPACE" \
    --timeout=300s
done

echo "failover complete; all deployments are ready"
EOF

chmod +x failover-script.sh

git add failover-script.sh
git commit -m 'registry: failover script'

The script automates the failover. It confirms the primary is unreachable, confirms the replica is reachable, updates the pull secret, restarts the Deployments, and waits for the Pods to be ready.

Task 6 — Build the cache restore script

# check-shell-blocks: allow-invalid
cd "$HOME/registry-outage-lab"

cat > cache-restore-script.sh <<'EOF'
#!/usr/bin/env bash
#
# cache-restore-script.sh — restore images from the
# S3 immutable cache when both ECR regions are
# unavailable.
#
# The script pulls the image blob from S3, reconstructs
# the OCI manifest, and pushes to a working registry.
#
# Required: aws CLI; crane; jq.
#
# Usage: CACHE_BUCKET=s3://runbook-deploy-artifacts \
#        TARGET_REGISTRY=ghcr.io/runbook-academy \
#        REPO=web DIGEST=sha256:abc123... \
#        ./cache-restore-script.sh

set -euo pipefail

: "${CACHE_BUCKET:?CACHE_BUCKET is required}"
: "${TARGET_REGISTRY:?TARGET_REGISTRY is required}"
: "${REPO:?REPO is required}"
: "${DIGEST:?DIGEST is required}"

NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

echo "=== restoring ${REPO}@${DIGEST} from ${CACHE_BUCKET} ==="
echo "  now: ${NOW}"

# Step 1: find the image archive in S3.
KEY="\${REPO}/\${DIGEST}/image.tar"
if ! aws s3 ls "${CACHE_BUCKET}/${KEY}"; then
  echo "ERROR: image not found in cache" >&2
  exit 1
fi

# Step 2: download the image archive.
WORK="$(mktemp -d)"
trap "rm -rf $WORK" EXIT
aws s3 cp "${CACHE_BUCKET}/${KEY}" "${WORK}/image.tar"

# Step 3: load the image into the local Docker daemon.
docker load -i "${WORK}/image.tar"

# Step 4: tag and push to the target registry.
docker tag "${REPO}@${DIGEST}" \
  "${TARGET_REGISTRY}/${REPO}@${DIGEST}"
docker push "${TARGET_REGISTRY}/${REPO}@${DIGEST}"

echo "image restored: ${TARGET_REGISTRY}/${REPO}@${DIGEST}"
EOF

chmod +x cache-restore-script.sh

git add cache-restore-script.sh
git commit -m 'registry: cache restore script'

The cache restore script is the last resort. When both ECR regions are unavailable, the engineer restores the image from S3 and pushes it to a working registry (GHCR is a common fallback).

Task 7 — Run the failover

cd "$HOME/registry-outage-lab"

PRIMARY_REGISTRY=123456789012.dkr.ecr.us-east-1.amazonaws.com
REPLICA_REGISTRY=123456789012.dkr.ecr.us-west-2.amazonaws.com
REPLICA_REGION=us-west-2
NAMESPACE=production
OPERATOR=jane.doe

PRIMARY_REGISTRY="$PRIMARY_REGISTRY" \
  REPLICA_REGISTRY="$REPLICA_REGISTRY" \
  REPLICA_REGION="$REPLICA_REGION" \
  NAMESPACE="$NAMESPACE" \
  OPERATOR="$OPERATOR" \
  ./failover-script.sh || echo "failover reported errors (expected in simulation)"

The failover script runs through all five steps. The output is captured in Task 8’s evidence bundle.

Task 8 — Capture the recovery evidence

# check-shell-blocks: allow-invalid
cd "$HOME/registry-outage-lab"

cat > recovery-evidence.md <<'EOF'
# Recovery evidence: registry outage — 2026-08-22

This document is the canonical evidence bundle for
the failover performed on 2026-08-22. The bundle
captures the detection, the failover actions, the
verification, and the post-failover state.

## Detection

- 14:15 UTC: synthetic check failed for
  `123456789012.dkr.ecr.us-east-1.amazonaws.com`.
- 14:16 UTC: PagerDuty page fired (synthetic check
  failure).
- 14:17 UTC: on-call engineer (jane.doe)
  acknowledged.
- 14:18 UTC: confirmed the primary is unreachable
  (`crane catalog` timed out).
- 14:20 UTC: confirmed the replica is reachable
  (`crane catalog` succeeded for `us-west-2`).

## Blast radius

- Affected region: `us-east-1`.
- Affected cluster: `production` (EKS in `us-east-1`).
- Affected deploys: all deploys that require a new
  image pull.
- Affected services: `web`, `api`.

## Failover actions

### Phase 1: detect and validate

- 14:18 UTC: confirmed primary unreachable.
- 14:20 UTC: confirmed replica reachable.

### Phase 2: check replication lag

- 14:22 UTC: compared manifest lists for `web:1.2.3`
  in primary and replica; replica was up to date.

### Phase 3: update the cluster

- 14:25 UTC: updated the `registry-credentials` pull
  secret in the `production` namespace to point at
  the replica.
- 14:28 UTC: opened PR #9876 to update the manifests'
  `image:` references.
- 14:32 UTC: PR #9876 merged; Argo CD synced.

### Phase 4: verify

- 14:35 UTC: rolled the `web` Deployment.
- 14:36 UTC: `web` Pods pulled images from
  `us-west-2`.
- 14:38 UTC: `web` is `Synced: True, Healthy: True`.

### Phase 5: monitor

- 14:38-15:08 UTC: monitored the cluster for 30
  minutes.
- Pod restarts: 0.
- Image pull errors: 0.
- Application health: Healthy.

## Post-failover state

- Cluster is pulling from `us-west-2`.
- All Pods are Running.
- All Deployments are Healthy.
- Synthetic check (us-west-2 check) is passing.

## Verification

- `crane catalog` on the replica returns the
  repositories.
- `kubectl get pods` returns Running Pods.
- `argocd app list` shows all apps `Synced: True`.

EOF

git add recovery-evidence.md
git commit -m 'registry: recovery evidence bundle'

The evidence bundle is the canonical record. The detection, the blast radius, the failover actions, the verification, and the post-failover state are the fields the team reviews at the post-incident review.

Task 9 — Build the audit trail

# check-shell-blocks: allow-invalid
cd "$HOME/registry-outage-lab"

cat > audit-trail.md <<'EOF'
# Audit trail: registry outage — 2026-08-22

This document is the audit trail for the failover on
2026-08-22. The trail is the canonical record for
the compliance review; every action is timestamped
and attributed.

## 14:15 — Synthetic check failed

- Source: monitoring stack.
- Check: `crane catalog` on the primary registry.
- Action: alert sent to jane.doe.

## 14:16 — PagerDuty page

- Source: PagerDuty.
- Operator: jane.doe.

## 14:18 — Primary confirmed unreachable

- Operator: jane.doe.
- Action: `crane catalog` timed out; confirmed the
  primary is unreachable.

## 14:20 — Replica confirmed reachable

- Operator: jane.doe.
- Action: `crane catalog` succeeded for the replica
  in `us-west-2`.

## 14:22 — Replication lag checked

- Operator: jane.doe.
- Action: compared manifest lists; replica is up to
  date.

## 14:25 — Pull secret updated

- Operator: jane.doe.
- Action: updated the `registry-credentials` pull
  secret in the `production` namespace.

## 14:28 — PR opened

- Operator: jane.doe.
- Action: opened PR #9876 to update the manifests'
  `image:` references.

## 14:32 — PR merged

- Operator: platform-team.
- Action: reviewed and merged PR #9876.

## 14:35 — Deployment rolled

- Operator: ci-bot.
- Action: rolled the `web` Deployment; Pods pulled
  images from `us-west-2`.

## 14:38 — Verified

- Operator: jane.doe.
- Action: `web` is `Synced: True, Healthy: True`.

## 14:38-15:08 — Monitored

- Operator: jane.doe.
- Action: monitored the cluster for 30 minutes.
- Verification: zero anomalies.

## 15:30 — Post-incident review

- Operators: jane.doe, sre-team.
- Action: reviewed the evidence bundle and the
  audit trail; opened a follow-up PR to improve the
  synthetic check.

EOF

git add audit-trail.md
git commit -m 'registry: audit trail'

The audit trail is the canonical record. Every action is timestamped and attributed.

Task 10 — Validate the deliverables

cd "$HOME/registry-outage-lab"

# Verify the topology has all components.
grep -c "^| " multi-region-topology.md
# expected: 6+ rows

# Verify the runbook has all five phases.
grep -c "^## Phase" failover-runbook.md
# expected: 5

# Verify the scripts pass syntax check.
bash -n failover-script.sh && echo "failover: syntax ok"
bash -n cache-restore-script.sh && echo "cache-restore: syntax ok"

# Verify the synthetic check is in place.
test -s synthetic-check.yaml && echo "synthetic: ok"

# Verify the evidence bundle has all sections.
grep -c "^## " recovery-evidence.md
# expected: 5 (Detection, Blast radius, Failover
#              actions, Post-failover, Verification)

# Verify the audit trail has timestamps.
grep -c "^## [0-9]" audit-trail.md
# expected: 11+ entries

The deliverables are validated.

Task 11 — Capture the deliverables

cd "$HOME/registry-outage-lab"

cp multi-region-topology.md \
   cross-region-replication.md \
   failover-runbook.md \
   synthetic-check.yaml \
   synthetic-check-cronjob.yaml \
   failover-script.sh \
   cache-restore-script.sh \
   recovery-evidence.md \
   audit-trail.md \
   "$HOME/"

ls -l "$HOME"/multi-region-topology.md \
       "$HOME"/failover-runbook.md \
       "$HOME"/failover-script.sh \
       "$HOME"/cache-restore-script.sh \
       "$HOME"/recovery-evidence.md \
       "$HOME"/audit-trail.md

The deliverables are in $HOME/.

Validation

multi-region-topology.md documents the primary, replica, and immutable cache with RPO/RTO.
cross-region-replication.md documents the ECR replication, the S3 replication, and the Lambda archiving.
failover-runbook.md documents all five phases: detect, validate, update, verify, monitor.
synthetic-check.yaml and synthetic-check-cronjob.yaml define the synthetic check and the CronJob.
failover-script.sh is executable, has set -euo pipefail, and automates the failover.
cache-restore-script.sh is executable, has set -euo pipefail, and restores from S3.
recovery-evidence.md captures detection, blast radius, failover actions, post-failover state, and verification.
audit-trail.md has timestamped entries for every action.

Expected Outcome

A multi-region topology, a replication configuration, a failover runbook, a synthetic check, a failover script, a cache restore script, a recovery evidence bundle, and an audit trail.

$HOME/registry-outage-lab/
├── multi-region-topology.md        # the topology
├── cross-region-replication.md     # the replication
├── failover-runbook.md             # the runbook
├── synthetic-check.yaml            # the check
├── synthetic-check-cronjob.yaml    # the CronJob
├── failover-script.sh              # the failover
├── cache-restore-script.sh         # the cache restore
├── recovery-evidence.md            # the canonical record
└── audit-trail.md                  # the audit trail

The topology is the design; the replication is the data flow; the runbook is the on-call reference; the scripts are the verbs; the evidence and the audit trail are the institutional knowledge.

Troubleshooting

The replica is also unavailable. Follow the cache restore procedure (Task 6). The script pulls from S3 and pushes to a working registry.

The replication lag is longer than expected. The team’s RPO is 15 minutes. If the lag is longer, the team investigates the ECR replication configuration.

The synthetic check is failing but the registry is reachable. The check may be testing an authenticated endpoint without a valid token. Verify the AWS CLI authentication and the ECR login.

The failover script times out on rollout. The Pods may be stuck in ImagePullBackOff because the image is not in the replica. Check the replication lag and the manifest list.

The cache restore script fails with “image not found”. The image may not have been archived. Verify with aws s3 ls s3://<bucket>/<repo>/.

Cleanup

LAB="$HOME/registry-outage-lab"

cp -r "$LAB"/* "$HOME"/ 2>/dev/null
rm -rf "$LAB"

# Revert the failover: update the pull secret back to
# the primary.
kubectl create secret docker-registry registry-credentials \
  --docker-server=123456789012.dkr.ecr.us-east-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password="$(aws ecr get-login-password --region us-east-1)" \
  --namespace=production \
  --dry-run=client -o yaml | kubectl apply -f -

If the kind cluster is no longer needed, delete it:

kind delete cluster --name argocd-lab

What You Learned

The topology is the design. A multi-region registry with cross-region replication and an immutable cache is the defence-in-depth that makes the failover possible.
The RPO and RTO drive the topology. The replication lag (5-15 min) is the RPO; the failover time (30 min) is the RTO. The team reviews the targets quarterly.
The synthetic check is the early warning. A check that runs every minute from the same region as the cluster catches the outage within 1 minute. The team also has a check in the replica region for defence-in-depth.
The failover is staged. Detect, validate, update, verify, monitor. The order minimises the time-to-recovery and the risk of cascading failures.
The cache restore is the last resort. When both ECR regions are unavailable, the S3 immutable cache is the only way to recover images. The cache is versioned and replicated across regions.
The audit trail is the canonical record. Every action is timestamped and attributed. The trail is the answer to “who did what, when?”.
The failover is rehearsed quarterly. A failover runbook that has never been tested is a liability. The team rehearses the failover quarterly in a non-production environment.

Lab 27: Recover from an artifact registry outage — multi-region, immutable cache

Objectives

Prerequisites