The registry drill — the quarterly rehearsal

A registry DR plan that has never been executed is a hypothesis. The drill is the proof that the runbook works. The drill converts the runbook from a document into a tested capability, and the tested capability is the difference between an outage the team recovers from and an outage the team learns from in production.

flowchart LR
    A["quarterly drill"] --> B["tabletop"]
    A --> C["live failover"]
    B --> D["runbook reviewed"]
    B --> E["gaps identified"]
    C --> F["timeline recorded"]
    C --> G["RTO measured"]
    C --> H["signature verified"]
    D --> I["updated runbook"]
    E --> I
    F --> I
    G --> I
    H --> I

The drill has two shapes: the tabletop exercise, which is a discussion of the runbook against a scripted scenario, and the live regional failover, which is an actual recovery in a non-production region. The two shapes are complementary: the tabletop catches gaps in the runbook text; the live failover catches gaps in execution.

The tabletop exercise

The tabletop is a meeting. The on-call engineer, the incident commander, and a representative from each team that owns part of the registry path — the platform team that owns the registry, the application team that owns the deploy, the security team that owns the signature store — sit in a room (or a video call) and walk through the registry runbook against a scripted scenario.

The scenario names a disaster — the manifest endpoint is returning 503 in one region, the blob endpoint is returning EOF in another, the signature store has lost its trust chain — and the team walks through the runbook step by step. For each step, the team states who acts and what they act on.

# Tabletop scenario script (excerpt)
echo "Manifest endpoint is returning 503 in us-east-1.
The kubelet in us-west-2 is failing pulls.
The /v2/ endpoint from a direct probe is 200.
The CDN in front of the registry reports no degraded origin."

The live regional failover

The live drill is a real recovery in a non-production region. The drill takes the primary registry offline in the non-production region, promotes the cross-region replica, reconfigures the kubelet, and verifies that pulls succeed.

# 1. Take the primary offline in the non-production region
ssh registry-primary-drill "systemctl stop harbor"

# 2. Verify the replica is current
DIGEST=$(crane manifest digest registry.replica-drill.example.com/app:$TAG)
echo "Replica digest: $DIGEST"

# 3. Promote the replica: reconfigure the kubelet
kubectl -n kube-system create configmap registry-config-drill \
    --from-literal=registry=registry.replica-drill.example.com \
    --dry-run=client -o yaml | kubectl apply -f -

# 4. Restart the kubelet on each node to flush the connection pool
kubectl rollout restart daemonset/kubelet-config -n kube-system

# 5. Verify reachability, digest presence, signature trust
curl -fsS https://registry.replica-drill.example.com/v2/
docker pull registry.replica-drill.example.com/app:$DIGEST
cosign verify --key cosign.pub \
    registry.replica-drill.example.com/app:$DIGEST

The verification is the proof that the failover works. Reachability, digest presence, and signature trust are the three artefacts; the elapsed time from the first command to the last cosign verify is the RTO measurement.

Measuring the failover against the RTO

The drill records the elapsed time from the first “registry offline” command to the last “cosign verify” command. The elapsed time is the actual RTO; the team’s RTO design constraint is the target.

A team whose RTO design constraint is sixty minutes and whose drill measures forty-five minutes has headroom. A team whose drill measures seventy minutes has a gap: the runbook steps need trimming or the RTO design constraint needs relaxing. The measurement is the basis of the gap list.

# The drill timeline
echo "T0: primary offline"
date -u +%FT%TZ
echo "T1: replica verified"
date -u +%FT%TZ
echo "T2: kubelet reconfigured"
date -u +%FT%TZ
echo "T3: signature verified"
date -u +%FT%TZ

The four artefacts the drill must produce

The drill is not complete until four artefacts are written:

1. The timeline. The moment-by-moment record of every command run, every observation made, and every verification passed. The timeline is the basis of the post-drill review and the rollback evidence if the drill reveals a gap.
2. The RTO measurement. The elapsed time from the first command to the last verification, compared against the RTO design constraint. The measurement is the proof that the runbook works or the proof that it does not.
3. The gap list. The list of every step that took longer than expected, every command that needed a fallback, and every assumption that turned out to be wrong. The gap list is the input to the runbook update.
4. The updated runbook. The runbook with the gaps closed. A runbook that has been drilled but not updated is a runbook with stale steps.

Cadence and scope

The drill is quarterly. The cadence is short enough that the runbook stays current with the team’s infrastructure (registries are added and removed, replication policies change, signature stores rotate keys) and long enough that the drill is a meaningful event, not a routine check-in.

The scope is every failure mode the runbook covers: manifest endpoint (Part XCIX-01), blob endpoint, auth endpoint, write API, replication lag (Part XCIX-05), storage full, signature-store loss. A drill that covers only the manifest endpoint leaves the other three scopes unrehearsed.

Production discipline

1. The drill is quarterly. A runbook that has not been drilled in a year is a hypothesis.
2. The drill is live, not just tabletop. The tabletop catches runbook gaps; the live drill catches execution gaps.
3. The drill produces four artefacts. The timeline, the RTO measurement, the gap list, the updated runbook.
4. The drill covers every scope from Part XCIX-01. A drill that covers only the manifest endpoint leaves the other scopes unrehearsed.
5. The signature verification is part of the drill. The signature store is the step the drill catches most often as a gap.

Cross-course references

• Git, CI/CD & GitOps — Part XCIX-01 (The Registry Failure Scenario) is the scenario library the tabletop draws from.
• Git, CI/CD & GitOps — Part XCIX-03 (Registry Replication) is the recovery path the live failover exercises.
• Git, CI/CD & GitOps — Part XCVII-06 (The Disaster Recovery Drill) is the parent pattern this lesson specialises for the registry.

Quiz