Registry replication — the multi-region strategy

A registry that lives in a single region is a single point of failure for every deployment in every other region. The multi-region strategy is to replicate the registry across two or more regions so that a regional outage of the primary does not freeze the deployments in the recovery region. The replication has three shapes — pull-through caches, cross-region replicas, and active-active registries — each with a different recovery path and a different cost.

flowchart LR
    A["primary region"] --> B["cross-region replica"]
    B --> C["recovery region"]
    A --> D["pull-through cache"]
    D --> E["edge cluster"]
    A --> F["active-active peer"]
    F --> G["peer region"]
    C --> H["failover flip"]
    E --> I["low-latency pull"]
    G --> J["writes to either"]

Pull-through caches

A pull-through cache sits between the cluster and the primary registry. The cluster pulls from the cache; the cache pulls from the primary on a miss and serves the cached blob on subsequent hits. The cache is in the same region as the cluster; the primary is in a second region.

The pull-through cache reduces latency for clusters that are far from the primary. It does not, on its own, provide failover: when the primary is down, the cache cannot pull new images, and the cluster cannot pull new images that the cache has not yet cached. The pull-through cache is a latency tool, not a DR tool.

# Configure a pull-through cache on the cluster side
# (registry.k8s.io is a public pull-through cache; private
# caches are configured via the Distribution registry
# proxy middleware)
docker pull registry.cache.example.com/app:$TAG

Cross-region replicas

A cross-region replica is a second registry in a second region that mirrors the primary. The replica is kept current by a replication policy — Harbor’s replication policy, ECR’s cross-region replication rule, GHCR’s multi-region mirror — that copies every push from the primary to the replica.

The recovery path is promotion: when the primary is down, the replica is promoted to primary in the recovery region, and the cluster’s kubelet is reconfigured to pull from the replica. The promotion is a configuration change (secret, ConfigMap, registry mirror) plus a pod restart to flush the kubelet’s connection pool.

# Verify the replica is current before the failover flip
crane manifest digest registry.replica.example.com/app:$TAG

# Compare against the primary's last known digest
# (from the most recent push event or from the manifest catalog)

The replication lag defines the RPO. A replica that replicates within thirty seconds has an RPO of up to thirty seconds; a replica that replicates within five minutes has an RPO of up to five minutes. The replication lag is the gap between the last replicated push and the disaster time.

Active-active registries

An active-active deployment runs two registries, one in each region, both writable. The cluster is configured to push to whichever registry is reachable and pull from whichever registry returns the digest. The recovery path is automatic: a regional outage of one registry leaves the other serving.

Active-active is the most expensive shape. It requires conflict resolution for concurrent pushes (the last-writer-wins for tags, digest equality for content), and the chain of trust (cosign signatures, attestations) must be replicated to both registries. The payoff is that the failover is invisible: the cluster never knows the primary was down because the peer was serving the entire time.

Verifying a replica before the failover flip

The replica is verified before the failover flip. The verification confirms three things: the replica is reachable, the replica has the digests the deploys pin, and the replica’s signature store is current.

# 1. Reachability
curl -fsS https://registry.replica.example.com/v2/

# 2. The pinned digest is present
DIGEST="sha256:abc..."
docker pull registry.replica.example.com/app:$DIGEST

# 3. The signature store is current (cosign verify)
cosign verify \
    --key cosign.pub \
    registry.replica.example.com/app:$DIGEST

A replica that passes all three verifications is a replica that can be promoted. A replica that passes reachability and digest presence but fails signature verification is a replica that has blobs but no trust chain — promote it only after the signature store is restored, or re-sign the artifacts from the team’s signing key.

Production discipline

1. Pull-through caches are latency tools; cross-region replicas are DR tools. The two are complementary, not interchangeable.
2. The replication lag is the RPO design constraint. Match the replication mechanism to the RPO, not the other way around.
3. The replica is verified before the failover flip. Reachability, digest presence, and signature trust are the three verifications.
4. The signature store is part of the replica. A replica without signatures is a replica without trust.

Cross-course references

• Git, CI/CD & GitOps — Part XLV-02 (Digests and Content-Addressing) covers the digest verification the failover flip relies on.
• Git, CI/CD & GitOps — Part XCVII-05 (Recovering the Artifact Registry) covers the recovery paths that the replica promotion slots into.
• Container Security for Production Sysadmins — Part VII (Registry Backup) covers the signature-store replication the trust chain depends on.

Quiz