Recovering the artifact registry — the backup plan

The artifact registry is the system that holds the container images and binaries the workflows produce and the deployments pull. A CI/CD disaster that destroys the registry removes the deploys’ ability to pull the artifacts. The recovery is the fourth step, after the control plane, the runners, and the secret store.

flowchart LR
    A["registry recovery"] --> B["replicate from peer"]
    A --> C["restore from backup"]
    A --> D["rebuild from source"]
    A --> E["pull from upstream"]
    B --> F["images present"]
    C --> F
    D --> F
    E --> F
    F --> G["tag immutability verified"]
    G --> H["signature trust verified"]
    H --> I["deploys resume"]

The recovery path depends on what survived the outage: a registry with a peer in another region recovers via replication; a registry with a backup recovers via restore; a registry with neither rebuilds from source or pulls from upstream.

Replication

A replicated registry — Harbor with a replication target in another region, ECR with cross-region replication enabled, GHCR with a multi-region mirror — recovers by promoting the peer to primary in the recovery region.

# Harbor replication
harbor-cli replication-policy update --name $POLICY --enabled true

# ECR replication
aws ecr put-replication-configuration \
    --replication-configuration file://replication.json

The replication lag defines the RPO: artifacts pushed within the lag window are present in the peer; artifacts pushed beyond the lag window are not. The DR runbook states the replication lag as a recovery constraint.

Restore from backup

A backed-up registry — a Harbor instance with a nightly database and blob backup to S3, an ECR with a cross-region snapshot — recovers by restoring the backup into a fresh registry in the recovery region.

The restore:

1. Provision a fresh registry in the recovery region.
2. Restore the database from the backup.
3. Restore the blob storage from the backup.
4. Verify the restored artifacts against the manifest.

Rebuild from source

A registry with no peer and no backup rebuilds from source: the workflows rerun, the images are pushed to a fresh registry, and the deploys pull from the fresh registry.

The rebuild from source is the slowest path. The recovery time is bounded by the time to rebuild every artifact, which can be hours for a large image matrix.

Pull from upstream

A registry that holds upstream images (base images, language runtimes) can pull from the upstream registry in the recovery region. The upstream registry must be reachable from the recovery region; an air-gapped region cannot pull from upstream.

The pull from upstream is the path for base images. The team’s own images are recovered via replication, backup, or rebuild.

Verifying the recovery

Three artefacts must be verified before deploys resume:

1. Image presence. Every image the team relies on exists in the recovered registry.
2. Tag immutability. The tags the team relies on still resolve to the digest the team pinned.
3. Signature trust chain. The signatures still validate.

Production discipline

1. The backup lives in a different region from the registry. Cross-region backup replication is the default; same-region backup is not a backup.
2. Tag immutability is verified before deploys resume. A mutable tag is a tag the next incident will not recognise.
3. The signature trust chain is verified before deploys resume. No signatures, no trust.

Cross-course references

• Git, CI/CD & GitOps — Part XCVII-03 (Recovering Runners) covers the runner rebuild that pushes artifacts to the registry.
• Container Security for Production Sysadmins — Part VII (Registry Backup) covers the registry backup.
• Terraform for Production Sysadmins — Part XXI (State Backup) covers the cross-region pattern.

Quiz