The CI/CD DR question — what fails, what survives, what to rebuild

A CI/CD disaster is the loss of the system that builds, tests, and deploys your software. It is not the loss of the source code (Git repositories survive if a clone is intact) or the production workloads (a cluster redeploys from Git). It is the loss of the machinery between commit and deploy: the control plane that schedules workflows, the runners that execute jobs, the secret store that hands credentials, and the artifact registry that holds the images and binaries.

flowchart LR
    A["CI/CD disaster event"] --> B["control plane lost"]
    A --> C["runners lost"]
    A --> D["secret store lost"]
    A --> E["artifact registry lost"]
    B --> F["workflows cannot schedule"]
    C --> G["scheduled workflows cannot run"]
    D --> H["jobs cannot authenticate"]
    E --> I["deploys cannot pull artifacts"]

The DR question every team must answer before the incident is: what fails, what survives, what to rebuild. The three categories drive different recovery paths.

What survives

The source repositories are usually the most resilient piece of the system. Git is distributed: every clone is a full backup. The repository survives the loss of the forge if at least one mirror, fork, or local clone is intact. The Terraform that provisioned the CI infrastructure lives in a repository of its own. The Dockerfile for the runner image lives in the same repository as the workflows.

What survives a regional outage of the cloud provider that runs the hosted control plane: the repositories, the local clones on developer laptops, the IaC that recreates the runners, the secret values held in the password manager.

What fails

The runners — the VMs or containers that execute workflows — are usually the first thing lost in a regional outage. Self-hosted runners are physical or virtual infrastructure; if the region is gone, the runners are gone. Hosted runners are shared across all tenants of the control plane, and the loss of the control plane takes the hosted runners with it.

The artifact registry — Docker Hub, GHCR, GitLab Container Registry, a private Harbor — is the second thing that fails. An artifact registry that is not mirrored to a second region is a single point of failure for every deployment that pulls from it.

The secret store is the third. The control plane’s secret store (GitHub Actions Secrets, GitLab CI Variables) lives in the control plane and is lost with it. An external secret store (Vault, AWS Secrets Manager) survives if its region survives; it does not survive a regional outage.

What to rebuild

The recovery order is determined by the dependency graph:

1. Control plane first. The orchestrator must exist before anything it schedules can run.
2. Runners second. The runners register against the control plane; without runners, no job executes.
3. Secret store third. Jobs cannot pull credentials until the store is reachable.
4. Artifact registry fourth. Deploys need artifacts; the artifacts must exist before a deploy can pull them.

Production discipline

Three rules that recur in every DR plan:

1. RTO and RPO are written before the incident. Recovery Time Objective (how long the team accepts the system being down) and Recovery Point Objective (how much data the team accepts losing) are design constraints, not incident decisions.
2. Backups live in a different region than the system they back up. A backup that shares a region with the system it backs up is lost in the same outage.
3. The DR runbook is rehearsed quarterly. A runbook that has never been executed is a hypothesis.

Cross-course references

• Git, CI/CD & GitOps — Part XCIV-01 (The Incident Arrives) covers the first ten minutes of a secret-leak incident, the same triage shape that opens a CI/CD DR.
• Container Security for Production Sysadmins — Part VII (Registry Backup) covers the registry backup mechanics in detail.
• Terraform for Production Sysadmins — Part XXI (State Backup) covers the state-backup pattern the control-plane rebuild relies on.

Quiz