Annualci-cd-dr

Pipeline Disaster Recovery Readiness Checklist

16 items ·11 critical ·5 warn ·0 info

How to use this checklist

This is the gate the CI/CD system must pass before the team is allowed to claim that production can be re-deployed after a disaster. It is deliberately about the pipeline itself: the pipelines that deploy your applications are themselves a system that needs a backup, a recovery plan, and a drill.

A team that has never run a full drill will discover, during the real incident, that the part they trusted was the part that failed. Run this checklist annually at minimum, and treat any partial drill as a finding, not as a pass.

Sign-off

Every critical item must pass. A failing critical item blocks the DR declaration of readiness; it is not a note for later. Record the date, the reviewer, and the disposition of every item that did not pass.

Critical11 items

The RTO and RPO for the CI/CD system are defined and signed off by the platform owner. A pipeline that has no stated recovery target cannot be considered "recovered".
Source code can be recovered from somewhere other than the primary git host: a mirror, a backup, an off-platform export. The recovery procedure has actually been executed, not just documented.
```
git clone &lt;mirror-url&gt; /tmp/recovered-repo && cd /tmp/recovered-repo && git log --oneline -5 && git tag --list | head
```
Pipeline configuration (workflow files, GitLab CI YAML, Jenkinsfile, Tekton tasks, Argo CD Applications, Flux Kustomizations) is backed up independently of the source repos. The configuration is itself part of the recovery package.
Self-hosted runners can be rebuilt from source (IaC: Terraform, Ansible, Helm, cloud-init). Rebuilding does not require an engineer hand-clicking through a console. The IaC lives in a repo that is itself backed up.
The secret manager (Vault, AWS Secrets Manager) can be restored from a backup, or re-seeded from an offline store. A secret manager that can only be repopulated by hand is a secret manager that has not been DR-tested.
The artifact registry is replicated or backed up (cross-region, cross-cloud, or to object storage). A registry that lives only in one region has a regional DR test as its first real recovery.
The GitOps controller cluster (Argo CD, Flux) can be rebuilt from manifests. Applications re-sync automatically once the controller is healthy and the source repos are reachable. The re-sync has been observed in a drill.
Signing keys (Cosign KMS keys, Fulcio roots, Rekor instance, repository deploy keys) are recoverable from a separate, controlled store. A lost signing key invalidates every artefact ever signed with it.
A documented order of operations exists: source → runners → secret manager → registry → controller → clusters. The order is enforced by the recovery playbook, not improvised by the responder.
A full end-to-end DR drill has been performed in the last 12 months: source rebuilt, runners rebuilt, controller restored, an artifact re-signed, deployed to a clean cluster. A drill that stops at "the data is backed up" is a partial drill.
The restore procedure can be executed by someone other than the author. The current author could be on holiday, off-network, or part of the incident. A runbook that only the author can follow is a single point of failure.

Warning5 items

OIDC trust relationships between CI and cloud providers can be re-established. The trust policies are exported, the issuer URLs are documented, and the re-establishment has been rehearsed.
If the primary CI region is lost, the secondary region can take over. The failover is tested by actually failing over, not by reading the runbook. A failover that has not been run will fail.
The behaviour of in-flight pipelines during a DR event is defined: are they abandoned, are they resumable, are they rolled back to a known state? "We'll figure it out" is not a behaviour.
Audit logs, pipeline history, and signed artefacts from before the disaster are preserved through the recovery. A pipeline that recovers to "the last release we can prove shipped" is more useful than one that recovers to "we think it's fine".
A DR drill produces a post-mortem-style review: what worked, what failed, what was missed, what needs to change. The review is filed and the actions are tracked to completion before the next drill.