critical riskcluster affecting~180 min

Runbook: Restore CI/CD After a Disaster

1 · Prerequisites

Confirm every item is in place before any state change.

git-cicd-gitops-rb-11-restore-runner-capacity
git-cicd-gitops-rb-20-recover-argocd-controller
git-cicd-gitops-rb-21-recover-git-hosting-dependency
git-cicd-gitops-rb-22-recover-registry-outage
git-cicd-gitops-rb-29-restore-ci-cd-after-disaster
A documented disaster recovery plan with RTO/RPO targets
Backups of the CI/CD control plane state (GitHub Enterprise backup, GitLab backup, Jenkins backup)
Backups of the runner configuration (Packer images, Terraform modules, Ansible playbooks)

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the disaster is real and assess the scope. CI/CD provider status pages: GitHub https://www.githubstatus.com/, GitLab https://status.gitlab.com/, CircleCI https://status.circleci.com/. Self-hosted: check the provider's health endpoint and the team's status page. The scope determines the recovery path: regional outage, provider outage, infrastructure loss, or compromise
· Identify every component of the CI/CD system. Build runners (GitHub Actions runners, GitLab runners, Jenkins agents); control plane (GitHub Enterprise, GitLab, Jenkins master); secrets store (HashiCorp Vault, sealed secrets, external secrets); artifact registry (ghcr.io, ECR, self-hosted); Git hosting (GitHub, GitLab, Gitea); deployment targets (Kubernetes clusters, cloud accounts). The component list is the recovery scope
· Identify the RTO (Recovery Time Objective) and RPO (Recovery Point Objective) for CI/CD. RTO is the maximum acceptable downtime; RPO is the maximum acceptable data loss. The RTO/RPO determine which recovery strategy to use: hot standby, warm standby, or cold rebuild
· Verify backups are available. For GitHub Enterprise: gh-admin backup status. For GitLab: gitlab-backup status. For Jenkins: jenkins-cli list-backups. For the artifact registry: the registry's backup or a mirror. The backup is the source of truth for recovery
· Verify the recovery infrastructure exists. A hot standby region with the CI/CD already running; a warm standby that can be activated; a cold rebuild from IaC. The recovery infrastructure determines the RTO
· Engage the incident response team. A disaster recovery is a multi-team effort. The CI/CD team restores CI/CD; the platform team restores the infrastructure; the security team verifies the recovery is not compromised; the engineering leadership communicates with stakeholders
· Confirm the disaster recovery is authorized. DR is expensive and disruptive; the activation must be authorized by the incident commander or the engineering leadership. A unilateral activation is a process violation

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Declare the disaster and notify all stakeholders. Post to the incident channel: "DR activated. Provider status: <url>. Reason: <disaster>. RTO target: <hours>. RPO target: <hours>. Teams engaged: <list>. ETA for first milestone: <time>". The notification is the audit trail and the coordination mechanism
2STEP 2 - Activate the recovery infrastructure. For hot standby: switch the DNS / load balancer to point at the standby. For warm standby: start the standby (e.g., terraform apply to provision the standby region). For cold rebuild: provision from scratch using IaC (terraform apply, ansible-playbook, kubectl apply). The activation is the first operational step
3STEP 3 - Restore the CI/CD control plane. For GitHub Enterprise: gh-admin restore --from-backup <backup-id>. For GitLab: gitlab-backup restore BACKUP=<timestamp>. For Jenkins: copy the JENKINS_HOME from the backup. The control plane is the brain; it must be restored before runners can register
4STEP 4 - Restore the runner fleet. For GitHub Actions: re-run the runner provisioning (terraform apply for self-hosted runners; contact GitHub for hosted runners). For GitLab: gitlab-runner register against the restored GitLab. For Jenkins: re-provision agents from the backup. The runners register against the restored control plane and begin accepting jobs
5STEP 5 - Restore the secrets store. For Vault: vault operator raft snapshot restore <snapshot>. For external secrets: the secret values are re-pulled from the cloud provider's secret manager. For sealed secrets: re-encrypt with the new cluster's key. The secrets store must be available before the runners can authenticate to deployment targets
6STEP 6 - Restore the artifact registry. For GHCR/ECR/GCR: the registry is usually regional; failover to a mirror (see git-cicd-gitops-rb-22-recover-registry-outage). For self-hosted: restore from the backup (docker load for each image). For Harbor: restore the database and the blob storage from the backup. The registry is required for image-based deploys
7STEP 7 - Restore the GitOps controller. Argo CD: see git-cicd-gitops-rb-20-recover-argocd-controller. Flux: re-apply the CRs from the GitOps repo. The controller must be running and able to reach Git before any production deploys can happen
8STEP 8 - Restore the deployment targets. For Kubernetes clusters: see git-cicd-gitops-rb-27-validate-gitops-after-cluster-recovery for each cluster. For cloud accounts: restore the IAM roles, the security groups, the VPCs (Terraform / CloudFormation / Pulumi). The deployment targets must be reachable from the runners and the GitOps controller
9STEP 9 - Test the end-to-end pipeline with a known-good commit. Push a test commit that triggers the build → test → deploy → verify chain. Use a non-production environment for the first end-to-end test. The test proves the chain is functional
10STEP 10 - Restore production deploys. Once the test passes, enable production workflows. gh workflow enable deploy-prod.yml. The first production deploy should be a known-good artifact (a tag that has been validated and is known to be clean). The first production deploy is a canary — watch closely
11STEP 11 - Validate every CI/CD component against the baseline. For each component: response time, error rate, throughput. The validation produces a checklist of "is this component back to normal?". A component that is "up" but slow is a partial recovery
12STEP 12 - Verify the recovery did not introduce new vulnerabilities. The disaster may have been caused by a compromise (see git-cicd-gitops-rb-28-respond-to-supply-chain-compromise). Every restored component must be scanned and verified. The recovery must not perpetuate the compromise
13STEP 13 - Document the recovery. Open a ticket with: the disaster, the scope, the RTO/RPO achieved vs target, the components restored, the data loss (if any), the operator, the timeline, the post-incident review date. The record is the audit trail and the basis for improving the DR plan

4 · Verification

Confirm the procedure actually fixed the problem.

✓The CI/CD control plane is reachable: gh api user/repos returns the expected response; glab api projects returns the expected list
✓Runners are registered and accepting jobs: gh api orgs/{org}/actions/runners --jq '.runners[] | {name, status, busy}' shows self-hosted runners with status online, or hosted runners show healthy; gitlab-runner list shows runners online
✓A test commit triggers a successful end-to-end pipeline: build → test → deploy to staging → verify. All stages pass
✓The artifact registry is reachable: crane manifest <registry>/<image>:<tag> returns the manifest
✓The GitOps controller reconciles: argocd app list shows Synced + Healthy; flux get kustomization -A shows Ready=True
✓Production deploys resume: a production deploy completes successfully and the application is healthy
✓Every component's response time and error rate returns to baseline. A "working but slow" component is a partial recovery
✓No new vulnerabilities were introduced by the recovery: the security team has signed off on the recovered CI/CD system

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the disaster recovery fails (the backup is corrupted, the warm standby cannot be activated, the cold rebuild hits a dependency that is also down): the RTO is breached. Escalate to the engineering leadership. The team must operate manually until CI/CD is restored
↶If the recovered CI/CD produces a different artifact than the pre-disaster one (the rebuild is not byte-for-byte identical): the recovery is not lossless. Investigate the difference; it may be a benign build-time variable (timestamp) or it may be a security issue (the build is producing different output). Engage the security team if in doubt
↶If the production deploy fails after the recovery: do not assume the failure is a CI/CD problem. The deploy may have exposed a latent application bug. Investigate the application before assuming the CI/CD is broken again
↶If the recovery was activated but the disaster was actually a regional outage that has now resolved: do not switch back immediately. Wait for the original region to be verified healthy for a sustained period (typically 1 hour) before failover back. Premature failover-back can cause data inconsistency
↶If the secrets store cannot be restored (the Vault cluster is gone and the backup is encrypted with a key that is also lost): the secrets are lost. Rotate every secret. The rotation is a multi-team effort; see git-cicd-gitops-rb-08-rotate-git-credentials and git-cicd-gitops-rb-25-rotate-deploy-identity
↶If the GitOps controller cannot reach Git (the Git hosting is also down): see git-cicd-gitops-rb-21-recover-git-hosting-dependency. The CI/CD recovery is blocked on Git access
↶If the recovery introduces a security regression (a credential is restored in plaintext, a service account is over-permissioned): the recovery is insecure. Engage the security team. The CI/CD may need to be re-bootstrapped
↶If the RPO is breached (more data was lost than the RPO allows): the recovery is a data-loss event. Engage the legal team and the communications team. The data loss may have regulatory implications

6 · Escalation

When the runbook isn't enough, contact:

· The disaster also affected the team's primary workload (production is down, customer-facing): the disaster is now a multi-system incident. Engage the incident response team. The CI/CD recovery continues in parallel with the customer-facing recovery
· The disaster was caused by a security incident (ransomware, supply-chain attack, malicious insider): see git-cicd-gitops-rb-28-respond-to-supply-chain-compromise. The CI/CD recovery must include a forensic analysis before the restored system is trusted
· The RTO is breached and CI/CD cannot be restored within the target time: engage the engineering leadership. The team may need to operate manually; the customer-facing impact is the priority. Communicate the timeline honestly
· The backup is corrupted or unavailable: the recovery is from a degraded state. Engage the platform team; the recovery may require manual reconstruction. Document the data loss and the impact
· The recovered CI/CD is incompatible with the production clusters (a Kubernetes version mismatch, a Terraform provider version mismatch): the recovery needs an upgrade as part of the bring-up. This is a multi-hour effort; engage the platform team
· Multiple regions are affected by the disaster: the warm standby is also down. Engage the cloud provider and the platform team. The cold rebuild may be the only option
· The disaster recovery process itself is the bottleneck (manual steps, undocumented dependencies, missing automation): the DR plan needs investment. Engage the platform team and the engineering leadership; the next disaster will be worse

A disaster that destroys CI/CD capability stops every team”s deployment pipeline. The recovery is multi-component: the control plane (GitHub Enterprise, GitLab, Jenkins), the runner fleet, the secrets store, the artifact registry, the GitOps controllers, and the deployment targets. Each must be restored from a backup or rebuilt from IaC, and the end-to-end chain must be tested before production resumes.

1. Declare the disaster and activate recovery

Read-only / Safe

$ INCIDENT_ID="DR-2026-001"
gh issue create --repo myorg/myorg --title "DR activated: $INCIDENT_ID" \
--body "Reason: REPLACE_WITH_DISASTER. RTO target: REPLACE_WITH_HOURS. RPO target: REPLACE_WITH_HOURS. Teams engaged: REPLACE_WITH_LIST. First milestone ETA: REPLACE_WITH_TIME." \
--label disaster-recovery --label p1
echo "--- activate the warm standby ---"
DR_REGION="us-west-2"
aws route53 create-health-check --caller-reference "$INCIDENT_ID" --health-check-config file://healthcheck.json
aws route53 change-resource-record-sets --hosted-zone-id Z123 --change-batch file://failover.json
echo "--- wait for the DNS change to propagate (repeat until INSYNC) ---"
aws route53 get-change --id REPLACE_WITH_CHANGE_ID --query 'ChangeInfo.Status' --output text
echo "--- or activate the cold rebuild ---"
TF_DIR="/infra/cold-standby"
cd "$TF_DIR"
terraform init
terraform apply -auto-approve

The declaration is the audit anchor. The activation is the first operational step. The choice between warm standby and cold rebuild depends on the RTO/RPO targets.

2. Restore the CI/CD control plane

Read-only / Safe

$ echo "--- GitHub Enterprise restore ---"
ghe-restore REPLACE_WITH_BACKUP_ID --confirm
echo "--- GitLab restore ---"
gitlab-backup restore BACKUP=REPLACE_WITH_TIMESTAMP --confirm
echo "--- Jenkins restore ---"
scp jenkins-backup.tar.gz jenkins-master:/var/lib/jenkins/
ssh jenkins-master 'cd /var/lib/jenkins && tar -xzf jenkins-backup.tar.gz && systemctl restart jenkins'
echo "--- verify ---"
gh api user --jq .login
glab api projects --jq '.[].path_with_namespace' | head
echo "--- or for Jenkins ---"
curl -fsS http://jenkins.internal/api/json | jq '.mode'

The control plane is the brain. The restore must complete before any other component can register against it.

3. Restore the runner fleet

Read-only / Safe

$ echo "--- GitHub Actions self-hosted runners ---"
TF_DIR="/infra/runners"
cd "$TF_DIR"
terraform apply -auto-approve
echo "--- register the runners ---"
ansible-playbook -i inventory runners.yml
echo "--- GitLab runners ---"
gitlab-runner register --non-interactive \
--url "https://gitlab.internal/" \
--registration-token "$RUNNER_TOKEN" \
--executor docker \
--docker-image "docker:24"
echo "--- verify ---"
gh api orgs/myorg/actions/runners --jq '.runners[] | {name, status, busy}'
gitlab-runner list

The runner fleet is restored from the IaC modules (Terraform, Ansible). The runners register against the control plane and begin accepting jobs.

4. Restore the secrets store

Read-only / Safe

$ echo "--- Vault ---"
vault operator raft snapshot restore /tmp/vault-snapshot.snap
vault operator unseal REPLACE_WITH_UNSEAL_KEY_1
vault operator unseal REPLACE_WITH_UNSEAL_KEY_2
vault operator unseal REPLACE_WITH_UNSEAL_KEY_3
echo "--- verify ---"
vault status
echo "--- external secrets re-sync ---"
kubectl get externalsecret -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | while read -r ns name; do
kubectl annotate externalsecret "$name" -n "$ns" force-sync=true --overwrite
done
echo "--- sealed secrets ---"
kubeseal --fetch-cert --controller-name=sealed-secrets > pub-cert.pem
for secret in $(find secrets/ -name '*.yaml'); do
kubeseal --cert pub-cert.pem < "$secret" | kubectl apply -f -
done

The secrets store is the most sensitive component. Vault is unsealed with the recovery keys. External secrets are re-pulled from the cloud. Sealed secrets are re-encrypted with the new cluster”s key.

5. Restore the artifact registry

Read-only / Safe

$ echo "--- failover to the mirror ---"
crane copy REPLACE_WITH_MIRROR/myorg/checkout-api:v1.2.3 REPLACE_WITH_PRIMARY/myorg/checkout-api:v1.2.3
echo "--- or restore from the backup ---"
HARBOR_BACKUP="/tmp/harbor-backup.tar.gz"
tar -xzf "$HARBOR_BACKUP" -C /var/lib/harbor/
docker-compose up -d
echo "--- verify ---"
crane manifest REPLACE_WITH_REGISTRY/REPLACE_WITH_IMAGE:REPLACE_WITH_TAG

The registry is restored from the backup or failed over to a mirror. See git-cicd-gitops-rb-22-recover-registry-outage for the detailed procedure.

6. Test the end-to-end pipeline

Read-only / Safe

$ TEST_REPO="myorg/test-repo"
echo "--- push a test commit ---"
git clone https://github.com/$TEST_REPO /tmp/test-repo
cd /tmp/test-repo
echo "test" >> README.md
git commit -am "DR test commit"
git push origin main
echo "--- watch the pipeline ---"
gh run watch
echo "--- verify the test deploy ---"
kubectl get deploy test-app -n staging -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}'
curl -fsS https://test-app.staging.internal/healthz | jq .status

The end-to-end test exercises every component: Git → CI → registry → GitOps controller → cluster → application. A passing test proves the chain is functional.

7. Restore production deploys

Read-only / Safe

$ KNOWN_GOOD_TAG="v1.2.3-known-good"
echo "--- enable production workflows ---"
gh workflow enable deploy-prod.yml
echo "--- deploy the known-good artifact ---"
gh workflow run deploy-prod.yml -f image_tag="$KNOWN_GOOD_TAG"
echo "--- watch ---"
gh run watch
echo "--- verify ---"
kubectl get deploy -n prod -o jsonpath='{.items[*].spec.template.spec.containers[0].image}{\"\n"}'
echo "--- canary watch ---"
sleep 300
echo "--- error rate check (must be at baseline) ---"
echo "Check observability dashboard for production error rate"

Production deploys resume with a known-good artifact. The first deploy is a canary — watch closely for the first 5 minutes.

8. Document the recovery

Read-only / Safe

$ INCIDENT_ID="DR-2026-001"
gh issue create --repo myorg/myorg --title "DR completed: $INCIDENT_ID" \
--body "Disaster: REPLACE_WITH_DESCRIPTION. RTO achieved: REPLACE_WITH_ACTUAL (target REPLACE_WITH_TARGET). RPO achieved: REPLACE_WITH_ACTUAL (target REPLACE_WITH_TARGET). Components restored: REPLACE_WITH_LIST. Data loss: REPLACE_WITH_LIST. Operator: $USER. Post-incident review: REPLACE_WITH_DATE." \
--label disaster-recovery --label post-incident
echo "--- schedule the post-incident review ---"
gh pr create --repo myorg/runbook --title "DR-2026-001 post-incident review" --body "See REPLACE_WITH_INCIDENT_URL. Action items to follow."

The documentation closes the loop. The post-incident review is scheduled; the action items are tracked.

Verification

The CI/CD control plane is reachable. Runners are registered and accepting jobs. A test commit triggers a successful end-to-end pipeline. The artifact registry is reachable. The GitOps controller reconciles. Production deploys resume and the application is healthy. Every component”s response time and error rate returns to baseline. No new vulnerabilities were introduced by the recovery.

Rollback

If the DR fails (backup corrupted, standby cannot activate), escalate to engineering leadership; operate manually. If the recovered CI/CD produces a different artifact, investigate — it may be benign or a security issue. If the production deploy fails, do not assume CI/CD is broken; investigate the application. If the disaster resolves during recovery, wait for sustained health before failing back. If the secrets store is lost, rotate every secret. If the GitOps controller cannot reach Git, see the Git hosting dependency runbook. If the recovery introduces a security regression, engage security. If the RPO is breached, the recovery is a data-loss event — engage legal.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Declare the disaster and activate recovery

2. Restore the CI/CD control plane

3. Restore the runner fleet

4. Restore the secrets store

5. Restore the artifact registry

6. Test the end-to-end pipeline

7. Restore production deploys

8. Document the recovery

Verification

Rollback

References