Recovering the control plane — hosted service versus self-hosted

The control plane is the orchestrator. Workflows do not exist until the control plane exists; jobs do not schedule until the control plane exists. The recovery path depends on whether the control plane is hosted (a service the team consumes) or self-hosted (a service the team runs).

flowchart LR
    A["control plane recovery"] --> B["hosted path"]
    A --> C["self-hosted path"]
    B --> D["provider recovers"]
    B --> E["backup import"]
    B --> F["fresh-org rebuild"]
    C --> G["Terraform restore"]
    C --> H["Ansible restore"]
    C --> I["Helm restore"]
    D --> J["workflows can schedule"]
    E --> J
    F --> J
    G --> J
    H --> J
    I --> J

The two paths differ in tooling but converge on the same four artefacts: the organisation, the repositories, the secrets, and the runners. A control plane that lacks any one of the four cannot schedule a workflow end-to-end.

Hosted path

A hosted control plane — GitHub Actions, GitLab SaaS, CircleCI, Buildkite — runs in the provider’s account. The team does not run the orchestrator; the team runs the workflows that the orchestrator schedules.

The recovery options:

1. Provider recovery. The provider’s own DR plan returns the service. The team waits. The provider’s status page is the source of truth for the recovery time.
2. Backup import. A nightly export of repositories, secrets metadata (not values), and runner configuration is restored into a new organisation or namespace.
3. Fresh-org rebuild. The team recreates the organisation from scratch: repositories pushed from local clones, secrets re-entered from the password manager, runners re-registered.

Self-hosted path

A self-hosted control plane — GitLab self-managed, Jenkins, Gitea, Drone — runs in the team’s own infrastructure. The team owns every byte of the orchestrator. The recovery is a restore of the orchestrator’s configuration and data.

The recovery commands depend on the orchestrator:

terraform -chdir=stacks/gitlab plan -out=tfplan
terraform -chdir=stacks/gitlab apply tfplan
ansible-playbook restore-gitlab.yml
helm install gitlab gitlab/gitlab -f values-backup.yaml

The Terraform restore recreates the infrastructure (VMs, networks, databases): the GitLab stack lives in its own root module or workspace, so the plan is scoped by the repository layout, and the full plan is reviewed before terraform apply tfplan runs. -target is an exceptional, documented deviation for error recovery only; any run that uses it must be followed by a full plan to reconcile the remainder. The Ansible restore applies the configuration. The Helm restore (for Kubernetes-deployed orchestrators) applies the chart with a values file backed up before the incident.

Production discipline

Three rules that recur in every control-plane recovery:

1. The provider’s status page is the source of truth for hosted outages. Do not improvise; do not rebuild while the provider is recovering.
2. Self-hosted orchestrators are restored from IaC, not from manual commands. A restore that requires a human to remember the steps is a restore that fails in the incident.
3. Secrets are re-entered from the password manager, not reconstructed from memory. A secret that is typed from memory is a secret that drifts from the one the password manager holds.

Cross-course references

• Git, CI/CD & GitOps — Part XCVII-01 (The CI/CD DR Question) defines the dependency order the control-plane recovery opens.
• Terraform for Production Sysadmins — Part XXI (State Backup) covers the state backup the self-hosted restore depends on.
• Ansible for Production Sysadmins — Part XXXVII (Repo Architecture) covers the Ansible repository pattern the self-hosted restore uses.

Quiz