Replica rebuild from manifests — when Git is the backup

When Git is the source of truth for a cluster’s workloads, total cluster state loss is recoverable without an etcd snapshot: apply the manifests from Git, and the controllers will reconcile the cluster back to the desired state. This lesson walks the recovery procedure, the ordering of add-ons, and the operational discipline of treating Git as the primary source of truth.

The Git-as-source-of-truth prerequisite

flowchart LR
    A[Git repository] -->|defines desired state| B[Kubernetes cluster]
    B -->|controllers reconcile| C[Actual state = desired state]
    D[Disaster] -->|cluster state lost| E[Apply Git manifests]
    E -->|controllers reconcile| C

The prerequisite: every workload manifest lives in Git. The cluster does not have manifests that exist only in etcd. If this is true, cluster recovery is to apply the Git manifests to a fresh cluster.

The prerequisite is not trivial. Many clusters have manifests that exist only in etcd:

• Workloads deployed imperatively with kubectl apply from a developer’s laptop, never committed.
• Helm-installed releases whose charts are not pinned in Git.
• Operator-installed components (cert-manager, ingress-nginx) installed by following a tutorial, not stored in Git.

A cluster with any of these cannot be fully recovered from Git alone.

The recovery procedure

flowchart TD
    A[1. kubeadm init] --> B[2. CNI]
    B --> C["3. Core add-ons: CoreDNS, kube-proxy"]
    C --> D["4. Operator-installed add-ons: ingress, cert-manager"]
    D --> E[5. Namespaces]
    E --> F[6. Workload manifests]
    F --> G[7. PVCs from Velero]
    G --> H[8. Validate]

The steps:

1. kubeadm init on the new control plane. Provision the control-plane nodes via IaC; run kubeadm init; save the join tokens.
2. Install the CNI. Apply the CNI manifests (Calico, Cilium, etc.). Wait for the CNI Pods to be Ready before proceeding.
3. Install the core add-ons. CoreDNS, kube-proxy. These are usually installed by kubeadm but verify.
4. Install operator-installed add-ons. ingress-nginx, cert-manager, metrics-server, Argo CD, Velero. These must be in Git or in the runbook.
5. Create namespaces. Apply the namespace manifests.
6. Apply workload manifests. Deployments, StatefulSets, Services, ConfigMaps. The controllers begin reconciling.
7. Restore PVCs from Velero. The PVC manifests reference CSI snapshots; the CSI driver provisions new volumes from the snapshots.
8. Validate. End-to-end tests against the live workloads.

The order matters

The order is enforced by dependencies:

• CNI before anything else. Pods cannot be scheduled without networking.
• CoreDNS before workloads that use DNS. Some workloads depend on cluster DNS for service discovery.
• ingress before external traffic. External traffic must reach the workloads.
• cert-manager before Ingress with TLS. Ingresses with TLS must have certificates issued.
• Workloads before PVC data. Workloads can start without their data; the data is restored into the PVC asynchronously.

Skipping the order causes subtle failures. Applying workloads before CoreDNS causes them to fail DNS resolution and crash. Applying Ingress before cert-manager causes TLS errors.

GitOps-driven recovery

flowchart LR
    A["Argo CD / Flux"] -->|watches Git| B[Cluster]
    C[Disaster] -->|cluster lost| D[New cluster]
    D --> E[Install Argo CD]
    E -->|reconciles from Git| B

A GitOps-driven cluster (Argo CD, Flux) recovers automatically:

1. kubeadm init the new control plane.
2. Install the CNI.
3. Install Argo CD (or Flux).
4. Argo CD reads the Git repository and applies every manifest.

Argo CD’s reconciliation replaces the operator’s manual apply. The operator’s job is to provision the nodes and install the bootstrap components; Argo CD does the rest.

What Git cannot recover

Git recovers the cluster’s workload manifests. It does not recover:

• Persistent volume data. The PVC manifests are in Git; the volume contents are not. Velero or CSI snapshots are required.
• Secrets. Secrets are typically not in Git. An external secret store (Vault, sealed-secrets) is required.
• Certificates. Certificates issued by cert-manager will be reissued, but only if the ACME account or CA is intact.
• etcd’s view of dynamic state. Lease objects, ephemeral containers, metrics — these are not in Git and are lost.

The operational failure modes

Git-driven recovery fails in production for predictable reasons:

• Manifests not in Git. Workloads deployed imperatively are not recovered. The gap is exposed during the disaster.
• Add-ons not in Git. CNI, ingress, cert-manager installed by tutorial. The runbook must include the install commands and configs.
• CRDs missing. Workloads use a CRD; the CRD is not in Git. The workload manifests are rejected.
• Secrets in etcd only. Secrets stored as Kubernetes Secrets (not Vault) are lost.
• Volume data not restored. The manifests come back but the PVCs are empty.

Quiz

Production discipline

Git-based cluster recovery in production rests on five non-negotiable elements:

• Git for everything. Workloads, add-ons, CRDs, namespaces, RBAC. Anything not in Git is a recovery gap.
• Sync waves for CRDs. Argo CD’s sync waves must be configured so CRDs apply before CRs that use them.
• External secret store. Vault or sealed-secrets for secrets. Kubernetes Secrets in etcd are not recoverable from Git.
• Volume data restoration as a separate step. The PVC manifests come from Git; the data comes from Velero.
• Rehearse the recovery. Quarterly Git-based recovery drills into a sandbox cluster catch the gaps (missing CRDs, missing add-ons, missing secrets) before the disaster.

Git is the source of truth. Recovery is the act of applying the source of truth to a fresh cluster. The discipline is to ensure the source of truth is complete.