GitOps sync and reconcile — the production entry point

The GitOps controller is the production entry point for Kubernetes. CI’s job ends at merge. From that point on, the controller pulls the rendered manifest bundle, applies it to the cluster, and reconciles the live state against the declared state on a loop. Treating kubectl apply from CI as a deploy mechanism is the architectural mistake this lesson is designed to correct.

The sync flow

flowchart LR
    A["OCI manifest bundle at digest"] --> B["Argo CD or Flux controller pulls"]
    B --> C["Server-side render against cluster API version"]
    C --> D["Diff against live cluster state"]
    D -->|"diff non-empty"| E["Server-side apply to API server"]
    D -->|"diff empty"| F["Healthy, no action"]
    E --> G["Cluster converges"]
    G --> H["Reconcile loop: poll or webhook"]
    H --> D
    H -->|"drift detected"| I["Self-heal: revert to declared"]
    I --> G

The controller polls the OCI registry (or is notified by a webhook), pulls the bundle, and renders it server-side against the cluster’s API version. The render at sync time is not the same render as CI’s - the controller uses its own API version set - but it consumes the same bundle CI produced.

Once rendered, the controller computes the diff between declared and live state. If the diff is empty, the cluster is healthy. If the diff is non-empty, the controller applies it with server-side apply.

Server-side apply and audit

The controller uses server-side apply (SSA) rather than client-side apply for a specific reason: SSA records field-level ownership of every field in every object. Two controllers (or a controller and a human) can manage disjoint fields of the same object without trampling each other. The trade-off is that SSA requires the controller to manage conflict markers; client-side apply is simpler but cannot model shared ownership.

For audit, every controller-driven change is recorded in the cluster’s audit log under the controller’s service account identity. A change made by kubectl apply from a laptop appears under the engineer’s kubeconfig; a change made by Argo CD or Flux appears under the controller. The controller path is auditable to the commit; the laptop path is not.

Self-heal and reconcile

The reconcile loop is the contract. The controller polls (or is notified), computes the diff, applies the diff, and repeats. This is what makes the cluster self-healing:

flowchart LR
    A["Pod deleted by node failure"] --> B["Controller detects drift"]
    B --> C["Controller re-creates pod"]
    A2["ConfigMap changed by hand"] --> B2["Controller detects drift"]
    B2 --> C2["Controller restores from declared"]
    A3["New commit merged"] --> B3["Controller detects drift"]
    B3 --> C3["Controller rolls deployment"]

Self-heal is the policy that says “on drift, revert to declared”. With self-heal off, the controller reports drift but does not act on it - and a manual change stays in the cluster until the next sync stomps it. With self-heal on, the controller reverts the manual change on the next reconcile tick.

Self-heal is not always appropriate. Two cases where it must be disabled:

• Node drains and maintenance. A node that the operator intentionally drained for maintenance looks like drift to the controller. Self-heal will try to re-create pods on it.
• Out-of-band debugging. An engineer who legitimately changes a resource limit to debug a memory leak should not have the controller revert the change mid-debug.

The right pattern is: self-heal on for production, with a documented bypass path for the cases above (sync windows, prune=false, or a temporary self-heal-disable annotation).

Sync windows and ordering

The controller respects sync windows. A sync window is a time interval during which the controller will or will not sync. Production sync windows typically allow syncs during business hours and freeze them during incident-response rotations.

Sync waves order the apply. A CRD must be applied before the CRs that use it; a Namespace must exist before the workloads it contains; a ConfigMap must exist before the Deployment that mounts it. Sync waves encode this order:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"

Wave 0 applies first; wave 5 applies after; wave 10 applies last. The numbers are arbitrary - what matters is the ordering between waves.

Production discipline

1. CI never applies to production. CI validates, renders, and packages; the controller applies. A CI job with cluster credentials can be hijacked into a deploy.
2. Self-heal is on by default. Manual changes are reverted; OutOfSync is treated as drift that the controller resolves.
3. Sync windows are explicit. Production sync windows are documented and respected; out-of-window changes require an override.
4. Server-side apply, always. SSA records field ownership; client-side apply does not.
5. Sync status is observable. Argo CD’s sync status, Flux’s Ready condition, and the cluster’s audit log are the signals the team monitors.

Cross-course references

• Kubernetes for Production Sysadmins - Parts XI-XIV cover Argo CD and Flux controllers in depth, including GitOpsRepository and Kustomization CRDs.
• Observability for Production Sysadmins - Parts XXIII-XXVI cover the kube-apiserver audit log and how to wire it into Prometheus.
• This course, Part V (GitOps) - lessons git-cicd-gitops-v-01 through git-cicd-gitops-v-06 cover the GitOps discipline this lesson applies to Kubernetes.
• This course, Part XLVII (PipelineGraph) - covers pipeline-as-code DAG and how the controller fits as a downstream node.

Quiz