Kubernetes in the pipeline — manifest validation and the GitOps sync

Kubernetes belongs at the workload-orchestration layer of the IaC model. It does not provision cloud resources (Terraform’s job), configure the host kernel (Ansible’s job), or build the container image (Docker’s job). What it does is reconcile the cluster to the desired state declared in a set of manifests. The CI/CD/GitOps posture that makes this work is the subject of this lesson.

Where Kubernetes sits in the flow

flowchart LR
    A["Git commit of manifests"] --> B["kubeconform schema check"]
    B --> C["Conftest and OPA policy"]
    C --> D["PR review and approval"]
    D --> E["Merge to GitOps branch"]
    E --> F["ArgoCD or Flux detects drift"]
    F --> G["Server-side apply to API"]
    G --> H["Cluster converges to desired state"]
    H --> I["Continuous reconciliation"]

Three properties of this flow are worth pulling out:

• CI validates; the controller reconciles. CI never applies to the cluster. CI runs schema checks, policy checks, and rendering; the controller does the apply, and does it on a loop.
• The controller is the production entry point. A change merged to Git is the change that lands in the cluster.
• The manifests are the artefact. The YAML committed to Git is what the cluster converges to, and what an auditor reads.

Manifest validation as a CI gate

The first gate is schema validation. A manifest that does not parse against the Kubernetes API schema cannot be applied, and catching it in CI is cheaper than catching it in the controller log. kubeconform is the standard tool:

kubeconform -strict -summary -kubernetes-version 1.30.0 manifests/

The -strict flag fails on unknown fields. The -kubernetes-version flag pins the schema set so a CI job does not silently start validating against a newer API.

The second gate is policy. Open Policy Agent, run through Conftest or as an admission controller, evaluates the rendered manifests against rules. Typical rules: no privileged containers, no host network, all images pinned by digest, every workload declaring resource limits.

conftest test --policy policies/ manifests/

Validation and policy are the two gates that prevent bad manifests from being merged. The third gate is human review on the pull request. After all three, the manifests are merged and the controller picks them up.

Server-side apply and audit

The controller uses server-side apply (SSA) rather than client-side apply for a specific reason: SSA records the field-level ownership of every field in every object, so two controllers (or a controller and a human) can manage disjoint fields of the same object without trampling each other. The trade-off is that SSA requires the controller to manage conflict markers; client-side apply is simpler but cannot model shared ownership.

For audit, every controller-driven change is recorded in the cluster’s audit log under the controller’s identity. A change made by kubectl apply from a laptop appears under the engineer’s kubeconfig; a change made by ArgoCD or Flux appears under the controller’s service account. The controller path is auditable to the commit; the laptop path is not.

Production discipline

The production framing of Kubernetes in a pipeline has three rules:

1. CI never applies to production. CI validates, renders, and packages; the controller applies. A CI job with cluster credentials can be hijacked into a deploy.
2. The GitOps branch is the deploy boundary. Whatever branch the controller watches is the branch the cluster is built from. Branch protection on the GitOps branch is not optional.
3. Manifests are pinned by digest. Every container image reference must be a digest, not a tag. The controller can enforce this in policy.

Cross-course references

• Kubernetes for Production Sysadmins - Parts XI-XIV (GitOps controllers) cover ArgoCD and Flux in depth.
• Containers for Production Sysadmins - Parts XI-XIV (Supply chain) cover the digest-pinning rule and cosign.
• Linux for Production Sysadmins - Parts XXXII-XXXV (Admission control) cover ValidatingAdmissionPolicy and OPA Gatekeeper.
• Observability for Production Sysadmins - Parts XXIII-XXVI (Audit log) cover the kube-apiserver audit log.

Quiz