The Kubernetes delivery pipeline — an end-to-end view · Git, CI/CD & GitOps

The Kubernetes delivery pipeline is the chain that takes a commit and produces a running workload. It is not one job and it is not one tool. It is a sequence of deterministic gates, each owning a single class of mistake, each producing an artefact the next stage consumes. Treating it as “push YAML, run kubectl apply, hope” is the architectural mistake this part of the course is designed to correct.

The end-to-end shape

The pipeline runs on every push to a feature branch and again on every merge to the protected GitOps branch. Every stage has its own credentials, its own timeout, and its own artefact contract.

flowchart LR
    A["Source commit pushed"] --> B["Image build and push with digest"]
    B --> C["Manifest render: helm template or kustomize build"]
    C --> D["Schema and policy validation: kubeconform and conftest"]
    D --> E["Package rendered manifests as OCI artefact"]
    E --> F["PR review and approval"]
    F --> G["Merge to GitOps branch"]
    G --> H["Argo CD or Flux detects commit"]
    H --> I["Server-side apply to cluster API"]
    I --> J["Reconcile loop watches live state"]
    J --> K["Runtime signals: metrics, logs, audit"]
    K -->|"drift or alert"| L["Issue or commit back to repo"]
    L --> A

Three properties of this diagram are worth pulling out before the rest of the part walks through it stage by stage:

Two artefacts. The image digest and the rendered manifest bundle are the two artefacts the cluster consumes. Everything else in the pipeline exists to produce them, validate them, or move them to the cluster.
Three credential boundaries. Image build holds registry credentials. Manifest render holds no cluster credentials. The controller holds the cluster credentials. The boundaries must not collapse.
The loop closes only at the runtime layer. A pipeline that ends at “merge to GitOps branch” is a CI/CD pipeline that ships changes but cannot learn from production. The feedback arrow from K back to A is what makes the loop a loop.

What each stage owns

Stage	Artefact produced	What it catches	What it deliberately does not catch
Image build	Signed container image with digest	Bad Dockerfiles, missing layers, supply-chain risk	Cluster shape, policy
Manifest render	Rendered YAML for one environment	Template errors, missing values, wrong overlay	Policy, runtime behaviour
Schema validation	Pass or fail report on parse	Unknown fields, malformed objects	Intent, policy
Policy validation	Pass or fail report on rules	Privileged containers, mutable tags, missing limits	Syntax, runtime behaviour
OCI package	Versioned manifest bundle	Drift between commit and artefact	Cluster reachability
PR approval	Approval token on the PR	Business rationale, author intent	Anything mechanical
GitOps sync	Live objects on the cluster	Nothing - the manifests are already approved	New errors - the cluster reports them
Reconcile loop	Restored desired state	Manual changes, deleted pods, drift	Author intent

The shape is intentional. Each row above the sync handles a class of mistake the sync would otherwise apply blindly.

Three credential boundaries

The pipeline touches credentials in three distinct ways, and the three must never collapse into one:

No-cluster boundary. Image build, manifest render, schema validation, and policy validation hold no cluster credentials. The runners cannot reach the API server and do not need to.
Render-time boundary. The OCI package push holds registry credentials only. The OCI registry is not a cluster.
Sync-time boundary. The GitOps controller holds cluster credentials and obtains them from the protected GitOps branch’s secrets store. No CI job can reach these credentials.

The boundaries are enforced by environment-variable scoping and by namespace-scoped service accounts. A pipeline that uses one credential set for all three is a pipeline that has already lost the chain of custody.

Production discipline

Two artefacts, one source of truth. The image digest and the rendered manifest bundle both derive from the same commit. Promote them together or not at all.
No cluster credentials in CI. Render and validate in CI; sync in the controller. A CI job with cluster credentials is a deploy that can be hijacked.
The GitOps branch is the deploy boundary. Whatever branch the controller watches is the branch the cluster is built from. Branch protection on it is not optional.
Reconcile loop is part of the pipeline. Self-heal must be on; manual changes must be reverted by the controller, not left in place.
The loop closes only when signals drive commits. A pipeline without runtime feedback is a one-way street that gets slower the longer it runs.

Cross-course references

Kubernetes for Production Sysadmins - Parts XI-XIV cover Argo CD and Flux controllers in depth.
Containers for Production Sysadmins - Parts XI-XIV cover digest pinning, cosign, and the image supply chain that produces one of the two artefacts.
This course, Part LII (KubernetesCI) - lessons git-cicd-gitops-lii-01 through git-cicd-gitops-lii-06 cover the render-and-validate stages of this pipeline in detail.
This course, Part LIII (ContainerSupplyChain) - lessons git-cicd-gitops-liii-01 through git-cicd-gitops-liii-06 cover the image-build stages.