The Kubernetes CI discipline — render, validate, package

Kubernetes CI is three stages — render, validate, package — and the three stages catch three different classes of error. The render stage produces a deterministic YAML set from the templater (Helm, Kustomize, or plain YAML). The validate stage checks that rendered output against the API server schema, against organisational policy, and against security rule sets. The package stage publishes the validated output as an immutable artifact — typically an OCI image — that the GitOps controller consumes. Conflating the three stages into one job loses the information that only one of them can produce.

The three stages

flowchart LR
    A["Sources\n(Helm / Kustomize / YAML)"] --> B["Render\n(helm template / kustomize build)"]
    B --> C["Validate\n(kubeconform / conftest / kyverno / trivy)"]
    C --> D["Package\n(helm package / OCI push)"]
    D --> E["GitOps controller"]

Render turns source into the YAML the API server will eventually receive. The output is a concrete, line-by-line artifact that can be diffed, hashed, and stored. helm template ./chart and kustomize build overlays/prod are the canonical render commands; both produce a stream of YAML resources on stdout.

Validate checks the rendered output. The checks fall into three families: schema validation (kubeconform against the OpenAPI schemas of a specific Kubernetes version), policy validation (conftest with Rego, or kyverno apply with CEL), and security scanning (trivy config, kubescape). Each family catches a different class of error and a different layer of the rendered artifact.

Package publishes the validated render as an immutable artifact. The canonical mechanism is the OCI registry — helm package produces a .tgz and pushes it to an OCI registry; Kustomize output is wrapped into an OCI artifact using kustomize build | crane push .... The artifact is the handoff to the GitOps controller.

What each stage catches

The three stages are not redundant. They overlap on common findings and diverge on the rest:

A helm template run catches a missing image.repository value; a kubeconform run catches a Deployment with apiVersion: apps/v1 referencing fields that don’t exist in Kubernetes 1.36; a conftest run catches a missing app.kubernetes.io/owner label; a trivy config run catches an image reference that points to a known-vulnerable image. Each finding has a different remediation, a different reviewer, and a different urgency.

Why the boundaries matter

The boundaries between render, validate, and package are not stylistic; they are operational. The render boundary is where the templater’s output becomes a concrete artifact — once it exists, it can be hashed, diffed, and reviewed. The validate boundary is where the render is checked against external truths (the Kubernetes schema, the organisational policy, the security rule set) — once it passes, the render is safe to publish. The package boundary is where the validated render becomes an immutable artifact identified by content hash — once it is in the registry, the GitOps controller can pull it by digest without re-rendering.

flowchart LR
    A["Render"] -->|content hash| B["Artifact identity"]
    B --> C["Validate"]
    C -->|pass| D["Signed artifact"]
    D --> E["Package"]
    E --> F["OCI registry"]
    F --> G["GitOps controller\n(pulls by digest)"]

The GitOps controller pulls by digest precisely because the digest is the artifact identity. A tag can move; a digest cannot. The package stage is what makes the digest stable.

Production discipline

1. Render every change, even small ones. A one-line values.yaml change produces a new render with a new content hash. The render is the review surface; skipping it skips the review.
2. Pin the validate stage to a specific Kubernetes version. kubeconform -kubernetes-version 1.36.0 makes the schema check reproducible; without the pin, the check is against whatever the tool’s default is, which drifts with tool versions.
3. Publish as OCI, not as raw YAML in the repository. The OCI artifact is the handoff. A GitOps controller pulls the artifact by digest and applies it; no re-render, no template re-execution, no environment-dependent template variables at apply time.
4. Sign the artifact. The package stage is also the provenance stage. A cosign signature on the OCI artifact is what the GitOps controller verifies before applying.

Cross-course references

• Kubernetes for Production Sysadmins - Part X (Manifests) covers the Kustomize and Helm mechanics the render stage assumes.
• Kubernetes for Production Sysadmins - Part XXXIII (Admission) covers OPA Gatekeeper and Kyverno, which enforce some of the same rules at admission time that the validate stage enforces at CI time.
• GitOps with Argo CD - Part IV (ArgoCD) covers the controller side of the artifact handoff.

Quiz