Skip to main content
RunBook Academy

Git, CI/CD & GitOpsCXI · Kubernetes Delivery PipelinePackaging

Helm or Kustomize build — the packaging decision

Advanced⏱ ~25 mingitkubectlhelmkustomize

What you'll learn

  • Choose between Helm and Kustomize based on the team's authoring model, not by inertia
  • Render Helm charts and Kustomize overlays in CI and package the output as an OCI artefact
  • Recognise why the rendered manifest bundle, not the chart or tree, is the unit the controller consumes
  • Identify the per-environment composition rules for each tool

Prerequisites

Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x

Not yet marked complete on this device.

The packaging step is where the team commits to a model for per-environment composition. Helm and Kustomize are the two dominant models in production Kubernetes today. They are not stylistic choices; each one encodes a different answer to the question “how does this application look in staging, in canary, and in production?”. The CI render step turns whichever model the team picks into the same artefact the controller consumes.

Two models for composition

flowchart LR
    A["Source commit"] --> B{"Authoring model"}
    B -->|"chart + values"| C["helm template ./chart --values values-prod.yaml"]
    B -->|"base + overlays"| D["kustomize build overlays/prod"]
    C --> E["Rendered bundle"]
    D --> E
    E --> F["kubeconform and conftest"]
    F --> G["OCI manifest bundle"]

Helm composes by values. A chart declares the application once; a values file declares the per-environment differences:

helm template ./chart \
  --values values-prod.yaml \
  --namespace prod \
  --include-crds

Kustomize composes by patch. A base declares the application once; an overlay directory declares the per-environment differences as patches:

kustomize build overlays/prod

The two models differ in where composition lives. In Helm, the values file is the source of truth for what changes per environment. In Kustomize, the overlay tree is. A team migrating from one to the other is migrating where composition is expressed, not merely which CLI they run.

When each model fits

The two models are not interchangeable. Each one fits a different authoring posture.

Helm fits when:

  • The application is packaged for many consumers who configure it through values.
  • The team wants Helm hooks and dependency management.
  • A library chart or a third-party chart is part of the dependency graph.
  • The team thinks in values.yaml, Chart.yaml, and templates/.

Kustomize fits when:

  • The application is an internal workload with no external consumers.
  • The team wants patch-style composition without templating logic.
  • Per-environment differences are additive (extra labels, extra sidecars) rather than substituting.
  • The team thinks in kustomization.yaml and patch files.

A common production posture: Kustomize for application workloads owned by the team, Helm for infrastructure add-ons (cert-manager, ingress controllers, observability stacks) where the chart is supplied by another team.

Packaging the render as an OCI artefact

The render output is YAML on stdout. The next step turns it into an OCI artefact the GitOps controller can pull from a registry, versioned, content-addressed, and signed.

For a Helm-rendered bundle, the typical flow is helm package followed by helm push to an OCI registry:

helm package ./chart \
  --version "${CHART_VERSION}" \
  --app-version "${GIT_SHA}"

For a Kustomize-rendered bundle, the typical flow is to tar the rendered output and push it as an OCI artefact using the registry CLI or oras. The registry does not care that the contents look like Kubernetes YAML; it treats the bundle as a content-addressed blob.

The artefact is keyed by commit and environment. The GitOps controller pulls it by tag or digest, then renders it server-side against the cluster’s API. Render-once-in-CI and render-once-in-the-controller are two different renders; the controller’s render uses its own API version set and its own values, but it consumes the artefact CI produced.

What packaging does not solve

Packaging turns rendered YAML into a versioned artefact. It does not:

  • Decide whether the change is correct. CI validation already did that.
  • Replace the GitOps controller. The controller still pulls and applies.
  • Make the chart or the tree immutable on its own. Helm supports dependency updates; Kustomize supports base changes. The artefact is immutable; the source is not.

Production discipline

  1. One model per repository. A repository with both Helm and Kustomize for the same workload has two sources of truth.
  2. Render flags are recorded in the artefact. If CI runs helm template --include-crds, the OCI artefact reflects that. The controller must consume the artefact, not re-render.
  3. Versioning follows Git. Chart versions and overlay tags are derived from the commit SHA. A new commit is a new version.
  4. OCI artefacts are signed. The render step produces the bundle; the sign step produces the signature. The controller verifies the signature before sync.
  5. Per-environment composition lives in the model. In Helm, the values file. In Kustomize, the overlay directory. Not in two places.

Cross-course references

  • Kubernetes for Production Sysadmins - Parts XI-XIV cover Helm hook sequencing and Kustomize component reuse.
  • Containers for Production Sysadmins - Parts XI-XIV cover OCI artefact signing with cosign.
  • This course, Part XLIV (Artefacts) - lesson git-cicd-gitops-xliv-05 covers images and manifests as artefacts.
  • This course, Part XLV (ImmutableIdentity) - covers content addressing and digest pinning of the packaged bundle.

Quiz

Knowledge check · 4 questions

  1. Q1. What is the right way to think about the Helm-versus-Kustomize choice?

  2. Q2. It is fine for the GitOps controller to re-render the Helm chart at sync time, because the chart is immutable once it is in the registry.

  3. Q3. Which of these best describes the unit the GitOps controller consumes?

  4. Q4. Diagnose a repository where both Helm and Kustomize are used for the same workload, and prescribe the correction.

    A team has a single repository with an application workload. Half the team uses the Helm chart in `charts/app` and supplies `values-prod.yaml` for production. The other half uses a Kustomize tree in `kustomize/base` with overlays for each environment, including `overlays/prod`. CI runs `helm template` and `kustomize build` separately, produces two artefacts, and uploads both to the OCI registry. Argo CD is configured to watch the registry and pulls whichever artefact was tagged last. A recent incident required reconstructing what landed in production, and the answer was unclear: one engineer thought the Helm render had shipped; another thought the Kustomize render had.

Passing score: 75%. Answers are checked in this browser.