Skip to main content
RunBook Academy

← All labs in Git, CI/CD & GitOps

Lab · intermediate · ~75 min

Lab 18: Validate Kubernetes manifests with `kubeconform`, `conftest`, `kyverno`

C · SimulationB · Nested virtualisation

Objectives

  • Author a CI workflow that runs `kubeconform` for upstream Kubernetes schema validation
  • Author OPA Rego policies for `conftest` that enforce organisation-specific rules (no `latest` tags, required labels, no privileged containers)
  • Author Kyverno policies that run as admission-controller manifests (production-side)
  • Author a Helm-values and Kustomize overlay validation that ensures the rendered manifests pass `kubeconform` and `conftest`
  • Document the validation layers: schema, policy, runtime
  • Compare the three tools: schema validation vs policy validation vs admission control

Prerequisites

Objective

By the end of this lab you will have authored the artefacts that implement three-layer Kubernetes manifest validation: kubeconform for upstream OpenAPI schema validation, conftest with OPA Rego for organisation-specific policy, and Kyverno policies that run at admission time in production. The lab also authors a Helm-values and Kustomize-overlay validation that ensures rendered manifests pass both kubeconform and conftest.

The point of this lab is not any single tool — the labs in Lessons LII-02 and LII-04 covered kubeconform and conftest respectively. The point is the layering: schema validation (catches malformed manifests), policy validation (catches manifests that pass schema but violate policy), and runtime admission control (catches policy violations at deploy time as a final gate).

Architecture

A three-layer validation: schema (kubeconform), policy (conftest), runtime (Kyverno). The first two run in CI on every PR; the third runs at admission time in the cluster.

flowchart LR
    A["manifests\nyaml"] --> B["schema\nkubeconform"]
    A --> C["policy\nconftest + Rego"]
    B -- pass --> D["merge allowed"]
    C -- pass --> D
    B -- fail --> Z1["PR red"]
    C -- fail --> Z1
    D --> E["kubectl apply"]
    E --> F["runtime\nkyverno admission"]
    F -- pass --> G["resource created"]
    F -- fail --> Z2["apply rejected"]

The CI layer (kubeconform + conftest) catches violations before merge; the runtime layer (Kyverno) catches violations at deploy time. The two layers are paired: the CI layer is the prevention, the runtime layer is the safety net for resources applied outside the pipeline.

Requirements

  • Git 2.55.x on Linux or macOS.
  • A GitHub repository with Kubernetes manifests under manifests/, Helm values under helm-values/, Kustomize overlays under kustomize/, and policies under policy/.
  • No cluster required for the CI jobs (kubeconform and conftest run locally); Kyverno policies are reviewed but not enforced in the lab.

Scenario

A platform team manages a fleet of Kubernetes clusters. Every manifest that lands in the cluster must (1) conform to the upstream OpenAPI schema, (2) satisfy the team’s organisation policies (no :latest tags, required labels, no privileged containers), and (3) be admitted by the cluster’s admission controller (Kyverno). The CI pipeline runs the first two on every PR; the third runs in production as the final gate.

The lab builds the workflow, the policies, and the layered validation.

Tasks

Task 1 — Build the sample manifests

# check-shell-blocks: allow-invalid
LAB="$HOME/manifest-validation-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

mkdir -p manifests

# A valid deployment with the team's required labels and a
# pinned image tag.
cat > manifests/web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
    owner: platform-team
    env: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
        owner: platform-team
        env: production
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 256Mi
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
EOF

# A deliberately invalid manifest: missing apiVersion. This
# file demonstrates kubeconform's failure mode.
cat > manifests/broken.yaml <<'EOF'
kind: ConfigMap
metadata:
  name: broken
data:
  key: value
EOF

# A manifest that violates policy: uses :latest tag and lacks
# the `owner` label. This file demonstrates conftest's failure
# mode.
cat > manifests/violates-policy.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: violates
  labels:
    app: violates
spec:
  replicas: 1
  selector:
    matchLabels:
      app: violates
  template:
    metadata:
      labels:
        app: violates
    spec:
      containers:
        - name: violates
          image: nginx:latest   # violates no-latest-tag policy
          ports:
            - containerPort: 80
EOF

git add manifests/
git commit -m 'initial: sample manifests including a broken and a policy-violating'

The repository has three manifests: a valid one, a broken one (missing apiVersion), and a policy-violating one (uses :latest tag, lacks owner label). The next tasks wire up the validators.

Task 2 — Author the OPA Rego policy

# check-shell-blocks: allow-invalid
cd "$HOME/manifest-validation-lab"

mkdir -p policy/conftest

cat > policy/conftest/policy.rego <<'EOF'
package kubernetes.validations

# ─────────────────────────────────────────────────────────────────
# Rule 1: no `latest` tag on container images.
# ─────────────────────────────────────────────────────────────────
deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  endswith(container.image, ":latest")
  msg := sprintf("container '%s' uses :latest tag; pin to a specific version", [container.name])
}

deny[msg] {
  input.kind == "StatefulSet"
  container := input.spec.template.spec.containers[_]
  endswith(container.image, ":latest")
  msg := sprintf("container '%s' uses :latest tag; pin to a specific version", [container.name])
}

# ─────────────────────────────────────────────────────────────────
# Rule 2: required labels.
# ─────────────────────────────────────────────────────────────────
required_labels := ["app", "owner", "env"]

deny[msg] {
  input.kind == "Deployment"
  label := required_labels[_]
  not input.metadata.labels[label]
  msg := sprintf("Deployment '%s' is missing required label '%s'", [input.metadata.name, label])
}

# ─────────────────────────────────────────────────────────────────
# Rule 3: no privileged containers.
# ─────────────────────────────────────────────────────────────────
deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  container.securityContext.privileged == true
  msg := sprintf("container '%s' is privileged; remove privileged: true", [container.name])
}

# ─────────────────────────────────────────────────────────────────
# Rule 4: resource limits must be set.
# ─────────────────────────────────────────────────────────────────
deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.resources.limits.cpu
  msg := sprintf("container '%s' has no CPU limit", [container.name])
}

deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.resources.limits.memory
  msg := sprintf("container '%s' has no memory limit", [container.name])
}

# ─────────────────────────────────────────────────────────────────
# Rule 5: replicas must be at least 2 for production Deployments.
# ─────────────────────────────────────────────────────────────────
deny[msg] {
  input.kind == "Deployment"
  input.metadata.labels.env == "production"
  input.spec.replicas < 2
  msg := sprintf("production Deployment '%s' has replicas=%d; minimum is 2", [input.metadata.name, input.spec.replicas])
}
EOF

cat > policy/conftest/policy_test.rego <<'EOF'
package kubernetes.validations

# ─────────────────────────────────────────────────────────────────
# Test: no-latest-tag
# ─────────────────────────────────────────────────────────────────
test_no_latest_tag_violation {
  deny["container 'web' uses :latest tag; pin to a specific version"] with input as {
    "kind": "Deployment",
    "metadata": {"name": "web"},
    "spec": {"template": {"spec": {"containers": [{"name": "web", "image": "nginx:latest"}]}}
  }
}

test_no_latest_tag_pass {
  count(deny) == 0 with input as {
    "kind": "Deployment",
    "metadata": {"name": "web"},
    "spec": {"template": {"spec": {"containers": [{"name": "web", "image": "nginx:1.27-alpine"}]}}
  }
}

# ─────────────────────────────────────────────────────────────────
# Test: required-labels
# ─────────────────────────────────────────────────────────────────
test_missing_owner_label {
  deny["Deployment 'web' is missing required label 'owner'"] with input as {
    "kind": "Deployment",
    "metadata": {"name": "web", "labels": {"app": "web", "env": "production"},
    "spec": {"template": {"spec": {"containers": [{"name": "web", "image": "nginx:1.27"}]}}
  }
}

# ─────────────────────────────────────────────────────────────────
# Test: privileged-containers
# ─────────────────────────────────────────────────────────────────
test_privileged_container {
  deny["container 'web' is privileged; remove privileged: true"] with input as {
    "kind": "Deployment",
    "metadata": {"name": "web"},
    "spec": {"template": {"spec": {"containers": [{"name": "web", "image": "nginx:1.27", "securityContext": {"privileged": true}]}}
  }
}

# ─────────────────────────────────────────────────────────────────
# Test: production-replicas
# ─────────────────────────────────────────────────────────────────
test_production_replicas_violation {
  deny["production Deployment 'web' has replicas=1; minimum is 2"] with input as {
    "kind": "Deployment",
    "metadata": {"name": "web", "labels": {"app": "web", "owner": "team", "env": "production"},
    "spec": {"replicas": 1, "template": {"spec": {"containers": [{"name": "web", "image": "nginx:1.27"}]}}
  }
}

test_production_replicas_pass {
  count(deny) == 0 with input as {
    "kind": "Deployment",
    "metadata": {"name": "web", "labels": {"app": "web", "owner": "team", "env": "production"},
    "spec": {"replicas": 2, "template": {"spec": {"containers": [{"name": "web", "image": "nginx:1.27"}]}}
  }
}
EOF

git add policy/conftest/
git commit -m 'policy: OPA Rego policies for conftest'

The Rego policy has five rules: no :latest tags, required labels, no privileged containers, required resource limits, and minimum replicas for production. Each rule has a unit test in policy_test.rego that the CI workflow runs with conftest verify.

Task 3 — Author the Kyverno policy

# check-shell-blocks: allow-invalid
cd "$HOME/manifest-validation-lab"

mkdir -p policy/kyverno

cat > policy/kyverno/require-labels.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-app-owner-env-labels
      match:
        resources:
          kinds:
            - Deployment
            - StatefulSet
      validate:
        message: "All Deployments and StatefulSets must have app, owner, and env labels."
        pattern:
          metadata:
            labels:
              app: "?*"
              owner: "?*"
              env: "?*"
EOF

cat > policy/kyverno/disallow-latest-tag.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: no-latest-image-tag
      match:
        resources:
          kinds:
            - Deployment
            - StatefulSet
            - DaemonSet
      validate:
        message: "Container images must not use the :latest tag. Pin to a specific version."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - image: "!*:latest"
EOF

git add policy/kyverno/
git commit -m 'policy: Kyverno ClusterPolicies for runtime admission'

The Kyverno policies mirror the conftest policies but run at admission time. validationFailureAction: Enforce means the admission controller rejects resources that violate the policy. The two policies are the runtime safety net for resources applied outside the pipeline (for example, an operator running kubectl apply directly).

Task 4 — Author Helm-values and Kustomize-overlay configurations

# check-shell-blocks: allow-invalid
cd "$HOME/manifest-validation-lab"

mkdir -p helm-values kustomize/overlays/production

cat > helm-values/values.yaml <<'EOF'
# Sample Helm values. The values file is rendered through
# `helm template` and the rendered output is validated.
image:
  repository: nginx
  tag: 1.27-alpine
  pullPolicy: IfNotPresent

replicaCount: 2

labels:
  app: web
  owner: platform-team
  env: production

resources:
  limits:
    cpu: 500m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 64Mi

podSecurityContext:
  runAsNonRoot: true
  fsGroup: 1000

containerSecurityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
EOF

cat > kustomize/overlays/production/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: production

labels:
  - includeSelectors: true
    pairs:
      app: web
      owner: platform-team
      env: production

resources:
  - ../../base
EOF

mkdir -p kustomize/base
cat > kustomize/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - web.yaml
EOF

cp manifests/web.yaml kustomize/base/web.yaml

git add helm-values/ kustomize/
git commit -m 'helm+kustomize: values and overlay for rendered validation'

The Helm-values file pins the image to 1.27-alpine (not :latest), sets replicas to 2, and includes the required labels. The Kustomize overlay adds the production namespace and the required labels via the labels: transformer.

The CI workflow renders the Helm chart and the Kustomize overlay, then validates the rendered output with kubeconform and conftest. This catches policy violations that exist only in the rendered output, not in the source manifests.

Task 5 — Author the CI workflow

# check-shell-blocks: allow-invalid
cd "$HOME/manifest-validation-lab"

mkdir -p .github/workflows

cat > .github/workflows/manifest-validation.yml <<'EOF'
name: manifest validation

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  contents: read

concurrency:
  group: manifest-validation-${ github.ref }
  cancel-in-progress: ${ github.ref != 'refs/heads/main' }

jobs:
  # ─────────────────────────────────────────────────────────────────
  # Job 1: kubeconform against raw manifests
  # ─────────────────────────────────────────────────────────────────

  kubeconform-raw:
    name: kubeconform (raw)
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - name: install kubeconform
        run: |
          curl -fsSLo kubeconform.tar.gz \
            https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
          tar -xzf kubeconform.tar.gz
          sudo mv kubeconform /usr/local/bin/
      - name: validate manifests
        run: |
          # -strict: fail on unknown fields
          # -summary: print counts
          # -kubernetes-version: validate against target cluster version
          kubeconform -strict -summary \
            -kubernetes-version 1.29.x \
            -schema-location default \
            -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{.Group}/{.ResourceKind}_{.ResourceAPIVersion}.json' \
            manifests/

  # ─────────────────────────────────────────────────────────────────
  # Job 2: conftest against raw manifests + policy tests
  # ─────────────────────────────────────────────────────────────────

  conftest-raw:
    name: conftest (raw)
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - name: install conftest
        run: |
          curl -fsSLo conftest.tar.gz \
            https://github.com/open-policy-agent/conftest/releases/latest/download/conftest_0.55.0_Linux_x86_64.tar.gz
          tar -xzf conftest.tar.gz
          sudo mv conftest /usr/local/bin/
      - name: run policy tests
        run: |
          # Verify the policy itself (runs policy_test.rego).
          conftest verify --policy policy/conftest/
      - name: test manifests against policy
        run: |
          conftest test \
            --policy policy/conftest/policy.rego \
            --namespace kubernetes.validations \
            manifests/

  # ─────────────────────────────────────────────────────────────────
  # Job 3: kubeconform + conftest against rendered manifests
  # ─────────────────────────────────────────────────────────────────

  rendered:
    name: rendered (helm + kustomize)
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - name: install tools
        run: |
          curl -fsSLo kubeconform.tar.gz \
            https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
          tar -xzf kubeconform.tar.gz
          sudo mv kubeconform /usr/local/bin/
          curl -fsSLo conftest.tar.gz \
            https://github.com/open-policy-agent/conftest/releases/latest/download/conftest_0.55.0_Linux_x86_64.tar.gz
          tar -xzf conftest.tar.gz
          sudo mv conftest /usr/local/bin/
          curl -fsSL https://get.helm.sh/helm-v3.14.0-linux-amd64.tar.gz \
            | tar -xz -C /tmp
          sudo mv /tmp/linux-amd64/helm /usr/local/bin/
          curl -fsSL https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%40v5.3.0/kustomize_v5.3.0_linux_amd64.tar.gz \
            | tar -xz
          sudo mv kustomize /usr/local/bin/
      - name: render Helm chart
        run: |
          # A minimal Helm chart for the demo. The chart lives
          # in helm-values/ but uses a `helm template` invocation
          # that pulls values from values.yaml.
          mkdir -p chart/templates
          cp manifests/web.yaml chart/templates/web.yaml
          helm template web chart/ \
            --values helm-values/values.yaml \
            --namespace production \
            > rendered/helm-rendered.yaml
      - name: render Kustomize overlay
        run: |
          kustomize build kustomize/overlays/production \
            > rendered/kustomize-rendered.yaml
      - name: validate rendered manifests
        run: |
          kubeconform -strict -summary \
            -kubernetes-version 1.29.x \
            rendered/
          conftest test \
            --policy policy/conftest/policy.rego \
            --namespace kubernetes.validations \
            rendered/

  # ─────────────────────────────────────────────────────────────────
  # Job 4: validate Kyverno policy syntax (does not require a cluster)
  # ─────────────────────────────────────────────────────────────────

  kyverno-policy:
    name: kyverno policy lint
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - name: install kyverno CLI
        run: |
          curl -fsSLo kyverno.tar.gz \
            https://github.com/kyverno/kyverno/releases/latest/download/kyverno-cli_v1.10.0_linux_amd64.tar.gz
          tar -xzf kyverno.tar.gz
          sudo mv kyverno /usr/local/bin/
      - name: validate policies
        run: |
          kyverno apply policy/kyverno/ --dry-run
EOF

git add .github/workflows/manifest-validation.yml
git commit -m 'ci: kubeconform, conftest, and rendered validation'

The workflow has four jobs that run in parallel: kubeconform-raw, conftest-raw, rendered, and kyverno-policy. The first three validate manifests against the upstream schema, the team’s policy, and the rendered output. The fourth validates the Kyverno policy syntax (the Kyverno CLI can validate policies without a cluster).

Task 6 — Document the validation layers

# check-shell-blocks: allow-invalid
cd "$HOME/manifest-validation-lab"

cat > validation-layers.md <<'EOF'
# Validation layers: schema, policy, runtime

This document is the canonical record of the three layers of
Kubernetes manifest validation. The workflow is the
implementation; this document is the rationale. Engineers
should be able to answer "what does each layer catch?" by
reading this file.

## Layer 1: Schema (kubeconform)

`kubeconform` validates against the upstream Kubernetes OpenAPI
schema. It catches:

- Missing or wrong `apiVersion`, `kind`, `metadata`.
- Wrong field names (e.g., `replicas` instead of
  `replicaCount`).
- Wrong field types (e.g., `replicas: "2"` instead of
  `replicas: 2`).
- Unknown fields (with `-strict`).

A schema failure is a malformed manifest. The CI workflow's
`kubeconform-raw` and `rendered` jobs both run `kubeconform`.

## Layer 2: Policy (conftest + OPA Rego)

`conftest` validates against organisation-specific Rego
policies. It catches:

- `:latest` tags (no unpinned images).
- Missing required labels.
- Privileged containers.
- Missing resource limits.
- Production-specific rules (e.g., `replicas >= 2`).

A policy failure is a manifest that passes schema but
violates the team's standards. The CI workflow's
`conftest-raw` and `rendered` jobs both run `conftest`.

## Layer 3: Runtime (Kyverno)

Kyverno runs as a Kubernetes admission controller. It catches
the same policy violations as conftest, but at deploy time.
This is the safety net for resources applied outside the
pipeline (e.g., `kubectl apply` from an operator's terminal).

A runtime failure is a resource that should have been
prevented by CI but slipped through. The CI workflow validates
the Kyverno policy syntax (`kyverno apply --dry-run`) but does
not run the admission controller itself; that requires a
cluster.

## Layer interaction

The three layers are paired:

kubeconform → schema correctness conftest → policy correctness kyverno → runtime enforcement


A manifest that passes `kubeconform` may fail `conftest` (a
schema-correct manifest that violates policy). A manifest
that passes `conftest` may fail Kyverno (a policy-correct
manifest that the cluster rejects for some other reason — for
example, a missing CRD).

## When each layer matters

| Scenario | Schema | Policy | Runtime |
|----------|--------|--------|---------|
| PR check | kubeconform | conftest | — |
| Deploy | — | — | Kyverno |
| Audit (raw) | kubeconform | conftest | — |
| Audit (rendered) | kubeconform | conftest | — |

EOF

git add validation-layers.md
git commit -m 'docs: validation layers description'

The validation-layers document is what the team reads when they ask “do we need both conftest and Kyverno?”. The answer is yes: conftest is prevention (CI), Kyverno is enforcement (runtime). The two are paired; neither alone is sufficient.

Task 7 — Validate the YAML and Rego structure

# check-shell-blocks: allow-invalid
cd "$HOME/manifest-validation-lab"

# Workflow parses and has four jobs.
python3 -c "
import yaml
with open('.github/workflows/manifest-validation.yml') as f:
    doc = yaml.safe_load(f)
jobs = doc['jobs']
print('jobs:', list(jobs.keys()))
print('rendered steps:', [s.get('name', '?') for s in jobs['rendered']['steps']])
"

# Rego policy parses (rough check: look for `package` and `deny`).
python3 -c "
import re
with open('policy/conftest/policy.rego') as f:
    content = f.read()
print('packages:', re.findall(r'^package\s+(\S+)', content, re.MULTILINE))
print('deny rules:', len(re.findall(r'^deny\[', content, re.MULTILINE)))
"

# Kyverno policies parse as valid YAML.
python3 -c "
import yaml, glob
for path in glob.glob('policy/kyverno/*.yaml'):
    with open(path) as f:
        doc = yaml.safe_load(f)
    print(path, '→', doc['kind'], doc['metadata']['name'])
"

Expected output (excerpt):

jobs: ['kubeconform-raw', 'conftest-raw', 'rendered', 'kyverno-policy']
rendered steps: ['install tools', 'render Helm chart',
                 'render Kustomize overlay', 'validate rendered manifests']
packages: ['kubernetes.validations']
deny rules: 5
policy/kyverno/require-labels.yaml → ClusterPolicy require-labels
policy/kyverno/disallow-latest-tag.yaml → ClusterPolicy disallow-latest-tag

The workflow has four jobs; the Rego policy has one package and five deny rules; the Kyverno policies are two ClusterPolicy resources.

Task 8 — Capture the deliverables

cd "$HOME/manifest-validation-lab"

cp .github/workflows/manifest-validation.yml "$HOME/manifest-validation.yml"
cp policy/conftest/policy.rego             "$HOME/policy.rego"
cp policy/conftest/policy_test.rego        "$HOME/policy_test.rego"
cp policy/kyverno/require-labels.yaml      "$HOME/require-labels.yaml"
cp policy/kyverno/disallow-latest-tag.yaml "$HOME/disallow-latest-tag.yaml"
cp helm-values/values.yaml                 "$HOME/values.yaml"
cp kustomize/overlays/production/kustomization.yaml "$HOME/kustomization.yaml"
cp validation-layers.md                    "$HOME/validation-layers.md"

ls -l "$HOME"/manifest-validation.yml \
       "$HOME"/policy.rego \
       "$HOME"/policy_test.rego \
       "$HOME"/require-labels.yaml \
       "$HOME"/disallow-latest-tag.yaml \
       "$HOME"/values.yaml \
       "$HOME"/kustomization.yaml \
       "$HOME"/validation-layers.md

The deliverables are the eight files in $HOME, plus the repository at $HOME/manifest-validation-lab.

Validation

  • .github/workflows/manifest-validation.yml parses as valid YAML and has four jobs: kubeconform-raw, conftest-raw, rendered, kyverno-policy.
  • policy/conftest/policy.rego parses as valid Rego and has five deny rules.
  • policy/conftest/policy_test.rego has at least one test per rule.
  • policy/kyverno/*.yaml are valid ClusterPolicy resources with validationFailureAction: Enforce.
  • helm-values/values.yaml and kustomize/overlays/production/kustomization.yaml parse as valid YAML and render to manifests that pass kubeconform and conftest.
  • Every uses: reference in the workflow is a pinned commit SHA.

Expected Outcome

A three-layer Kubernetes manifest validation: schema (CI), policy (CI), runtime (admission). The CI workflow runs the first two on every PR, plus a rendered validation that catches violations that exist only in Helm or Kustomize output.

$HOME/manifest-validation-lab/
├── .github/workflows/manifest-validation.yml  # the workflow
├── policy/
│   ├── conftest/
│   │   ├── policy.rego                        # OPA policy
│   │   └── policy_test.rego                   # OPA tests
│   └── kyverno/
│       ├── require-labels.yaml                # admission policy
│       └── disallow-latest-tag.yaml           # admission policy
├── manifests/                                 # raw manifests
├── helm-values/values.yaml                    # Helm values
├── kustomize/overlays/production/             # Kustomize overlay
└── validation-layers.md                       # the rationale

The workflow is the implementation; the policies are the rules; the document is the rationale.

Troubleshooting

kubeconform fails on a CRD. Add a -schema-location flag pointing at the CRD definitions. The lab uses the datreeio CRD catalog; teams with internal CRDs should host them on an internal URL.

conftest reports a violation that the team disagrees with. Either fix the manifest or update the Rego policy. A policy change must update policy_test.rego to add a test for the new behaviour.

helm template fails to render the chart. The chart’s template references a value that is not in values.yaml. Add the value or remove the reference.

kustomize build reports an error. The overlay references a resource that does not exist in the base. Verify the path in resources:.

kyverno apply --dry-run reports a policy error. The Kyverno policy has a syntax error or references a field that does not exist. The kyverno CLI’s error message points to the line; fix and re-run.

The rendered job catches a violation that the raw job does not. A Kustomize overlay or Helm value is overriding the source. The rendered job is the one that catches it; the fix is in the overlay or values, not in the source.

Cleanup

LAB="$HOME/manifest-validation-lab"

mv "$LAB"/validation-layers.md "$HOME"/ 2>/dev/null
mv "$LAB/.github/workflows/manifest-validation.yml" \
   "$HOME/manifest-validation.yml" 2>/dev/null
mv "$LAB/policy/conftest/policy.rego" "$HOME/policy.rego" 2>/dev/null
mv "$LAB/policy/conftest/policy_test.rego" \
   "$HOME/policy_test.rego" 2>/dev/null
mv "$LAB/policy/kyverno/require-labels.yaml" \
   "$HOME/require-labels.yaml" 2>/dev/null
mv "$LAB/policy/kyverno/disallow-latest-tag.yaml" \
   "$HOME/disallow-latest-tag.yaml" 2>/dev/null
mv "$LAB/helm-values/values.yaml" "$HOME/values.yaml" 2>/dev/null
mv "$LAB/kustomize/overlays/production/kustomization.yaml" \
   "$HOME/kustomization.yaml" 2>/dev/null

rm -rf "$LAB"

find "$HOME" -maxdepth 1 -name 'manifest-validation-lab' -print
# expected: (no output)

If you applied manifests to a real cluster during the lab, remove them with kubectl delete:

kubectl delete -f manifests/ --ignore-not-found
kubectl delete namespace production --ignore-not-found

What You Learned

  • Three layers, three tools. kubeconform for schema, conftest for policy, Kyverno for runtime. The three are not alternatives; they are layers. Each catches what the others miss.
  • The rendered job is the one that catches override violations. A Kustomize overlay or Helm value can produce a manifest that violates policy even if the source manifest passes. The rendered job renders and validates the final output.
  • Conftest and Kyverno policies must agree. The lab authors both with the same rules (no :latest, required labels). A divergence is a bug; the team’s discipline: any policy change updates both.
  • Rego policies need tests. A Rego policy without policy_test.rego can break silently. The CI workflow runs conftest verify to execute the tests and fail on any failure.
  • Kyverno is cluster-scoped, conftest is CI-scoped. A Kyverno policy deployed to a cluster is enforced for every resource in the cluster; a conftest policy is enforced only for manifests in the CI pipeline. The two are paired.
  • The -schema-location flag is required for CRDs. kubeconform only knows the upstream Kubernetes API types by default; for CRDs, point at a CRD catalog (datreeio) or an internal URL.
  • validation-layers.md is the rationale. The document is what the team reads when they ask “do we need all three?”. The answer is yes, and the document explains why.

Deliverables

  • · .github/workflows/manifest-validation.yml — the GitHub Actions workflow with three jobs
  • · policy/conftest/policy.rego — the OPA Rego policy for `conftest`
  • · policy/conftest/policy_test.rego — the unit tests for the Rego policy
  • · policy/kyverno/require-labels.yaml — a Kyverno ClusterPolicy for label requirements
  • · helm-values/values.yaml — a sample Helm values file used in the Helm-template job
  • · kustomize/overlays/production/kustomization.yaml — a sample Kustomize overlay
  • · validation-layers.md — a written description of the three layers (schema, policy, runtime)

Verification status

Last reviewed
2026-08-25
Executed end to end
not yet run on hardware

The commands and configuration here have been reviewed against the verified software versions, but nobody has run this lab start to finish on a system meeting its prerequisites. Treat the Expected Outcome as the intended result rather than an observed one, and keep the Cleanup section to hand.