KubernetesCXXIII · NetworkPolicy TroubleshootingNetworkPolicy troubleshooting
Policy testing and CI gates — the prevention strategy
What you'll learn
- Reason about NetworkPolicy as code
- Build a CI pipeline that validates the policy
- Test the policy in staging before production
- Identify the production failure modes of policy testing
Prerequisites
Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16
NetworkPolicy as code is the prevention strategy. The CI pipeline validates the policy; the staging cluster tests the policy; the production cluster enforces the policy. The discipline is the same scale-free.
The policy as code
The NetworkPolicy is a Kubernetes object. The object’s YAML is the source of truth. The YAML is committed to a git repo; the CI pipeline validates the YAML; the GitOps controller applies the YAML to the cluster.
flowchart LR
A[Git repo] --> B[CI pipeline]
B --> C[Policy validation]
C -->|Pass| D[GitOps controller]
C -->|Fail| E[CI failure]
D --> F[Staging cluster]
F --> G[Production cluster]
The policy as code is the cluster’s network posture.
The CI pipeline
The CI pipeline validates the policy:
# Validate the policy with conftest (OPA)
conftest test policy.yaml --policy policies/
# Validate the policy with kubeconform
kubeconform -strict -summary policy.yaml
# Validate the policy with kubectl --dry-run
kubectl apply -f policy.yaml --dry-run=server
# Validate the policy with kyverno
# (requires the policy to be applied to a test cluster)
The CI pipeline is the policy’s gate.
The staging cluster
The staging cluster is the policy’s rehearsal. The staging cluster is a representative of production: the same CNI, the same CSI, the same workload shape.
# Substitute your own values before running:
SOURCE_POD=web-5f9c7d8b6c-2xk9p
TARGET_IP=192.0.2.24
TARGET_PORT=8080
TARGET_SVC=billing
TARGET_NS=prod
# Apply the policy to staging
kubectl apply -f policy.yaml -n prod
# Test the connection
kubectl exec -it "$SOURCE_POD" -n prod -- curl -v "$TARGET_IP:$TARGET_PORT"
# Test the egress
kubectl exec -it "$SOURCE_POD" -n prod -- nslookup "$TARGET_SVC.$TARGET_NS.svc.cluster.local"
The staging cluster is the policy’s gym.
The production cluster
The production cluster is the policy’s enforcement. The GitOps controller applies the policy to production; the CNI enforces the policy.
# Substitute your own values before running:
SOURCE_POD=web-5f9c7d8b6c-2xk9p
TARGET_IP=192.0.2.24
TARGET_PORT=8080
# Apply the policy to production
kubectl apply -f policy.yaml -n prod
# Verify the policy is enforced
kubectl exec -it "$SOURCE_POD" -n prod -- curl -v "$TARGET_IP:$TARGET_PORT"
The production cluster is the policy’s destination.
The policy validation
The policy validation is the CI pipeline’s gate. The validation checks:
- Schema validity. The YAML is valid Kubernetes.
- Best practices. The policy uses podSelector and namespaceSelector correctly.
- Default-deny. The policy is paired with a default-deny.
- DNS allowance. The policy allows DNS to kube-system.
# conftest policy example
package main
deny[msg] {
input.spec.policyTypes[_] == "Egress"
not input.spec.egress[_].to[_].namespaceSelector.matchLabels["kubernetes.io/metadata.name"] == "kube-system"
msg := "Egress policy must allow DNS to kube-system"
}
The policy validation is the cluster’s network posture.
Common failures
- Policy not validated. The CI pipeline does not validate the policy.
- Policy not tested in staging. The policy is applied to production without testing.
- Policy not committed. The policy is applied manually without committing to git.
Production discipline
The policy as code is the cluster’s network hypothesis. The discipline is to validate the policy in CI, test the policy in staging, enforce the policy in production. The network is the cluster’s connectivity; the policies are the cluster’s network posture.
- Commit the policy to git before applying it. A policy applied manually never passed the CI gate.
Quiz
Knowledge check · 4 questions
Q1. What is the role of the CI pipeline in policy as code?
Q2. A NetworkPolicy should be applied to production without testing in staging.
Q3. An operator wants to apply a new NetworkPolicy to production. The policy is in a git repo. The CI pipeline is set up. What is the workflow?
The cluster is a 1.34.x kubeadm install. The policy is in a git repo. The CI pipeline runs conftest and kubeconform. The staging cluster mirrors production. The production cluster is managed by ArgoCD.
Q4. Name three checks the CI pipeline should run on a NetworkPolicy and explain what each one does.
Passing score: 75%. Answers are checked in this browser.