NetworkPolicy troubleshooting — diagnosing the cluster firewall

NetworkPolicy troubleshooting verifies the CNI enforces NetworkPolicy, the policy rules are correct, and the selectors match. This lesson walks the diagnostic flow, the failure modes, and the operational discipline.

The diagnostic flow

The diagnostic flow for NetworkPolicy:

flowchart TD
    A[NetworkPolicy failed] --> B{CNI enforces?}
    B -->|No| C[Switch CNI]
    B -->|Yes| D{Policy exists?}
    D -->|No| E[Create policy]
    D -->|Yes| F{Selectors match?}
    F -->|No| G[Fix selectors]
    F -->|Yes| H{Ports correct?}
    H -->|No| I[Fix ports]
    H -->|Yes| J{Default-deny wrong?}
    J -->|Yes| K[Add allow rule]
    J -->|No| L[Check CNI logs]

The diagnostic flow is the operator’s guide. The cluster operator must follow the flow.

Step 1: Verify the CNI enforces NetworkPolicy

The first step is to verify the CNI enforces NetworkPolicy:

# Substitute your own value before running
# (calico-node, cilium, or weave-net):
CNI_LABEL=calico-node

# Check the CNI plugin
kubectl get pods -n kube-system -l k8s-app="$CNI_LABEL"

The CNI must be one that enforces NetworkPolicy (Calico, Cilium, or Weave). Flannel does NOT enforce.

Step 2: Verify the policy

The second step is to verify the policy:

# Substitute your own value before running:
POLICY=allow-frontend-to-billing

kubectl describe networkpolicy "$POLICY"

The policy must have the right selectors, the right ports, and the right policy types.

Step 3: Verify the selectors

The third step is to verify the selectors:

# Substitute your own values before running - these are the selectors
# copied out of the policy inspected in step 2:
POD_SELECTOR=app=billing
NAMESPACE_SELECTOR=environment=production

# Verify the podSelector
kubectl get pods -l "$POD_SELECTOR"

# Verify the namespaceSelector
kubectl get namespace -l "$NAMESPACE_SELECTOR"

The selectors must match the intended Pods and namespaces.

Step 4: Verify the ports

The fourth step is to verify the ports:

# Substitute your own value before running:
SERVICE=billing

kubectl describe svc "$SERVICE"

The ports must match the Pod’s ports.

Step 5: Verify the default-deny

The fifth step is to verify the default-deny:

kubectl get networkpolicy -A | grep default-deny

The default-deny must be in place. The allow rules must be explicit.

Step 6: Test the connection

The sixth step is to test the connection:

# Substitute your own values before running:
POD=frontend-5f9c7d8b6c-2xk9p
SERVICE=billing.production.svc.cluster.local

kubectl exec "$POD" -- curl "http://$SERVICE"

The connection must succeed if the policy allows or fail if the policy denies.

The failure modes

The NetworkPolicy’s failure modes:

• CNI not enforcing: the cluster runs Flannel alone. The fix is to switch the CNI.
• Selector mismatch: the selector does not match the Pods. The fix is to verify the labels.
• Port mismatch: the ports do not match the Pod’s ports. The fix is to verify the ports.
• Default-deny too strict: the default-deny blocks legitimate traffic. The fix is to add the allow rules.
• DNS rule missing: the Pods cannot resolve DNS. The fix is to add the DNS allow rule.

The operational discipline

The NetworkPolicy troubleshooting’s operational discipline:

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the NetworkPolicy metrics. The alerts are the leading indicator of outages.

Quiz

Production discipline

• The diagnostic flow is systematic. The cluster operator must follow the flow.
• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the NetworkPolicy metrics. The alerts are the leading indicator of outages.
• Document the troubleshooting in the runbook. The runbook is the cluster’s operational reference.
• Plan the troubleshooting’s evolution. The troubleshooting is the cluster’s operational reference; the operator must keep it current.