medium riskservice affecting~25 min

Runbook: Troubleshoot NetworkPolicy

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the client Pod and the target Pod: kubectl get pod <client> -n <ns1> and kubectl get pod <target> -n <ns2>
· Capture all NetworkPolicies in both namespaces: kubectl get networkpolicy -A -o wide
· Capture the labels of the client and target Pods: kubectl get pod <client> -n <ns1> --show-labels
· Confirm the CNI enforces NetworkPolicy (Calico, Cilium, Weave): some CNIs in policy: disabled mode silently ignore policies
· Confirm the test connection fails on the right port: kubectl exec <client> -- nc -vz <target-ip> <port>
· Capture the CNI policy inspection if available (Cilium: cilium policy get; Calico: calicoctl get networkpolicy)

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read every NetworkPolicy in the source namespace and the target namespace
2Identify whether the namespace has a default-deny policy (one that selects all Pods and denies all ingress/egress)
3For each policy, identify the selector and the rules; check that the client Pod matches the policy selector and the target is in the from list (ingress) or to list (egress)
4Test connectivity in the absence of policy by temporarily labelling the client with cni.kubernetes.io/policy: off (Calico) or by using cilium policy trace (Cilium) to identify which policy blocks
5Apply the smallest fix: add the missing from/to rule, fix the selector, label the Pod correctly, or remove the over-broad deny
6Test the fix from the client Pod: kubectl exec <client> -- nc -vz <target-ip> <port>
7Confirm no collateral damage: test that other allowed connections still work
8Apply the fix in Git and kubectl apply

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get networkpolicy -A -o wide lists the updated policy
✓kubectl exec <client> -n <ns1> -- nc -vz <target-ip> <port> returns succeeded
✓kubectl exec <client> -n <ns1> -- nc -vz <other-target-ip> <other-port> for an unrelated allowed flow still returns succeeded
✓CNI policy inspection (cilium policy get, calicoctl get networkpolicy) shows the policy the cluster is enforcing
✓No new Warning events in either namespace
✓No collateral test fails

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the fix involved adding an from/to rule that is too broad, narrow it
↶If a default-deny was removed to fix one flow, restore the default-deny and add a specific allow
↶If the fix involved labelling a Pod that should not be labelled, remove the label and adjust the policy to match the Pod actual labels
↶Capture the pre-fix and post-fix policy in Git so the change is auditable
↶If a CNI-side rule is stale, restart the CNI agent on the affected node

6 · Escalation

When the runbook isn't enough, contact:

· Multiple policies in the same namespace interact and the combined effect is unclear: capture the full policy set and escalate to platform/network ownership
· CNI reports the policy is enforced but traffic still flows when it should not: a CNI bug or stale agent state; restart the agent on a single node and test
· CNI reports policy: disabled for the cluster: NetworkPolicy is not enforced at all; this is a CNI configuration issue, not a Kubernetes object issue
· Egress to a public endpoint fails for every Pod: the cluster egress NetworkPolicy is over-restrictive; escalate to network ownership before opening egress
· Cross-namespace traffic is blocked but the policy allows it: the namespace selector is wrong or the labels on the target namespace are missing; escalate to tenancy ownership

NetworkPolicies are additive: if any policy in a namespace selects a Pod and allows the traffic, it is allowed. If multiple policies select the same Pod, the union of allows is permitted; no single policy denies. The cluster default is “allow all” — a policy that denies only takes effect if it selects a Pod and no other policy allows the same flow.

1. Confirm the CNI enforces NetworkPolicy

Read-only / SafeConfirm the CNI enforces NetworkPolicy


kubectl -n kube-system get ds cilium -o yaml | grep -E 'enable-policy'
kubectl -n kube-system logs ds/cilium --tail=50 | grep -i policy || true

# Calico
kubectl -n kube-system get ds calico-node -o yaml | grep -E 'FELIX_|CALICO_NETWORKING_BACKEND'

# Generic test - create a default-deny in a test namespace and verify it blocks
kubectl create namespace policy-test
kubectl run test --image=registry.k8s.io/pause:3.10 --restart=Never -n policy-test
kubectl apply -f - <<'YAML'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: policy-test
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
YAML
kubectl exec test -n policy-test -- nc -vz 10.96.0.10 53 || echo "policy enforced"
kubectl delete ns policy-test --wait=false

If the test Pod can reach the cluster DNS despite the default-deny, the CNI is not enforcing NetworkPolicy. The cluster does not have network isolation. Escalate to platform.

2. Read the policies in scope

Read-only / SafeRead the policies in scope


kubectl get networkpolicy -n <ns2> -o wide
kubectl get networkpolicy -A -o json | jq -r '.items[] | select(.spec.podSelector != {}) | .metadata.namespace + "/" + .metadata.name'

For each policy, capture:

The podSelector (which Pods it applies to)
The policyTypes (Ingress, Egress, or both)
The ingress/egress rules: from/to (selectors or IPBlock), ports

3. Test the failing flow

Read-only / SafeTest the failing flow


CLIENT=<client-pod>
TARGET_IP=$(kubectl get pod <target> -n <ns2> -o jsonpath='{.status.podIP}')
TARGET_PORT=8080

# Confirm the connection fails for the expected reason
kubectl exec "$CLIENT" -n <ns1> -- nc -vz "$TARGET_IP" "$TARGET_PORT" --wait=3
# Expect: timeout or connection refused

# Confirm baseline DNS works (CoreDNS, not subject to the failing policy)
kubectl exec "$CLIENT" -n <ns1> -- nslookup kubernetes.default

If DNS works and the target connection fails, NetworkPolicy is the likely cause. If DNS fails too, the runbook order is wrong; see kubernetes-rb-troubleshoot-coredns.

4. Identify the offending policy

For each policy in <ns1> and <ns2>:

Read-only / SafeIdentify the offending policy


echo "==== $p ===="
kubectl get "$p" -n <ns1> -o yaml | tee /tmp/policy.yaml | yq '.spec' || true
echo
done

Cross-reference:

Does the client’s podSelector match the client Pod’s labels?
Does the policy have an ingress rule with a from that includes the target Pod or namespace?
Does the policy have an egress rule with a to that includes the target IP or namespace?

A Pod that is matched by a podSelector: {} (all Pods) is matched by every policy in its namespace. A default-deny that selects all Pods is the most common cause of “NetworkPolicy broke my app”.

5. CNI-side tracing

Read-only / SafeCNI-side tracing


kubectl -n kube-system exec ds/cilium -- cilium policy trace \
--src-identity <client-identity> \
--dst-identity <target-identity> \
--dport <port>/<proto>

# Calico
kubectl exec -n kube-system ds/calico-node -- calicoctl get policy -o yaml | head -200

# Hubble (Cilium) for live flow observation
kubectl -n kube-system exec ds/hubble -- hubble observe --namespace <ns1> --pod <client> --follow

CNI-side tracing shows the rule that drops the flow without changing anything. Use it to confirm before fixing.

6. Apply the smallest fix

Cluster-wide riskApply the smallest fix


kubectl patch networkpolicy <policy> -n <ns1> --type=merge -p '
egress:
- to:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: <ns2>
    podSelector:
      matchLabels:
        app: <target-app>
  ports:
  - protocol: TCP
    port: <target-port>
'

# Or fix the selector
kubectl patch networkpolicy <policy> -n <ns1> --type=json -p '
[{"op": "replace", "path": "/spec/podSelector/matchLabels/app", "value": "<correct-label>"}]
'

7. Verify

Read-only / SafeVerify


$CLIENT" -n <ns1> -- nc -vz "$TARGET_IP" "$TARGET_PORT" --wait=3"
# Expect: succeeded

# Confirm no regression
kubectl exec "$CLIENT" -n <ns1> -- nc -vz <other-allowed-target> <port> --wait=3

Common pitfalls

Symptom	Cause	Action
All traffic blocked after adding default-deny	Default-deny was meant to be additive but was the only policy	Add explicit allows for required flows
Policy allows but traffic still blocked	CNI does not enforce NetworkPolicy	See step 1; escalate to platform
Egress to DNS blocked	Default-deny egress without explicit DNS allow	Add the `kube-dns` allow rule (see `kubernetes-cxxiii-03-egress-dns`)
Cross-namespace traffic allowed by policy but blocked at runtime	Namespace label missing	Verify `kubectl get ns <ns> --show-labels`
Multiple policies, one denies, none allows	All policies must allow for traffic to flow; one deny wins	Audit the policy set; add explicit allow

A NetworkPolicy is a statement of intent expressed as code. The fix is almost always an addition, not a deletion: add the missing rule, add the missing label, add the missing port. Removing policy opens the cluster wider than it should be.