Lab 15: Enforce and prove a NetworkPolicy

Objective

By the end of this lab you will have written a NetworkPolicy that the API server accepted, stored, and returned on request — and proved with a connection test that it blocked nothing at all. You will then have installed a CNI that enforces policy, watched the same unchanged policy object start cutting traffic, and rebuilt a working application path out of it one allow rule at a time.

The lab is built around one discipline: a policy is not evidence, a connection test is. Every claim below is checked with the same script against the same five flows, so that every change has a before and an after that a reviewer can read.

Architecture

Two namespaces and four Pods. Nothing in the picture is complicated; the whole difficulty is in which arrows survive which policy.

flowchart LR
    subgraph MON[namespace netpol-mon · label zone=monitoring]
        S[Pod scraper · nginx + netshoot sidecar]
    end
    subgraph LAB[namespace netpol-lab]
        A[Pod api · nginx + netshoot sidecar]
        D[Pod db · nginx]
        P[Pod probe · netshoot]
    end
    A -->|F1 to db:80| D
    P -->|F2 to db:80| D
    S -->|F3 to api:80| A
    P -->|F4 to api:80| A
    A -->|F5 to scraper pod IP| S

The five flows, and what a correctly policed cluster should say about each:

The api Pod carries two containers, web (nginx) and shell (netshoot). They share one network namespace, which is why a NetworkPolicy that selects the Pod governs both of them identically. That is the unit of policy: the Pod, never the container.

Requirements

• A disposable kubeadm cluster, one control-plane node and two workers, Kubernetes 1.34.x, built per the Part LXXIV lessons or Lab 01, with flannel as the CNI and all nodes Ready. Task 4 replaces the CNI on a running cluster. Do not run this against anything anyone else is using.
• The Lab 01 working directory $HOME/kubeadm-lab still present on the control-plane node, containing kube-flannel.yml. Cleanup re-applies that exact file. If you no longer have it, download the same flannel release again before you start, not after.
• kubectl 1.34.x with cluster-admin.
• SSH with sudo to all three nodes. Task 4 and Cleanup remove a stale CNI configuration file from /etc/cni/net.d on each node. There is no way to do that through the API.
• Ability to pull nginx:1.27.2, nicolaka/netshoot, and the Calico and flannel images the two CNI manifests reference. Mirror them first if the cluster has no registry route.
• Roughly 500 MiB of spare memory across the workers for the lab Pods, plus headroom for one more DaemonSet on every node.
• No out-of-band access requirement for the nodes’ own connectivity. Nothing here changes a node’s interfaces, routes to the outside world, firewall, or SSH daemon, so your SSH session cannot drop. The Pod network is a different matter: Task 4 interrupts Pod-to-Pod traffic cluster-wide for the length of a DaemonSet rollout.

Scenario

A security review has landed on your desk. It says the cluster has “no network segmentation” and asks for a default-deny posture in the payments namespace within the quarter. Someone on the platform team has already responded by merging a NetworkPolicy with podSelector: {}, closing the ticket, and moving on. The object is in the cluster. kubectl get networkpolicy lists it. kubectl describe prints its rules. The auditor’s screenshot shows it.

Your job is to find out whether any packet in the cluster has ever been dropped because of it, and if not, to get to a state where the answer is yes — with evidence that does not depend on anyone’s reading of a YAML file.

Tasks

Task 1: Capture the starting state and name what enforces policy

Cleanup is only honest if you know what you started from, and this lab changes a cluster-wide component. Capture it before anything else.

WORKDIR="$HOME/k8s-lab15"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

kubectl get nodes -o wide            | tee nodes.pre-lab.txt
kubectl get daemonset -A             | tee daemonsets.pre-lab.txt
kubectl get networkpolicy -A         | tee netpol.pre-lab.txt
kubectl -n kube-system get pods -o wide > kube-system-pods.pre-lab.txt
kubectl -n kube-flannel get pods -o wide > flannel-pods.pre-lab.txt

Then look at what is actually on a node, because the CNI configuration directory is the thing that decides which plugin wires a new Pod:

# Substitute your own node names before running:
CP=k8s-cp-1
W1=k8s-w-1
W2=k8s-w-2

for NODE in "$CP" "$W1" "$W2"; do
  echo "== $NODE"
  ssh "$NODE" 'ls -l /etc/cni/net.d/'
done

$ ls -l /etc/cni/net.d/

total 4
-rw-r--r-- 1 root root 292 Aug 19 09:12 10-flannel.conflist

That single file is the whole answer to “what networks my Pods”. Record it. In Task 4 a second file appears beside it, and in Cleanup you put this state back.

Now the question the scenario asks. There are two ways to answer “does this cluster enforce NetworkPolicy”, and only one of them is worth anything:

• Read which CNI is installed and look up whether it enforces policy. flannel does not; Calico, Cilium and Weave do. This is the answer the lessons give you, and it is correct, but it is a lookup, not a measurement.
• Create a namespace, put a Pod in it, deny everything, and try to open a connection. This is a measurement.

Tasks 2 and 3 are the measurement, on the estate you are about to build.

Task 2: Build the estate and record the baseline matrix

Write estate.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: netpol-lab
---
apiVersion: v1
kind: Namespace
metadata:
  name: netpol-mon
  labels:
    zone: monitoring
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
  namespace: netpol-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
        tier: data
    spec:
      containers:
        - name: web
          image: nginx:1.27.2
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: db
  namespace: netpol-lab
spec:
  selector:
    app: db
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: netpol-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
        tier: front
    spec:
      containers:
        - name: web
          image: nginx:1.27.2
          ports:
            - containerPort: 80
        - name: shell
          image: nicolaka/netshoot
          command: ["sleep", "infinity"]
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: netpol-lab
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
  name: probe
  namespace: netpol-lab
  labels:
    app: probe
spec:
  containers:
    - name: shell
      image: nicolaka/netshoot
      command: ["sleep", "infinity"]
---
apiVersion: v1
kind: Pod
metadata:
  name: scraper
  namespace: netpol-mon
  labels:
    app: scraper
spec:
  containers:
    - name: web
      image: nginx:1.27.2
      ports:
        - containerPort: 80
    - name: shell
      image: nicolaka/netshoot
      command: ["sleep", "infinity"]

scraper carries the same two-container shape as api for one reason: F5 needs something on the other end that answers. A Pod with nothing listening returns the same 000 as a Pod behind a policy that drops the packet, and a probe that cannot tell “blocked” from “nobody home” is not a probe.

$ kubectl apply -f estate.yaml

cd "$HOME/k8s-lab15"
kubectl -n netpol-lab rollout status deployment/api --timeout=180s
kubectl -n netpol-lab rollout status deployment/db  --timeout=180s
kubectl -n netpol-lab wait --for=condition=Ready pod/probe --timeout=180s
kubectl -n netpol-mon wait --for=condition=Ready pod/scraper --timeout=180s

Confirm the namespace labels, because one rule later depends on them:

$ kubectl get ns netpol-lab netpol-mon --show-labels

NAME         STATUS   AGE   LABELS
netpol-lab   Active   40s   kubernetes.io/metadata.name=netpol-lab
netpol-mon   Active   40s   kubernetes.io/metadata.name=netpol-mon,zone=monitoring

kubernetes.io/metadata.name is applied automatically to every namespace by an admission controller, and it is immutable, which is what makes it safe to select on. zone=monitoring is yours, which means somebody can remove it, and a rule that depends on it silently stops matching when they do.

Now the instrument. Write probe.sh:

#!/usr/bin/env bash
# probe.sh - run the same five flows and print the HTTP status of each.
# 000 means no response: blocked, dropped, or unresolvable. curl does not
# distinguish those, and neither should you until you have looked.
set -u

hit() {
  LABEL="$1"; NS="$2"; TARGET="$3"; URL="$4"; EXTRA="${5:-}"
  CODE=$(kubectl -n "$NS" exec $EXTRA "$TARGET" -- \
           curl -s -m 4 -o /dev/null -w '%{http_code}' "$URL" 2>/dev/null || true)
  printf '%-6s %-46s %s\n' "$LABEL" "$URL" "${CODE:-000}"
}

SCRAPER_IP=$(kubectl -n netpol-mon get pod scraper -o jsonpath='{.status.podIP}')

echo "flow   target                                         code"
hit F1 netpol-lab deploy/api "http://db.netpol-lab.svc.cluster.local" "-c shell"
hit F2 netpol-lab pod/probe  "http://db.netpol-lab.svc.cluster.local"
hit F3 netpol-mon pod/scraper "http://api.netpol-lab.svc.cluster.local" "-c shell"
hit F4 netpol-lab pod/probe  "http://api.netpol-lab.svc.cluster.local"
hit F5 netpol-lab deploy/api "http://$SCRAPER_IP" "-c shell"

F5 targets a Pod IP rather than a Service name on purpose: it is the flow that tells you whether egress out of the namespace is open, without dragging DNS into the answer.

cd "$HOME/k8s-lab15"
chmod +x probe.sh
./probe.sh | tee matrix-0-baseline.txt

$ ./probe.sh

flow   target                                         code
F1     http://db.netpol-lab.svc.cluster.local         200
F2     http://db.netpol-lab.svc.cluster.local         200
F3     http://api.netpol-lab.svc.cluster.local        200
F4     http://api.netpol-lab.svc.cluster.local        200
F5     http://10.244.2.14                             200

All five return 200, which is the Kubernetes default and the reason this lab exists: every Pod in the cluster can open a connection to every other Pod, in any namespace, on any port. No policy has been written yet, and nothing in the cluster’s configuration expresses an opinion about it.

Two notes on reading the numbers. 000 in this script means “curl received no HTTP response” and covers three different things — the name did not resolve, the connection was dropped, or the connection was refused. Tasks 5 and 8 pull those apart with targeted commands, and until then treat 000 as “no answer”, not as “blocked”. And keep matrix-0-baseline.txt: every later claim in this lab is a diff against it.

Task 3: Apply a default-deny ingress and watch nothing happen

This is the policy from the scenario. Write p1-default-deny-ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: netpol-lab
spec:
  podSelector: {}
  policyTypes:
    - Ingress

podSelector: {} selects every Pod in the namespace. policyTypes: [Ingress] declares those Pods isolated for ingress. There are no ingress: rules, so nothing is allowed in. On a cluster that enforces policy, this is the strongest statement the API can make about inbound traffic to netpol-lab.

$ kubectl apply -f p1-default-deny-ingress.yaml

Confirm the API server took it, exactly as an auditor would:

kubectl -n netpol-lab get networkpolicy
kubectl -n netpol-lab describe networkpolicy default-deny-ingress

$ kubectl -n netpol-lab describe networkpolicy default-deny-ingress

Name:         default-deny-ingress
Namespace:    netpol-lab
Created on:   2026-08-19 09:21:44 +0000 UTC
Labels:       <none>
Annotations:  <none>
Spec:
PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
Allowing ingress traffic:
  <none> (Selected pods are isolated for ingress connectivity)
Not affecting egress traffic
Policy Types: Ingress

Now measure.

cd "$HOME/k8s-lab15"
./probe.sh | tee matrix-1-unenforced.txt
diff matrix-0-baseline.txt matrix-1-unenforced.txt && echo "IDENTICAL"

The two files are identical. Every flow still returns what it returned before. The policy that the API server accepted, that describe renders in the words “Selected pods are isolated for ingress connectivity”, that satisfies the ticket and the screenshot, has dropped nothing.

Task 4: Install an enforcer — replace flannel with Canal

Canal is Calico’s policy engine running alongside flannel’s dataplane: flannel still assigns addresses and carries traffic over VXLAN, Calico’s Felix programs the filtering. Upstream ships it as a single manifest.

Read the manifest before you apply it. The number that matters is the flannel network, which must equal the --pod-network-cidr the cluster was built with:

cd "$HOME/k8s-lab15"
CALICO_VERSION=v3.32.1
curl -fsSLO "https://raw.githubusercontent.com/projectcalico/calico/$CALICO_VERSION/manifests/canal.yaml"

grep -n '"Network"' -A 4 canal.yaml
grep -n 'CNI_CONF_NAME' -A 1 canal.yaml
grep -c '^kind: CustomResourceDefinition' canal.yaml

$ grep -n '"Network"' -A 4 canal.yaml

100:      "Network": "10.244.0.0/16",
101-      "Backend": {
102-        "Type": "vxlan"
103-      }
104-    }

10.244.0.0/16 over VXLAN is exactly what Lab 01’s flannel install used, so the dataplane is not changing shape — only the plugin that owns it is. CNI_CONF_NAME is 10-canal.conflist, which is the second file that will appear in /etc/cni/net.d. And the manifest carries a large set of CustomResourceDefinitions, which is what makes Cleanup dangerous enough to have its own warning.

$ kubectl delete -f $HOME/kubeadm-lab/kube-flannel.yml

$ kubectl apply -f canal.yaml

kubectl -n kube-system rollout status daemonset/canal --timeout=300s
kubectl -n kube-system rollout status deployment/calico-kube-controllers --timeout=300s
kubectl -n kube-system get pods -l k8s-app=canal -o wide

The canal DaemonSet runs two containers per node — calico-node and kube-flannel — which is the split stated in one line of kubectl get pods: the READY column reads 2/2, not 1/1.

Now clear the stale configuration from every node:

# Substitute your own node names before running:
CP=k8s-cp-1
W1=k8s-w-1
W2=k8s-w-2

for NODE in "$CP" "$W1" "$W2"; do
  echo "== $NODE"
  ssh "$NODE" 'ls -l /etc/cni/net.d/'
done

Both files are there: 10-canal.conflist and the 10-flannel.conflist that nothing installs any more.

$ ssh "$NODE" 'sudo rm -f /etc/cni/net.d/10-flannel.conflist && ls -l /etc/cni/net.d/'

Now the step that is easy to skip and invalidates everything after it:

cd "$HOME/k8s-lab15"

kubectl -n kube-system delete pod -l k8s-app=kube-dns
kubectl -n netpol-lab delete pod --all
kubectl -n netpol-mon delete pod --all

# api and db are Deployments and come back by themselves. probe and scraper are
# bare Pods with no controller behind them, so re-apply the manifest that
# defines them.
kubectl apply -f estate.yaml

kubectl -n kube-system rollout status deployment/coredns --timeout=180s
kubectl -n netpol-lab rollout status deployment/api --timeout=180s
kubectl -n netpol-lab rollout status deployment/db  --timeout=180s
kubectl -n netpol-lab wait --for=condition=Ready pod/probe --timeout=180s
kubectl -n netpol-mon wait --for=condition=Ready pod/scraper --timeout=180s

kubectl -n kube-system get daemonset canal
kubectl -n netpol-lab get pods -o wide
kubectl -n netpol-mon get pods -o wide

Task 5: Run the same probe again

Nothing about the policy has changed. You have not edited p1-default-deny-ingress.yaml, re-applied it, or touched the object.

cd "$HOME/k8s-lab15"
kubectl -n netpol-lab get networkpolicy
./probe.sh | tee matrix-2-enforced.txt
diff matrix-1-unenforced.txt matrix-2-enforced.txt || true

$ ./probe.sh

flow   target                                         code
F1     http://db.netpol-lab.svc.cluster.local         000
F2     http://db.netpol-lab.svc.cluster.local         000
F3     http://api.netpol-lab.svc.cluster.local        000
F4     http://api.netpol-lab.svc.cluster.local        000
F5     http://10.244.2.9                              200

Four flows died and one did not, and the one that lived is the tell. F1 to F4 all terminate on a Pod in netpol-lab, and the policy isolates every Pod in netpol-lab for ingress. F5 terminates on a Pod in netpol-mon, where no policy exists, so it is unaffected — including the fact that its source is a Pod the policy selects. A NetworkPolicy with policyTypes: [Ingress] says nothing whatsoever about what the selected Pods may send.

Two more things worth reading directly:

kubectl -n netpol-lab exec deploy/api -c shell -- \
  nslookup db.netpol-lab.svc.cluster.local

DNS still resolves. Egress is untouched, so the query reaches CoreDNS and the answer comes back — the connection dies afterwards, on the way in to db. Keep that distinction: a name that resolves and a port that will not open is an ingress problem, and it is a different first command from a name that will not resolve.

kubectl -n netpol-lab exec deploy/api -c shell -- echo "exec still works"

kubectl exec is served by the API server through the kubelet, not across the Pod network, which is why it keeps working under a total deny and is the right vehicle for every probe in this lab. It is also why an operator who tests connectivity only from their laptop learns nothing about Pod-to-Pod policy.

Task 6: Open the smallest hole, in both directions

The application has to work. F1 must come back without F2 coming back with it.

Write p2-db-allow-api.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-allow-api
  namespace: netpol-lab
spec:
  podSelector:
    matchLabels:
      app: db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 80

And p3-api-allow-mon.yaml for the cross-namespace scrape:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-mon
  namespace: netpol-lab
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              zone: monitoring
        - podSelector:
            matchLabels:
              app: scraper
      ports:
        - protocol: TCP
          port: 80

$ kubectl apply -f p2-db-allow-api.yaml -f p3-api-allow-mon.yaml

cd "$HOME/k8s-lab15"
./probe.sh | tee matrix-3-allowed.txt

$ ./probe.sh

flow   target                                         code
F1     http://db.netpol-lab.svc.cluster.local         200
F2     http://db.netpol-lab.svc.cluster.local         000
F3     http://api.netpol-lab.svc.cluster.local        200
F4     http://api.netpol-lab.svc.cluster.local        000
F5     http://10.244.2.9                              200

That is the intended matrix from the Architecture table, minus the egress row. Note what was not required: no rule mentions the api Pod’s egress, because nothing has isolated it for egress yet. Task 8 changes that, and F1 will break again even though db-allow-api is untouched.

p3-api-allow-mon.yaml is also wrong, in a way the matrix cannot see. Task 7 is about that.

Task 7: The from list is an OR, and that is how policies leak

Read p3-api-allow-mon.yaml again. Under from: there are two list entries: one namespaceSelector and one podSelector. Two entries mean either: traffic is admitted if it comes from any Pod in a namespace labelled zone=monitoring, or from any Pod labelled app=scraper in netpol-lab itself.

The rule reads, in English, “let the scraper in the monitoring namespace through”. It means “let the whole monitoring namespace through, and separately let anything in this namespace calling itself a scraper through”. Prove it:

$ kubectl -n netpol-lab label pod probe app=scraper --overwrite

cd "$HOME/k8s-lab15"
./probe.sh | tee matrix-4-leak.txt

F4 returns 200. A Pod that no rule was ever meant to admit reached api, because a workload author who can set labels on their own Pod can satisfy half of an OR. Nothing in the policy changed, nothing was applied, and no event was generated.

Fix it. Write p3-api-allow-mon-fixed.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-mon
  namespace: netpol-lab
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              zone: monitoring
          podSelector:
            matchLabels:
              app: scraper
      ports:
        - protocol: TCP
          port: 80

$ kubectl apply -f p3-api-allow-mon-fixed.yaml

cd "$HOME/k8s-lab15"
./probe.sh | tee matrix-5-fixed.txt
kubectl -n netpol-lab label pod probe app=probe --overwrite
./probe.sh

With the AND form, F4 returns 000 even while probe still wears the app=scraper label, because it is not in a namespace labelled zone=monitoring. F3 keeps working. Then the label is restored so the rest of the lab reads normally.

Task 8: Deny egress, and lose DNS in the same second

Ingress-only isolation stops the world reaching your Pods. It does nothing about a compromised Pod reaching the world — F5 has been returning 200 throughout. Close it.

Write p4-default-deny-egress.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: netpol-lab
spec:
  podSelector: {}
  policyTypes:
    - Egress

$ kubectl apply -f p4-default-deny-egress.yaml

cd "$HOME/k8s-lab15"
./probe.sh | tee matrix-6-egress-denied.txt

Every flow out of netpol-lab is now 000, F1 included, even though db-allow-api still permits exactly that connection on the way in. A same-namespace flow under two-way isolation needs two rules: an egress allow on the client and an ingress allow on the server. That symmetry is the single most common cause of a default-deny rollout that “half works”.

Diagnose it the way you would at 03:00, one layer at a time:

kubectl -n netpol-lab exec deploy/api -c shell -- \
  nslookup db.netpol-lab.svc.cluster.local

DB_IP=$(kubectl -n netpol-lab get pod -l app=db \
          -o jsonpath='{.items[0].status.podIP}')
echo "db pod IP: $DB_IP"

kubectl -n netpol-lab exec deploy/api -c shell -- \
  curl -s -m 4 -o /dev/null -w '%{http_code}\n' "http://$DB_IP"

nslookup fails first, which is the important part: the very first casualty of a default-deny egress is name resolution, and every symptom downstream of it looks like something else. The IP probe fails too, which tells you the DNS failure is not the whole story — the connection is being dropped on the way out, not merely mis-addressed.

Open the two holes, DNS first. Write p5-allow-dns.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: netpol-lab
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

And p6-api-allow-db-egress.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-db-egress
  namespace: netpol-lab
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: db
      ports:
        - protocol: TCP
          port: 80

$ kubectl apply -f p5-allow-dns.yaml -f p6-api-allow-db-egress.yaml

cd "$HOME/k8s-lab15"
./probe.sh | tee matrix-7-final.txt

F1 returns 200 again. F5 stays 000 — api may talk to db and to CoreDNS, and to nothing else. F2 and F4 stay 000, now for two independent reasons each. That is the intended posture from the Architecture table, reached by addition rather than by subtraction.

The DNS rule selects k8s-app: kube-dns, which is the label CoreDNS Pods carry on a kubeadm cluster. Confirm it rather than trusting it — a cluster running a different DNS deployment will use different labels, and this rule fails closed and silently when it matches nothing:

kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide

Task 9: Prove that no NetworkPolicy can deny anything

default-deny-ingress is still in place, and F2 is still blocked. Add a policy that is permissive rather than restrictive, and watch what the “deny” is worth.

Write p7-oops.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: temporary-debug-access
  namespace: netpol-lab
spec:
  podSelector:
    matchLabels:
      app: db
  policyTypes:
    - Ingress
  ingress:
    - {}

$ kubectl apply -f p7-oops.yaml

cd "$HOME/k8s-lab15"
./probe.sh
kubectl -n netpol-lab get networkpolicy

F2 returns 200. default-deny-ingress has not been deleted, edited or superseded — it is still listed, still selects db, still has no ingress rules. It simply never denied anything: policies are additive, and the effective rule for a Pod is the union of the allows in every policy that selects it. There is no precedence, no ordering, and no deny verb in the API.

$ kubectl -n netpol-lab delete networkpolicy temporary-debug-access

cd "$HOME/k8s-lab15"
./probe.sh | tee matrix-8-restored.txt
diff matrix-7-final.txt matrix-8-restored.txt && echo "POSTURE RESTORED"

Validation

Run all of these. Each one proves a different claim.

cd "$HOME/k8s-lab15"

# 1. The enforcer is installed and running on every node.
kubectl -n kube-system get daemonset canal
kubectl -n kube-system get pods -l k8s-app=canal -o wide

# 2. Only one CNI configuration is present on each node.
#    Substitute your own node names before running:
CP=k8s-cp-1
W1=k8s-w-1
W2=k8s-w-2
for NODE in "$CP" "$W1" "$W2"; do ssh "$NODE" 'ls /etc/cni/net.d/'; done

# 3. The six policies are present.
kubectl -n netpol-lab get networkpolicy

# 4. The final matrix matches the intended posture.
./probe.sh

# 5. The unenforced baseline is still on disk as evidence.
diff matrix-0-baseline.txt matrix-1-unenforced.txt && echo "flannel enforced nothing"

Expected results:

• daemonset canal shows DESIRED, CURRENT and READY all equal to the node count, and each Pod’s READY column reads 2/2.
• Each node lists 10-canal.conflist and nothing else.
• Six policies: default-deny-ingress, default-deny-egress, db-allow-api, api-allow-mon, allow-dns, api-allow-db-egress.
• The matrix reads F1 200, F2 000, F3 200, F4 000, F5 000.
• The diff in step 5 succeeds, printing flannel enforced nothing.

Expected Outcome

A cluster whose CNI enforces NetworkPolicy, one namespace in a proven default-deny posture with three deliberate holes, and a working directory that documents the whole transition:

k8s-lab15/
├── canal.yaml
├── estate.yaml
├── probe.sh
├── p1-default-deny-ingress.yaml
├── p2-db-allow-api.yaml
├── p3-api-allow-mon.yaml
├── p3-api-allow-mon-fixed.yaml
├── p4-default-deny-egress.yaml
├── p5-allow-dns.yaml
├── p6-api-allow-db-egress.yaml
├── p7-oops.yaml
├── matrix-0-baseline.txt
├── matrix-1-unenforced.txt
├── matrix-2-enforced.txt
├── matrix-3-allowed.txt
├── matrix-4-leak.txt
├── matrix-5-fixed.txt
├── matrix-6-egress-denied.txt
├── matrix-7-final.txt
├── matrix-8-restored.txt
├── nodes.pre-lab.txt
├── daemonsets.pre-lab.txt
├── netpol.pre-lab.txt
├── kube-system-pods.pre-lab.txt
└── flannel-pods.pre-lab.txt

You can state, with a file behind each claim: which CNI the cluster ran and what that meant, what the same policy did before and after the swap, which single label change defeated a rule that read correctly in English, why closing egress broke a flow whose ingress rule was already in place, and why an additive policy change is not automatically a safe one.

Troubleshooting

Nodes go NotReady after Task 4 and stay there. Look at the canal Pods first: kubectl -n kube-system get pods -l k8s-app=canal -o wide and then kubectl -n kube-system logs against the failing Pod, naming the container with -c calico-node or -c kube-flannel. The usual cause is a Pod CIDR mismatch — net-conf.json in the manifest must equal the --pod-network-cidr the cluster was initialised with, which Task 4 has you check before applying.

All five flows return 000 after Task 4, including F5. The Pods were not recreated, or CoreDNS was not. Re-run the delete/rollout block at the end of Task 4 and confirm kubectl -n kube-system get pods -l k8s-app=kube-dns shows Pods younger than the Canal rollout.

A flow that should be blocked is not, and the policy looks right. Check when the source Pod was created: kubectl -n netpol-lab get pods -o wide and compare AGE to the age of the canal DaemonSet. A Pod older than the swap has no Calico workload endpoint and is not filtered. Delete it and let it come back.

F3 stops working after Task 7. The AND form requires the namespace label. kubectl get ns netpol-mon --show-labels must show zone=monitoring. If the namespace was recreated at some point without the label, the rule matches nothing and fails closed.

nslookup works but every HTTP flow is 000, after Task 8. That is the correct symptom of an egress allow for DNS with no egress allow for the application. Confirm api-allow-db-egress exists and that its podSelector matches app: api rather than the Deployment name.

kubectl exec itself fails. That is not a policy problem — policy does not touch the exec path. Check the API server and kubelet, not the CNI.

Cleanup

Cleanup has two halves: remove what the lab created, and put the cluster’s CNI back the way Task 1 recorded it. Do both, in this order.

cd "$HOME/k8s-lab15"
kubectl delete namespace netpol-lab netpol-mon

$ kubectl delete -f $HOME/k8s-lab15/canal.yaml

$ kubectl apply -f $HOME/kubeadm-lab/kube-flannel.yml

kubectl -n kube-flannel rollout status daemonset/kube-flannel-ds --timeout=300s

Then clear Canal’s leftovers from each node, the same way Task 4 cleared flannel’s:

$ ssh "$NODE" 'sudo rm -f /etc/cni/net.d/10-canal.conflist /etc/cni/net.d/calico-kubeconfig && sudo rm -rf /var/lib/cni/networks/k8s-pod-network && ls -l /etc/cni/net.d/'

Finally, recreate the Pods so they are wired by flannel again, and prove the cluster is back:

kubectl -n kube-system delete pod -l k8s-app=kube-dns
kubectl -n kube-system rollout status deployment/coredns --timeout=300s
kubectl get nodes -o wide
kubectl -n kube-system get pods -o wide
kubectl get networkpolicy -A

Compare kubectl get nodes -o wide and kubectl -n kube-system get pods -o wide against nodes.pre-lab.txt and kube-system-pods.pre-lab.txt. All nodes Ready, one CNI configuration per node, and kubectl get networkpolicy -A back to whatever netpol.pre-lab.txt recorded.

Production notes

The CNI swap is the change, not the policy. Every policy in this lab is a namespaced object that can be applied and deleted in seconds with a blast radius of one namespace. Task 4 is the only step with real risk, and in production it is not a step — it is a cluster migration with its own plan, its own window, and its own rollback. If your cluster runs a CNI that does not enforce policy, the security ticket that asks for segmentation is a platform project, and saying so early is the whole job. Merging the policy object and closing the ticket, which is exactly what the scenario’s platform team did, converts a known gap into an unknown one.

Sequence a real default-deny rollout in this order. Confirm enforcement with a throwaway namespace and a connection test, exactly as Tasks 2 and 3 do. Build the probe matrix for the namespace you intend to lock down, from the flows the service actually uses — the ones nobody remembers are the reason rollouts fail. Apply the DNS allow before the deny, not after; it is idempotent and harmless on its own, and it removes the worst failure from the window. Then apply the ingress deny, re-run the matrix, and only then the egress deny with its own re-run.

“Hold” is a first-class outcome. If the matrix after the ingress deny shows a flow you cannot explain, stop there. An ingress-only default-deny with the application flows open is a genuine improvement and a stable state you can sit in for weeks. The owner of that hold is the team that owns the namespace, and the end condition is “the unexplained flow has a name and a rule”. Pushing on to the egress deny with an unexplained flow outstanding is how a change window turns into an incident.

The rollback is a delete, and it is fast — until it is not. Removing a NetworkPolicy restores traffic within the CNI’s programming interval, which is seconds. That makes policy changes unusually safe to reverse, and it is worth saying explicitly in the change record. What does not roll back that way is the Pod recreation in Task 4: a Pod restarted during a window comes back under whatever policy is live at that moment, so the rollback plan must name which workloads will be restarted and by whom.

Monitor the thing that has no status field. A NetworkPolicy has no conditions and reports nothing about its own effect, so there is no alert to build from the object. The monitorable artefact is the probe: a small scheduled job that runs the matrix from a Pod in each protected namespace and alerts when a flow’s verdict changes in either direction. A flow that starts working is as much a signal as a flow that stops.

What You Learned

• A policy is not evidence. matrix-0-baseline.txt and matrix-1-unenforced.txt are identical across a policy that describe renders as “Selected pods are isolated for ingress connectivity”. The only proof of enforcement is a connection that fails.
• The CNI, not the API server, decides whether policy is real. The same unchanged object went from inert to blocking four flows because a different DaemonSet was running — and the Pods that predated the swap stayed unpoliced until they were recreated.
• Isolation is per direction and per Pod. policyTypes: [Ingress] left every outbound flow open, and closing egress broke a flow whose ingress rule was already correct. Same-namespace traffic under two-way isolation needs two rules.
• Two entries in a from: list mean OR. One label on an unprivileged Pod was enough to satisfy half of that OR and walk in through a rule that read correctly in English. The AND form is one dash different.
• Nothing in the API can deny. An added policy widens the allowed set; the standing default-deny was never overridden because it was never doing the denying. That is why the review artefact for a policy change is the matrix, not the diff.