Skip to main content
RunBook Academy

← All break/fix scenarios in Kubernetes

intermediatekubernetes-networkpolicy~30 min

NetworkPolicy blocks app

Reported symptoms

  • ●The public checkout page returns 502 for one product area; the ingress controller Pods are healthy and every other backend behind the same controller is fine
  • ●Every Grafana panel for the billing service went flat at the same minute, and the metrics team can show that Prometheus is scraping every other namespace normally
  • ●`kubectl get pods -n billing` shows every Pod Running and Ready, and none of them have restarted
  • ●`kubectl get endpointslice -n billing` lists all six Pod addresses as ready, so the Service is not the usual empty-endpoints fault
  • ●A nightly canary Job in the billing namespace times out calling the billing Service; it was closed twice as a flaky test
  • ●The billing containers log nothing unusual, because no connection ever arrives for them to log
  • ●The change that preceded all of this was reviewed and merged as "tighten billing ingress, no functional change" - a policy that only adds an allow rule

Evidence

  • · `kubectl get networkpolicy -n billing` lists `billing-default-deny`, `billing-allow-dns` and the new `billing-allow-ingress`
  • · `kubectl get networkpolicy billing-allow-ingress -n billing -o yaml` shows one `from` entry containing both a namespaceSelector and a podSelector
  • · `kubectl get pods -n monitoring -l app=api-gateway` returns no resources
  • · `kubectl exec -n billing deploy/canary -- curl -sS --max-time 5 http://billing:8080/` hangs for the full five seconds and exits 28, rather than being refused
  • · `kubectl exec -n billing deploy/billing -- curl -sS localhost:8080/healthz` returns 200 immediately
  • · The billing Deployment uses an `exec` readiness probe, which runs inside the container and never touches the network
  • · `kubectl describe networkpolicy` renders the rule in a form where an AND-combined peer and two OR-combined peers look nearly identical
  • · The CNI is Calico, which does enforce NetworkPolicy, so the rules on the page are the rules in the dataplane
Diagnosis and resolutionclick to reveal

Root cause

The new policy was meant to allow two sources into the billing Pods on port 8080: the api-gateway Pods in the billing namespace, and Prometheus in the monitoring namespace. It was written as a single `from` entry holding a namespaceSelector and a podSelector together. Selectors inside one entry are AND-combined; selectors across entries are OR-combined. Written that way the rule permits exactly one thing - a Pod labelled `app: api-gateway` that lives in the `monitoring` namespace - and no such Pod exists anywhere in the cluster. The rule therefore matches nothing. Because the namespace already carried a default-deny ingress policy, and because NetworkPolicy rules are purely additive with no way to deny, a rule that matches nothing leaves the deny standing and every inbound connection to billing is dropped. The whole defect is one list marker: with a leading dash on the podSelector the two selectors are separate peers and the rule reads as intended; without it they are one peer and the rule is unsatisfiable. Everything else in the incident follows from the fact that a dropped connection produces no evidence anywhere except at the client, which is in a different team and a different ticket.

Remediation

Split the single peer into two, so the rule expresses OR rather than AND, and prefer to express it as two separate NetworkPolicy objects with descriptive names rather than as two list entries in one - the distinction that caused this cannot be encoded in indentation if each intent has its own object. Restoring service is not the same as reverting the commit: with a default-deny policy in place, deleting the broken allow rule leaves the namespace with no ingress at all and the outage continues. The rollback is to reapply the previous allow policy, and if the previous state was a namespace with no default-deny at all, then reverting means reopening the namespace entirely, which is a security decision and not a rollback. Where the earlier policy cannot be recovered quickly, a deliberately broad temporary allow on port 8080 is a defensible hold, provided it is recorded as a security exception with a named owner and a stated expiry rather than left to become permanent.

Verification

A NetworkPolicy is only verified by connection tests, in both directions. Prove the intended traffic now works: the canary Job completes, the ingress controller reaches billing and the 502s stop, and the Prometheus targets in the billing namespace return to green and stay green for a full scrape interval plus a margin. Then prove the policy still denies what it is supposed to deny: run a Pod in a namespace that has no business talking to billing and confirm the connection hangs and times out rather than connecting. That negative test is the one that matters, because the most common way this incident is closed badly is by deleting the default-deny, which makes every positive test pass and silently removes the control the change was made to add. Confirm the selectors match real objects, not just that the YAML parses: `kubectl get pods` with each selector must return the Pods you expect.

Prevention

Treat a NetworkPolicy change as a change with two required test results, a positive and a negative, and make both of them part of the review artefact rather than the YAML diff alone. Give each intent its own policy object with a name that states the intent, so the AND versus OR distinction is never carried by a list marker. Lint every selector against the live cluster before merge: a podSelector that matches zero Pods is a rule that does nothing, and it is cheap to detect and impossible to see by reading. Remember that policies are additive and never deny, so a broken allow rule always fails closed and always fails silently - there is no error, no event and no log line, only absence. Do not read Ready as reachable: an exec readiness probe asserts that a process is alive inside the container and asserts nothing about whether any packet can arrive. Finally, confirm at cluster level that the CNI enforces NetworkPolicy at all, because on a dataplane that ignores it every one of these tests passes for the wrong reason.

Reported symptoms

Three tickets, opened within twenty minutes of each other, by three teams who have not spoken to one another.

  • Storefront. Checkout returns 502 for one product area. The platform team confirms the ingress controller is healthy, its Pods have not restarted, and every other backend behind the same controller is serving normally. They hand the ticket to the billing team.
  • Observability. Every Grafana panel for billing went flat at once. The metrics team demonstrates that Prometheus is up, that its other targets are being scraped, and that nothing in their configuration changed. They hand the ticket to the billing team.
  • Quality. The nightly canary Job that calls the billing Service from inside the same namespace timed out. It has been closed as flaky twice before, so it is closed as flaky again.

The billing team looks at their service and finds nothing wrong. Every Pod is Running and Ready. Nothing has restarted. The containers log normal startup and then nothing at all, which is what a healthy idle service looks like. kubectl exec into a Pod and curl localhost:8080/healthz returns 200 instantly. The Service has endpoints - all six Pod addresses, all marked ready - so the usual empty-endpoints fault is ruled out early and confidently.

The only change anywhere near this is a NetworkPolicy merged the previous afternoon, reviewed and approved with the note “tighten billing ingress, no functional change”. Two reviewers looked at it. It adds an allow rule. Nobody believes an added allow rule can take a service off the network, which is why the change is not suspected for six hours.

Evidence provided

Read-only / Safea default-deny, a DNS allow, and yesterday's new rule
$ kubectl get networkpolicy -n billing
NAME                    POD-SELECTOR   AGE
billing-allow-dns       <none>         96d
billing-allow-ingress   app=billing    19h
billing-default-deny    <none>         96d

Illustrative output

Read-only / Saferead the list markers, not the words
$ kubectl get networkpolicy billing-allow-ingress -n billing -o yaml
spec:
podSelector:
matchLabels:
app: billing
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
podSelector:
matchLabels:
app: api-gateway
ports:
- protocol: TCP
port: 8080

Illustrative output

Read-only / Safethe set of sources this rule permits
$ kubectl get pods -n monitoring -l app=api-gateway
No resources found in monitoring namespace.

Illustrative output

Read-only / Safetimed out, not refused - the packet was dropped, not rejected
$ kubectl exec -n billing deploy/canary -- curl -sS --max-time 5 -o /dev/null -w '%{http_code}' http://billing:8080/
curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
command terminated with exit code 28

Illustrative output

Read-only / Safethe application is fine, and always was
$ kubectl exec -n billing deploy/billing -- curl -sS -o /dev/null -w '%{http_code}' localhost:8080/healthz
200

Illustrative output

Read-only / Safesix ready endpoints, none of them reachable
$ kubectl get endpointslice -n billing -l kubernetes.io/service-name=billing -o jsonpath='{.items[*].endpoints[*].conditions.ready}'
true true true true true true

Illustrative output

Work the evidence before reading on

Four facts are on the table. Three of them are consistent with a healthy service and one is not.

  1. The application answers on loopback. So the process is fine and the port is open.
  2. The Service has six ready endpoints. So Pod readiness, label selectors on the Service, and the endpoint controller are all fine.
  3. A caller inside the same namespace times out rather than being refused. A closed port refuses immediately; a dropped packet produces exactly this - silence until the client gives up.
  4. The only recent change adds an allow rule to a namespace that already has a default-deny.

Before reading on, look again at the YAML above and answer one question: the rule names two selectors. Are they two permitted sources, or one - and which character in that file decides?

Then ask what set of Pods actually satisfies the reading you arrived at, and whether any such Pod exists.

Root cause

1. Selectors inside one peer are AND-combined

A NetworkPolicy from block is a list of peers. Each list entry is one permitted source. Within a single entry, the selectors are combined with AND: a source must satisfy all of them. Across entries, the combination is OR: a source may satisfy any one of them.

The rule as merged has one entry:

ingress:
- from:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: monitoring
    podSelector:
      matchLabels:
        app: api-gateway

There is one dash under from, so there is one peer, so this reads as: a Pod labelled app: api-gateway, in a namespace labelled kubernetes.io/metadata.name: monitoring. The api-gateway runs in the billing namespace and Prometheus runs in monitoring, so the intersection is empty. The rule permits nothing.

What was intended is two peers:

ingress:
- from:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: monitoring
  - podSelector:
      matchLabels:
        app: api-gateway

One extra dash and two fewer spaces. That is the entire defect, and it changes the meaning of the file from “nothing” to “two things”.

2. A rule that matches nothing does not fail; it just does not help

NetworkPolicy has no deny. Every rule is additive: the set of permitted traffic is the union of everything every policy allows, and nothing subtracts from it. That design has a consequence worth stating plainly - a broken allow rule is indistinguishable from an absent one. There is no validation failure, because the YAML is valid. There is no admission error, because the selectors are well-formed. There is no event, no log line and no metric, because from the cluster’s point of view nothing went wrong: a rule was created, and it permits an empty set.

The namespace still has billing-default-deny, which selects every Pod and declares Ingress. With no rule permitting anything, that deny is the whole policy for billing, and it silently was from the moment the change merged.

3. Every health check the team ran was blind to this by construction

This is why six hours passed before the policy was suspected.

The readiness probe is an exec probe. It runs a command inside the container, so no packet crosses any interface and no policy is consulted. It reports the process is alive, which is true, and it reports it whether or not anything in the cluster can reach the Pod.

Because readiness passes, the endpoint controller lists the Pod addresses as ready in the EndpointSlice. So kubectl get endpointslice shows six healthy endpoints. Readiness is a statement the Pod makes about itself; it is not a measurement of reachability, and the endpoint list inherits that limitation.

The loopback curl has the same blind spot for the same reason. Traffic to localhost never leaves the Pod network namespace.

Three independent checks, all green, none of which sends a packet along the path that is broken.

Resolution

  1. Decide first whether you are restoring service or reverting a commit, because with a default-deny in place they are different actions. Deleting billing-allow-ingress does not restore anything: it leaves the namespace with a deny and no allow, which is the current outage with fewer objects in it.
  2. Restore service by reapplying the previous allow policy from version control, verbatim. If that policy cannot be recovered in the time available, apply a deliberately broad temporary rule permitting TCP 8080 to app=billing, and record it in the incident as a security exception with a named owner and an expiry, not as the fix.
  3. Confirm the restore with a connection test rather than with a green dashboard. The canary Job, or a one-off Pod in the billing namespace, must reach billing:8080 and get a 200.
  4. Now write the intended rule properly. Give each permitted source its own policy object - one named for the gateway, one named for the monitoring scrape - so that the AND-versus-OR distinction is carried by object boundaries rather than by a list marker that a reviewer has to notice.
  5. Before applying, prove every selector matches something real. kubectl get pods -n billing -l app=api-gateway and kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus must both return Pods. A selector that matches zero objects is a rule that does nothing, and it is the same defect in a different shape.
  6. Apply the two policies, then remove the temporary broad rule in the same change window. A temporary exception that outlives the incident is how a default-deny quietly becomes decorative.
  7. Write down in the incident record which reading of the rule was intended and which was applied, with both YAML fragments. The next person to write a two-source policy will find it, and the failure is not memorable enough to survive as folklore.

Verification

  1. The positive test: a Pod running as the api-gateway in the billing namespace reaches billing:8080 and receives a 200. Run it from a Pod carrying the real labels, not from a debug Pod with no labels, because the labels are what the policy matches on.
  2. The second positive test: the Prometheus targets for the billing namespace return to UP and stay UP across at least three scrape intervals. One successful scrape can be a retry landing in a gap.
  3. The negative test, which is the one that can fail and the one most often skipped: a Pod in an unrelated namespace attempts billing:8080 and must time out. If it connects, the default-deny is not in force any more and the incident has been closed by removing the control rather than by fixing the rule.
  4. The ingress path end to end: the checkout page returns 200 from outside the cluster, not merely from inside it. The 502s were the reported symptom and they are what the storefront team will check.
  5. Every selector in the new policies resolves to at least one live object, checked with kubectl get pods -l for each. Re-run this after any namespace relabelling, because kubernetes.io/metadata.name is set by the control plane but other namespace labels are not.
  6. The temporary broad allow rule no longer exists. kubectl get networkpolicy -n billing lists only the policies you intended to keep.
  7. The canary Job that was closed twice as flaky is re-enabled and passes on its next scheduled run. It was the only automated test in the estate that exercised the broken path, and it was right both times.

Prevention

  • Require two test results on every NetworkPolicy change: one connection that must succeed and one that must still fail. Attach both to the review. A YAML diff is not evidence about a dataplane.
  • Give every intent its own policy object with a name that states the intent. Two sources means two objects. The AND-versus-OR distinction then cannot be expressed by indentation, which is the only form in which it is invisible.
  • Lint selectors against the live cluster before merge. A podSelector or namespaceSelector matching zero objects should block the change; it takes one kubectl get per selector and it catches the whole family of unsatisfiable rules.
  • Know which namespaces carry a default-deny, and treat every policy change in those namespaces as service-affecting by default. The blast radius of a policy edit depends entirely on what is already denying.
  • Do not read Ready as reachable. An exec probe, and to a lesser degree a probe that only touches loopback, tells you a process is alive. If your readiness signal never crosses the interface, it cannot detect anything that blocks the interface.
  • Verify once, at cluster level, that the CNI enforces NetworkPolicy at all. Flannel on its own does not, and on such a cluster every negative test above passes for the wrong reason - which is a worse outcome than this incident, because it is permanent and silent.