Lab 13: Diagnose a Service that routes nowhere

Objective

By the end of this lab you will have four broken Services in front of you. Three of them return the same error to the client, character for character. You will name the failing layer for each one from read-only control-plane evidence, repair all four, and prove the repair with a script that exits non-zero if you were wrong.

The skill being built is not “know the fix”. It is: stop treating the client’s error message as a diagnosis. At the client, a Service with no matching Pods, a Service pointed at a port nothing listens on, and a Service whose Pods have never passed readiness are one single symptom. The evidence that separates them lives one API call away and costs nothing to collect.

Architecture

Two namespaces. One healthy workload with a correct Service in front of it as a control, three Services broken in three different places, and a fourth fault that lives in the client’s resolver rather than in any Service at all.

svc-lab                                        svc-lab-ext
├── deploy/orders          3 x nginx:1.27.2    └── deploy/payments   1 x nginx
│      labels app=orders, probe GET /             svc/payments  80 -> http
├── deploy/orders-canary   1 x nginx:1.27.2
│      labels app=orders-canary, probe GET /healthz  (nginx answers 404)
├── svc/orders    selector app=orders            80 -> http     healthy control
├── svc/orders-a  selector app=orders,tier=api   80 -> 80       fault A
├── svc/orders-b  selector app=orders            80 -> 8080     fault B
├── svc/orders-c  selector app=orders-canary     80 -> 80       fault C
└── pod/client    nicolaka/netshoot, sleep infinity

Fault D has no object of its own: it is the client asking for payments by its short name from a namespace that does not contain it.

flowchart LR
    C[client Pod] --> L1[Layer 1 DNS]
    L1 --> L2[Layer 2 Service ClusterIP]
    L2 --> L3[Layer 3 EndpointSlice]
    L3 --> L4[Layer 4 Pod IP]
    L4 --> L5[Layer 5 application port]
    D[Fault D short name] -.-> L1
    A[Fault A selector] -.-> L3
    CC[Fault C readiness] -.-> L3
    B[Fault B targetPort] -.-> L5

Four faults, three layers, one client-visible symptom for three of them.

Requirements

A kubeadm cluster on Kubernetes 1.34.x, built as in Lab 01, with kubectl 1.34.x and cluster-admin on it. A single-node cluster works if the control-plane taint has been removed; three nodes is closer to production but changes nothing in this lab.
A working CNI. Every Pod must get an IP and be able to reach every other Pod. This lab does not test the CNI; if pod-to-pod is already broken, none of the evidence below will mean what it says.
Ability to pull nginx:1.27.2 and nicolaka/netshoot. If the cluster has no route to a registry, mirror those two images first. The lab names no others.
No out-of-band access requirement. Every object the lab creates lives in two namespaces it also creates. Nothing touches kube-system, the CNI, the kubelet, or any node.
Blast radius: two namespaces, svc-lab and svc-lab-ext. Cleanup deletes both. Nothing outside them is modified, so there is no pre-lab state to capture on the host.

Read your kube-proxy mode before Task 3

The client-side symptom of a Service with no ready endpoints depends slightly on how kube-proxy is programming the data plane.

kubectl -n kube-system get configmap kube-proxy -o yaml | grep -E '^\s+mode:'

An empty value or mode: "" means the default, which is iptables. Note what you find; Task 4 refers back to it.

Scenario

03:10. The checkout service is returning 5xx and the on-call dashboard blames the orders API. You look at orders and everything is green: the Deployment reports 3/3, the Pods are Running, CPU is flat, and no image has changed in nine days.

The only evidence anyone has is one line from the client’s application log:

dial tcp 10.96.144.21:80: connect: connection refused

That line is compatible with at least three completely different faults, two of which are in the Service definition and one of which is in the workload. Guessing costs a rollout of a Deployment that was never broken — and a rollout at 03:10 replaces the fault you have with a fault you have not seen before.

This lab builds all three faults, plus a fourth that does look different, so you can practise telling them apart on a cluster where being wrong costs nothing.

Tasks

Task 1 — Build the namespaces and the healthy baseline

Write svc-lab.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: svc-lab
  labels:
    lab: svc-lab
---
apiVersion: v1
kind: Namespace
metadata:
  name: svc-lab-ext
  labels:
    lab: svc-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  namespace: svc-lab
  labels:
    lab: svc-lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders
  template:
    metadata:
      labels:
        app: orders
    spec:
      containers:
        - name: nginx
          image: nginx:1.27.2
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: orders
  namespace: svc-lab
  labels:
    lab: svc-lab
spec:
  selector:
    app: orders
  ports:
    - name: http
      port: 80
      targetPort: http
      protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
  namespace: svc-lab-ext
  labels:
    lab: svc-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: payments
  template:
    metadata:
      labels:
        app: payments
    spec:
      containers:
        - name: nginx
          image: nginx:1.27.2
          ports:
            - name: http
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: svc-lab-ext
  labels:
    lab: svc-lab
spec:
  selector:
    app: payments
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: svc-lab
  labels:
    app: client
    lab: svc-lab
spec:
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      command: ["sleep", "infinity"]

Configuration changeworkstation— Creates two namespaces and everything in them. Touches nothing else.

$ kubectl apply -f svc-lab.yaml

kubectl -n svc-lab rollout status deployment/orders --timeout=120s
kubectl -n svc-lab-ext rollout status deployment/payments --timeout=120s
kubectl -n svc-lab wait --for=condition=Ready pod/client --timeout=120s

Task 2 — Walk the canonical flow while nothing is wrong

This is the task people skip, and it is the one that makes the rest of the lab cheap. Run the five layers against the healthy orders Service and keep the output.

NS=svc-lab

# Layer 1 - DNS: does the name resolve, and to what
kubectl exec -n "$NS" client -- nslookup orders.svc-lab.svc.cluster.local

# Layer 2 - Service: what does the object claim
kubectl get service orders -n "$NS" -o yaml

# Layer 3 - EndpointSlice: which Pods does the controller say back it
kubectl get endpointslices -n "$NS" -l kubernetes.io/service-name=orders -o yaml

# Layer 4 - Pod IP: what addresses exist
kubectl get pods -n "$NS" -l app=orders -o wide

# Layer 5 - application port: what is actually listening
kubectl exec -n "$NS" client -- curl -s -o /dev/null -w '%{http_code}\n' http://orders

The layer that matters most, and the one people read last, is layer 3:

Read-only / Safeworkstation— The authoritative answer to: which Pods does this Service route to

$ kubectl get endpointslices -n svc-lab -l kubernetes.io/service-name=orders

NAME            ADDRESSTYPE   PORTS   ENDPOINTS                             AGE
orders-7hq2v    IPv4          80      10.244.1.14,10.244.2.9,10.244.1.15    41s

Illustrative output

Three addresses, one port. Record four things from this task, because every later task is a comparison against them:

The ClusterIP of orders.
The three Pod IPs and the port in the EndpointSlice.
The READY column from kubectl get pods -n svc-lab — all 1/1.
The restart counts — all 0.

Task 3 — Break it four ways

Write svc-lab-faults.yaml:

apiVersion: v1
kind: Service
metadata:
  name: orders-a
  namespace: svc-lab
  labels:
    lab: svc-lab
spec:
  selector:
    app: orders
    tier: api
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: orders-b
  namespace: svc-lab
  labels:
    lab: svc-lab
spec:
  selector:
    app: orders
  ports:
    - name: http
      port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-canary
  namespace: svc-lab
  labels:
    lab: svc-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: orders-canary
  template:
    metadata:
      labels:
        app: orders-canary
    spec:
      containers:
        - name: nginx
          image: nginx:1.27.2
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: orders-c
  namespace: svc-lab
  labels:
    lab: svc-lab
spec:
  selector:
    app: orders-canary
  ports:
    - name: http
      port: 80
      targetPort: 80

Configuration changeworkstation— Creates three broken Services and one Deployment that never becomes Ready

$ kubectl apply -f svc-lab-faults.yaml

Give the canary thirty seconds to fail its probe, then ask the client what it sees. This is the entire diagnostic most teams have:

for s in orders orders-a orders-b orders-c payments; do
  out=$(kubectl exec -n svc-lab client -- \
        curl -sS -o /dev/null -m 5 -w '%{http_code}' "http://$s" 2>&1)
  printf '%-10s %s\n' "$s" "$out"
done

Read-only / Safeworkstation— One symptom, four causes. Three of these lines are identical.

$ bash client-probe.sh

orders     200
orders-a   000curl: (7) Failed to connect to orders-a port 80 after 1 ms: Could not connect to server
orders-b   000curl: (7) Failed to connect to orders-b port 80 after 2 ms: Could not connect to server
orders-c   000curl: (7) Failed to connect to orders-c port 80 after 1 ms: Could not connect to server
payments   000curl: (6) Could not resolve host: payments

Illustrative output

Read that carefully. orders-a, orders-b and orders-c are broken at three different layers by three unrelated mistakes, and the client cannot tell them apart — not because curl is unhelpful, but because from a TCP client’s point of view all three genuinely are the same event. Only payments fails differently, and it fails at the one layer that happens before TCP exists.

Note also how fast the failures are: single-digit milliseconds. A refusal is not a timeout. If you get a five-second hang instead, you are looking at a different class of problem — a dropped packet rather than a rejected connection — and the first suspect is a NetworkPolicy or the CNI, not the Service. That distinction is the subject of Lab 15.

Task 4 — Fault A: the Service that selects nothing

Start at layer 3, because layer 3 is where a Service either has backends or does not, and the answer is one command:

Read-only / Safeworkstation— Layer 3 for the first broken Service

$ kubectl get endpointslices -n svc-lab -l kubernetes.io/service-name=orders-a

NAME             ADDRESSTYPE   PORTS   ENDPOINTS   AGE
orders-a-kx4ml   IPv4          <unset>   <unset>     94s

Illustrative output

No addresses. kubectl describe service orders-a says the same thing in one line: Endpoints: followed by <none>. The EndpointSlice controller is working perfectly — it looked for Pods matching the selector, found none, and wrote a placeholder slice recording that fact.

So compare the two halves of the match:

kubectl get service orders-a -n svc-lab -o jsonpath='{.spec.selector}{"\n"}'
kubectl get pods -n svc-lab -l app=orders --show-labels

The Service asks for app=orders,tier=api. The Pods carry app=orders and a pod-template-hash, and nothing else. Selector labels are ANDed, so one absent label is a total miss, not a partial one.

Now decide which side is wrong — and notice that the evidence does not tell you

There are two repairs and they are not equivalent:

Remove tier: api from the Service. Instant, no rollout, no Pod restarts. It is also a widening: if tier was there to stop this Service picking up a second workload that also carries app=orders, you have just pointed production traffic at that workload.
Add tier: api to the Pod template. Correct if the label was meant to exist. It rewrites the Deployment’s template, which triggers a full rolling replacement of all three Pods — at 03:10, on a workload with no proven fault.

The cluster cannot tell you which is right, because both produce a Service with endpoints. Only the intent behind tier decides, and the intent is in someone’s head or in a git history.

Holding is a first-class option here. If the Service was working an hour ago and nobody deployed, the selector did not change itself — something changed the Pod labels, and finding that is a better use of the next ten minutes than guessing. Write the hold down: “holding orders-a at 03:15, owner = the orders service owner on call, decision point 03:45; if unreached, widen the Service selector as the reversible option and open a follow-up.”

The reversible option is the one you can undo without a rollout. That is the tiebreaker when nobody answers.

For this lab, tier was a typo that never existed anywhere else, so remove it:

Configuration changeworkstation— Removes one key from the Service selector. No rollout, no Pod restart.

$ kubectl patch service orders-a -n svc-lab --type=json -p='[{"op": "remove", "path": "/spec/selector/tier"}]'

kubectl get endpointslices -n svc-lab -l kubernetes.io/service-name=orders-a
kubectl exec -n svc-lab client -- curl -s -o /dev/null -w '%{http_code}\n' http://orders-a

Three addresses appear within a second or two, and the client gets 200. Nothing restarted: check the restart counts against your Task 2 baseline and confirm they are still 0.

Task 5 — Fault B: the same symptom, one layer deeper

orders-b produced a byte-identical client error. Run the same layer-3 command and watch it produce a different answer:

Read-only / Safeworkstation— Layer 3 for the second broken Service. Compare with Task 4.

$ kubectl get endpointslices -n svc-lab -l kubernetes.io/service-name=orders-b

NAME             ADDRESSTYPE   PORTS   ENDPOINTS                             AGE
orders-b-2wc8f   IPv4          8080    10.244.1.14,10.244.2.9,10.244.1.15    3m2s

Illustrative output

Endpoints are present. The selector matched, the Pods are Ready, the EndpointSlice is populated — and the PORTS column reads 8080 where the healthy Service reads 80. That is the whole diagnosis, visible in the same command that diagnosed fault A.

The slice’s port is the resolved targetPort, so it is the cluster’s record of where traffic will actually be delivered. Compare it against what the container declares:

kubectl get pods -n svc-lab -l app=orders \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].ports[*].containerPort}{"\n"}{end}'

Then isolate the layer without touching the Service at all, by going straight to a Pod IP and skipping the ClusterIP entirely:

POD_IP=$(kubectl get pods -n svc-lab -l app=orders \
  -o jsonpath='{.items[0].status.podIP}')

kubectl exec -n svc-lab client -- curl -s -o /dev/null -m 5 -w '%{http_code}\n' "http://$POD_IP:80"
kubectl exec -n svc-lab client -- curl -sS -o /dev/null -m 5 -w '%{http_code}\n' "http://$POD_IP:8080"

Port 80 answers 200. Port 8080 refuses. The Pod, the CNI and the routing are all fine; the Service is DNATing to a port nothing binds. This two-command test is worth memorising, because it cleanly separates “the Service is wrong” from “the workload is wrong” and needs no cluster changes to run.

Fix it the durable way — with the named port the healthy Service already uses, so the Service survives a change of container port:

Configuration changeworkstation— Repoints targetPort at the container's named port. Takes effect immediately for new connections.

$ kubectl patch service orders-b -n svc-lab -p='{"spec":{"ports":[{"name":"http","port":80,"targetPort":"http","protocol":"TCP"}]}}'

Task 6 — Fault C: endpoints that exist and are not ready

Same command, third answer:

Read-only / Safeworkstation— Layer 3 for the third broken Service. Endpoints exist. Look at where they are not.

$ kubectl get endpointslices -n svc-lab -l kubernetes.io/service-name=orders-c -o yaml

addressType: IPv4
endpoints:
- addresses:
- 10.244.2.11
conditions:
  ready: false
  serving: false
  terminating: false
targetRef:
  kind: Pod
  name: orders-canary-6c9d47b8f4-t8xzq
ports:
- name: http
port: 80
protocol: TCP

Illustrative output

The address is listed and the port is right, but ready: false. kube-proxy only programs endpoints whose ready condition is true, so this Service has an EndpointSlice with an entry in it and still no backends. The ENDPOINTS column in the short output is misleading here for exactly that reason — read the conditions, not the count.

The three conditions are not synonyms:

ready — the Pod passed its readiness gate. This is what routing uses.
serving — the Pod is capable of serving, even while terminating. It lets a client that already has a connection drain gracefully.
terminating — the Pod has a deletion timestamp.

ready: false, serving: false on a Pod with no deletion timestamp means it has never been ready, not that it is going away. Confirm at the Pod:

kubectl get pods -n svc-lab -l app=orders-canary
kubectl describe pod -n svc-lab -l app=orders-canary | tail -20

READY 0/1, STATUS Running, and an event stream repeating Readiness probe failed: HTTP probe failed with statuscode: 404. The container started, the process is up, the port is bound — and nginx has no /healthz, so it answers 404 and the kubelet correctly refuses to mark the Pod ready.

kubectl patch deployment orders-canary -n svc-lab --type=json \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/"}]'

kubectl -n svc-lab rollout status deployment/orders-canary --timeout=120s
kubectl get endpointslices -n svc-lab -l kubernetes.io/service-name=orders-c

Note that this repair is a rollout — the probe lives in the Pod template, so changing it replaces the Pod. That is the third distinct cost profile in three faults: fault A cost nothing, fault B cost a data-plane rewrite, fault C costs a Pod replacement.

Task 7 — Fault D: the failure that looks different

payments failed with Could not resolve host, not a refused connection. That is layer 1, and it means no TCP connection was ever attempted.

kubectl exec -n svc-lab client -- cat /etc/resolv.conf
kubectl exec -n svc-lab client -- nslookup payments
kubectl exec -n svc-lab client -- nslookup payments.svc-lab-ext.svc.cluster.local

Read-only / Safeclient Pod— The resolver configuration the kubelet wrote for this Pod

$ kubectl exec -n svc-lab client -- cat /etc/resolv.conf

nameserver 10.96.0.10
search svc-lab.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Illustrative output

The search path is built from the Pod’s own namespace. A short name payments is tried as payments.svc-lab.svc.cluster.local, then payments.svc.cluster.local, then payments.cluster.local, and finally as an absolute name against the upstream resolver. None of those is payments.svc-lab-ext.svc.cluster.local, so every attempt is NXDOMAIN and the resolver gives up.

Nothing is broken. CoreDNS answered every query correctly and quickly. The client asked the wrong question.

The fix requires no cluster change at all — use the fully qualified name:

kubectl exec -n svc-lab client -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://payments.svc-lab-ext.svc.cluster.local

Task 8 — The triage order, and what it buys you

Go back and look at what actually did the work. Every one of the first three faults was diagnosed by the same two read-only commands, run before anything else:

kubectl get svc,endpointslices -n svc-lab
kubectl get pods -n svc-lab -o wide --show-labels

They cost one round trip each, change nothing, need no exec into a Pod, and require no access to a node. Run them first — always — because they collapse the one client symptom into three distinguishable states:

What the two commands show	Failing layer	Fault
Slice exists, no addresses	3 — selector matched no Pod	A
Addresses present, `PORTS` disagrees with the healthy Service	5 — nothing listens on that port	B
Addresses present, Pods show `0/1` READY	3 — no endpoint is `ready`	C
Name does not resolve at all	1 — DNS, usually the search path	D

The expensive tools — exec into the workload, kubectl debug node, reading iptables, packet capture — are for the cases these two commands do not settle. Reaching for them first is how a fifteen-minute incident becomes a two-hour one.

Validation

Save as validate.sh and run it. It exits non-zero while any fault remains, which is the point: a validation you can pass by squinting at output is not a validation.

#!/usr/bin/env bash
set -euo pipefail
NS=svc-lab

for s in orders orders-a orders-b orders-c; do
  ready=$(kubectl get endpointslices -n "$NS" -l "kubernetes.io/service-name=$s" \
    -o jsonpath='{range .items[*].endpoints[*]}{.conditions.ready}{"\n"}{end}' \
    | grep -c true || true)
  if [ "$ready" -lt 1 ]; then
    echo "FAIL $s has no ready endpoint"
    exit 1
  fi
  echo "ok   $s has $ready ready endpoint(s)"
done

for s in orders orders-a orders-b orders-c; do
  code=$(kubectl exec -n "$NS" client -- \
    curl -s -o /dev/null -m 5 -w '%{http_code}' "http://$s")
  if [ "$code" != "200" ]; then
    echo "FAIL $s returned $code"
    exit 1
  fi
  echo "ok   $s returned 200"
done

code=$(kubectl exec -n "$NS" client -- curl -s -o /dev/null -m 5 \
  -w '%{http_code}' "http://payments.svc-lab-ext.svc.cluster.local")
if [ "$code" != "200" ]; then
  echo "FAIL payments returned $code by FQDN"
  exit 1
fi
echo "ok   payments reachable by FQDN"

echo "--- restart counts, compare against the Task 2 baseline"
kubectl get pods -n "$NS" \
  -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount

Three separate claims are being proved, and they are not the same claim:

The EndpointSlice has a ready endpoint. Control-plane truth: the Service has backends the data plane will use.
The client gets 200. Data-plane truth: the path from a Pod through DNS, kube-proxy and the CNI to the application actually works. A Service can have ready endpoints and still be unreachable, which is what Lab 15 is about.
Restart counts are unchanged for orders. You fixed four faults without restarting the workload that was never at fault. If orders shows restarts, something you did was wider than it needed to be.

The orders-canary Pod will show as a new Pod after Task 6, because a probe change is a template change. That one is expected and is the difference the table in Task 8 is teaching.

Expected Outcome

svc-lab
├── deploy/orders          3/3 Ready, 0 restarts since Task 1
├── deploy/orders-canary   1/1 Ready, replaced once by the Task 6 probe fix
├── svc/orders     -> 3 ready endpoints, port 80
├── svc/orders-a   -> 3 ready endpoints, port 80   (selector repaired)
├── svc/orders-b   -> 3 ready endpoints, port 80   (targetPort repaired)
├── svc/orders-c   -> 1 ready endpoint,  port 80   (probe repaired)
└── pod/client     -> 200 from all four, 200 from payments by FQDN

svc-lab-ext
└── svc/payments   -> 1 ready endpoint, reachable only by FQDN from svc-lab

validate.sh exits 0. You have a written fault table mapping each Service to its layer, and the two-command triage from Task 8 in your notes.

Troubleshooting

Every Service returns 000, including orders. The control is broken, so this is not a Service fault. Check that the client Pod is Running in svc-lab (kubectl get pod client -n svc-lab -o wide) and that Pods have IPs. If pod-to-pod is broken the CNI is the suspect, and this lab cannot proceed.

ImagePullBackOff on the client Pod. nicolaka/netshoot comes from Docker Hub and anonymous pulls are rate limited. Any image with a shell, an HTTP client and a resolver works instead — this course also uses registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7. Before you depend on a substitute, ask it what it has: kubectl exec -n svc-lab client -- sh -c 'command -v curl wget nslookup'. If it has wget but not curl, the probe becomes wget -q -O /dev/null -T 5 http://orders && echo 200.

kubectl exec fails with “unable to upgrade connection”. That is the kubelet on the Pod’s node being unreachable from the API server, not a Service fault. Nothing in this lab will work until it is fixed, and it is a control-plane problem.

orders-a has no EndpointSlice at all rather than an empty one. You are looking at a Service with no selector field. The EndpointSlice controller only manages slices for Services that have a selector; a selectorless Service expects you to create and maintain slices yourself, so the controller creates nothing. Both states present as zero backends, and the difference tells you whether the controller looked and found nothing, or was never asked to look.

The client hangs for the full five seconds instead of failing instantly. A refusal is a returned packet; a hang is a dropped one. Something is silently discarding traffic — a NetworkPolicy, a node firewall, or a CNI fault. That is a different investigation, and Lab 15 builds it deliberately.

Endpoints appear and then vanish repeatedly. The Pods are flapping between Ready and not Ready. Look at kubectl get events -n svc-lab --sort-by=.lastTimestamp for probe failures; an under-resourced readiness probe (too short a timeout, too few failures allowed) will do this under load.

The PORTS column reads <unset>. That is the placeholder slice for a Service with no matching Pods, which is fault A. There is nothing to read there; go and compare the selector against the Pod labels.

Cleanup

Everything this lab created is inside two namespaces, and deleting a namespace deletes everything in it.

Namespace deletion is recursive, immediate and not reversible

kubectl delete namespace removes every object in that namespace — Deployments, Services, Pods, ConfigMaps, Secrets, PVCs — including anything you added while working. There is no undo and no confirmation prompt.

Confirm you are pointed at the lab cluster before you run it, and that these are the namespaces you think they are:

kubectl config current-context
kubectl get all -n svc-lab
kubectl get all -n svc-lab-ext

Destructiveworkstation— Deletes both lab namespaces and everything inside them

$ kubectl delete namespace svc-lab svc-lab-ext

kubectl get namespace svc-lab svc-lab-ext 2>&1 | grep -q NotFound \
  && echo "both namespaces gone"
rm -f svc-lab.yaml svc-lab-faults.yaml validate.sh

Namespace deletion is asynchronous — the namespace sits in Terminating until every object in it is finalised, which for this lab is a few seconds because nothing here has a finaliser.

No node, kube-system object, CNI configuration or kube-proxy setting was modified at any point, so there is nothing else to restore. That property is worth noticing: a Service investigation should not need to touch anything outside the namespace that owns the Service, and if yours does, that is a signal about the fault rather than about the method.

Production notes

The diagnosis needs no change window. Every command in Tasks 2, 4, 5, 6, 7 and 8 is read-only. That is why the discipline is worth building: it means the part of an incident where you find out what is wrong carries no risk of making it worse, and there is no excuse for skipping it under pressure.

The three repairs do not carry equal risk, and should not be treated as one change type:

Repair	Mechanism	Blast radius	Rollback
Service selector	API write, no rollout	Every client of that Service, immediately	Re-patch; seconds
Service `targetPort`	API write, kube-proxy rewrite on every node	Every new connection cluster-wide	Re-patch; seconds, but in-flight clients reconnect gradually
Readiness probe	Pod template change	A full rolling replacement of the workload	`kubectl rollout undo`, one rollout

The first two are reversible in seconds and are reasonable emergency changes. The third replaces Pods, so it inherits every risk the workload’s normal deploy carries — and it should go through whatever gate the workload’s normal deploy goes through, even at 03:10.

Hold is a decision, not a delay. When the evidence cannot tell you whether the Service or the workload is wrong, holding is correct, and it needs the same paperwork as an action: what you are holding, who owns the answer, when the hold expires, and what you will do at expiry. A hold without an end time is an outage nobody is working on.

Bake Task 2 into the deploy. The five-layer capture is a five-line script. Running it in CI after every deploy of a critical Service, and keeping the output with the deploy record, converts “is this new?” from a judgement call into a diff.

Under the hood

Why can the client not tell fault A, B and C apart? Because from the kernel’s point of view they are the same event, produced by two different mechanisms that happen to look identical from the outside.

For faults A and C — a Service whose EndpointSlice has no ready: true entry — kube-proxy in its default iptables mode does not leave the ClusterIP unrouted. It installs a rule in the KUBE-SERVICES chain that rejects traffic to that ClusterIP and port with an ICMP port-unreachable. The client’s connect() fails instantly with ECONNREFUSED. This is a deliberate choice: silently dropping would make every caller wait for a TCP timeout, and an immediate refusal lets clients fail fast and retry.

For fault B the ClusterIP is routed. kube-proxy DNATs the connection to a real Pod IP on port 8080, the packet arrives in the Pod’s network namespace, and the kernel there finds nothing bound to 8080 and answers with a TCP RST. The client’s connect() fails with ECONNREFUSED.

Two different subsystems, two different packets, one identical errno. No amount of client-side instrumentation separates them, which is why the separation has to come from the control plane.

You can see the rules for yourself, though it is rarely the cheapest step. There is no kubectl exec into a node — a node is not a Pod — so use an ephemeral debug Pod in the node’s namespaces:

# Substitute a node name from: kubectl get nodes
NODE=worker-1

kubectl debug node/"$NODE" -it --image=nicolaka/netshoot -- \
  iptables-save -t nat

Two honest caveats. The debug Pod shares the node’s network namespace but has its own filesystem, so it uses its own iptables binary — and if the node runs the nft backend while the image defaults to legacy, you will see an empty or misleading ruleset rather than an error. Check with iptables -V inside the debug Pod before trusting what it prints. And if kube-proxy is in IPVS mode, iptables-save shows only the auxiliary chains; the service table is ipvsadm -Ln, and the endpointless-service rejection is still an iptables rule.

The EndpointSlice conditions are worth one more look. ready gates routing. serving exists so that a terminating Pod can keep serving connections it already has while being excluded from new routing decisions — which is how graceful shutdown works without dropping in-flight requests. A Pod that has never been ready shows ready: false, serving: false, terminating: false, and that triple is diagnostic: it is not draining, it never started.

What You Learned

The client’s error message is one bit of information, not a diagnosis. Three unrelated faults produced a byte-identical connection refused, because two different kernel mechanisms produce the same errno.
Layer 3 is where you start. kubectl get svc,endpointslices -n <namespace> distinguished all three of those faults on its own, read-only, in one round trip. Everything more expensive is for what it does not settle.
An EndpointSlice with an entry in it is not an EndpointSlice with a backend. Fault C had an address listed and ready: false, and kube-proxy routes on the condition, not the count.
A baseline taken while healthy is what makes broken output readable. The PORTS column of fault B only means something next to the healthy Service’s.
The repair choice is a risk choice. Removing a selector key costs nothing and may widen the Service; changing a probe costs a full rolling replacement. The cluster does not tell you which is correct — intent does, and when intent is unavailable, hold with an owner and an end time.
DNS failing differently is DNS working correctly. A short name resolves against the client’s own namespace, so a cross-namespace Service has no short name, and the FQDN is the fix rather than a workaround.

Why this matters

Service routing failures are the most common Kubernetes incident that is genuinely nobody’s fault: no crash, no OOM, no bad image, nothing in the logs. The Deployment is green, the Pods are green, the dashboard is green, and traffic is going nowhere.

Teams handle this badly in a predictable way. Because the client’s error looks like the workload refused the connection, the first action is nearly always to restart the workload — which cannot help, takes minutes, and replaces a well-understood system with a freshly started one. Then the readiness probe gets deleted “temporarily”, and the mechanism that would have caught the next fault is gone.

The two-command triage in Task 8 is not clever. It is just the habit of asking the control plane what it thinks before asking the data plane what it feels, and it turns the most confusing class of Kubernetes incident into a three-way lookup. The cost of building the habit is one lab. The cost of not having it is paid in the first twenty minutes of every Service incident for the rest of the cluster’s life.

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1 — Build the namespaces and the healthy baseline

Task 2 — Walk the canonical flow while nothing is wrong

Task 3 — Break it four ways

Task 4 — Fault A: the Service that selects nothing

Task 5 — Fault B: the same symptom, one layer deeper

Task 6 — Fault C: endpoints that exist and are not ready

Task 7 — Fault D: the failure that looks different

Task 8 — The triage order, and what it buys you

Validation

Expected Outcome

Troubleshooting

Cleanup

Production notes

What You Learned

Deliverables

Verification status