Lab 12: Services and EndpointSlices

Objective

By the end of this lab you will stop thinking of a Service as one thing.

Applying a single ten-line Service manifest sets three independent machines in motion, each owned by a different component, each producing an artefact you can read on its own:

1. The API server allocates a ClusterIP out of the service CIDR and writes it into spec.clusterIP. That address exists on no interface anywhere.
2. The EndpointSlice controller, inside kube-controller-manager, evaluates your selector and writes discovery.k8s.io/v1 EndpointSlice objects listing the Pods it found.
3. kube-proxy, on every node, reads those slices and programs the data plane.

Meanwhile CoreDNS, watching the same Service, starts answering for its name.

Lab 13 asks you to find which of these has broken. This lab is the one where you learn what each of them looks like when it is working — because you cannot recognise a wrong EndpointSlice if you have never deliberately read a right one.

Architecture

One Deployment, four Services over it, chosen so that each Service removes or adds exactly one thing relative to the first.

svc-basics
├── deploy/web             3 x nginx:1.27.2, containerPort named http (80)
│                          preStop sleep 30, grace period 60s
├── svc/web                ClusterIP,  port 8080 -> targetPort http
├── svc/web-headless       clusterIP: None, same selector
├── svc/web-np             NodePort,   port 8080 -> targetPort http
├── svc/legacy-db          no selector at all
│      endpointslice/legacy-db-manual   written by hand
└── pod/client             nicolaka/netshoot, sleep infinity

flowchart TD
    S[Service object] --> A[API server: allocate spec.clusterIP]
    S --> B[EndpointSlice controller: evaluate selector]
    S --> C[CoreDNS: serve the name]
    B --> D[EndpointSlice objects]
    D --> E[kube-proxy on every node: program the data plane]
    A --> E

The arrows are the point. Nothing in that diagram is a single step, and the four Services below each break one arrow deliberately so you can see it exists.

Requirements

• A kubeadm cluster on Kubernetes 1.34.x, built as in Lab 01, with kubectl 1.34.x and cluster-admin on it. A single-node cluster works if the control-plane taint has been removed; more nodes make Task 6 more interesting but change nothing else.
• A working CNI. Every Pod must get an IP and reach every other Pod. This lab reads Pod IPs constantly and none of it means anything if pod-to-pod is already broken.
• A functioning cluster DNS. Task 5 reads DNS answers. If kube-dns in kube-system has no ready endpoints, do Lab 14 first.
• Ability to pull nginx:1.27.2 and nicolaka/netshoot. The lab names no other images.
• Blast radius: one namespace, svc-basics. Cleanup deletes it. Nothing outside the namespace is created or modified, and no node, CNI or kube-proxy setting is touched. Task 6 opens one port in the NodePort range on every node for the duration of the lab.

Scenario

You are reviewing a manifest a team wants merged. It contains a Service with port: 80, targetPort: 80, nodePort: 30080 and clusterIP: None, and the author cannot explain what any of the four fields do, only that “this is what the old one had”. The reviewer before you approved it.

Nothing here is broken yet, which is exactly why it is worth an hour. The Service API’s fields look interchangeable and are not: two of them are ports on different machines, one of them is a port on every node in the cluster, and one of them silently turns off the load balancer entirely. The cheapest place to learn the difference is a cluster where nothing depends on you being right.

Tasks

Task 1 — One Deployment, one Service, three artefacts

kubectl create namespace svc-basics
kubectl label namespace svc-basics lab=svc-basics

cat > svc-basics.yaml <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: svc-basics
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: nginx
          image: nginx:1.27.2
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 5
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 30"]
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: svc-basics
spec:
  selector:
    app: web
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
---
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: svc-basics
spec:
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      command: ["sleep", "infinity"]
YAML

kubectl apply -f svc-basics.yaml
kubectl -n svc-basics rollout status deploy/web --timeout=120s
kubectl -n svc-basics wait --for=condition=Ready pod/client --timeout=120s

The port numbers are deliberately different so you can tell them apart later: clients talk to 8080, the container listens on 80, and targetPort: http resolves by name rather than by number.

The preStop hook is what makes Task 4 observable. It is also the correct production pattern, for a reason worth stating now: when a Pod is deleted, removing it from the EndpointSlice and telling the kubelet to stop it happen concurrently, not in sequence. A preStop sleep gives the endpoint removal time to propagate to every node’s kube-proxy before the application starts shutting down, which is how you avoid dropping in-flight requests.

Now collect all three artefacts from that one Service:

kubectl -n svc-basics get svc web
kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=web
kubectl -n svc-basics exec client -- nslookup web.svc-basics.svc.cluster.local

Three commands, three different components answering. Keep the outputs; the deliverable table is built from them.

Task 2 — The ClusterIP is an address that exists nowhere

kubectl -n svc-basics get svc web -o jsonpath='{.spec.clusterIP}{"\n"}'

That address came from the API server’s service CIDR, and the API server is the only allocator — which is why you cannot create two Services with the same ClusterIP, and why a clusterIP you pick yourself is either accepted or refused with a conflict rather than silently duplicated.

Now prove it is virtual. From inside the client Pod:

kubectl -n svc-basics exec client -- ip -brief addr show
kubectl -n svc-basics exec client -- ip route get 10.96.0.1

The ClusterIP appears on no interface, in the client Pod or anywhere else. There is no host, no NIC and no ARP entry behind it. What makes it reachable is that kube-proxy on the client’s own node has programmed a rule that rewrites the destination before the packet leaves the node.

Confirm the address works from a Pod:

kubectl -n svc-basics exec client -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://web.svc-basics.svc.cluster.local:8080

Note the port. 8080 is the Service’s port — a number that exists only in the Service object and in kube-proxy’s rules. Nothing in the cluster is listening on 8080; nginx is listening on 80. Sending traffic to :80 here fails, which is a useful thing to try once on purpose.

Task 3 — The EndpointSlice is the controller’s answer to your selector

kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=web

$ kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=web

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                             AGE
web-fk29t   IPv4          80      10.244.1.7,10.244.2.9,10.244.1.8      3m

Four things in that one line are worth reading deliberately:

• The name is generated, not chosen. Never reference an EndpointSlice by name in a script; find it by the kubernetes.io/service-name label, which is exactly how kube-proxy finds it too.
• PORTS shows 80, not 8080. The slice records the target port — the one on the Pod. targetPort: http was resolved against the container’s named port by the controller, and the resolved number is what appears here. When port and targetPort differ, this column is the fastest way to confirm which one the cluster actually resolved.
• ENDPOINTS lists Pod IPs, not node IPs, not the ClusterIP.
• ADDRESSTYPE is IPv4. A dual-stack Service gets separate slices per address family, which is why the field exists at all.

Now read the structure underneath the columns:

kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=web \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"  ready="}{.conditions.ready}{"  serving="}{.conditions.serving}{"  terminating="}{.conditions.terminating}{"  node="}{.nodeName}{"\n"}{end}'

Three conditions per endpoint, and they are not redundant. ready is what kube-proxy routes on. serving and terminating exist for the shutdown case, which Task 4 is about. nodeName is what makes topology-aware routing possible.

Watch the controller work:

kubectl -n svc-basics scale deploy/web --replicas=5
kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=web -w

Press Ctrl-C once it settles at five, then scale back:

kubectl -n svc-basics scale deploy/web --replicas=3

Notice what did not happen: no new Service object, no new ClusterIP, no DNS change. The Service is the stable part; the slice is the moving part. That separation is the entire reason Services exist.

Task 4 — Catch an endpoint mid-shutdown

This is the observation that explains why the three conditions exist. In one terminal, start a watch on the raw conditions:

while true; do
  date +%T
  kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=web \
    -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{" ready="}{.conditions.ready}{" serving="}{.conditions.serving}{" terminating="}{.conditions.terminating}{"\n"}{end}'
  echo "---"
  sleep 3
done

In a second terminal, delete one Pod:

POD=$(kubectl -n svc-basics get pod -l app=web -o jsonpath='{.items[0].metadata.name}')

kubectl -n svc-basics delete pod "$POD"

Because of the 30-second preStop sleep you get roughly half a minute to watch the doomed endpoint hold this state:

10.244.1.7 ready=false serving=true terminating=true

Read that triple carefully, because it is the mechanism behind every graceful deployment you have ever run:

• ready=false — kube-proxy stops sending it new connections. Immediately.
• serving=true — the container is still accepting and completing work. Its existing connections are not being cut.
• terminating=true — this is a planned shutdown, not a health failure.

A Pod that has never been ready shows ready=false serving=false terminating=false, and that difference is diagnostic: one is draining, the other never started. Capture both triples for the deliverable — the second one you can produce by scaling up and catching a new Pod before its first probe succeeds.

Task 5 — Headless removes one thing, and only one

cat > svc-headless.yaml <<'YAML'
apiVersion: v1
kind: Service
metadata:
  name: web-headless
  namespace: svc-basics
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
YAML

kubectl apply -f svc-headless.yaml
kubectl -n svc-basics get svc

web-headless shows None where the ClusterIP would be. Now check what it did not remove:

kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=web-headless

The EndpointSlice is still there, with the same Pod IPs. The selector still ran, the controller still reconciled, the backends are still tracked. Headless removes the virtual IP and the kube-proxy hop — nothing else.

The difference lands in DNS:

kubectl -n svc-basics exec client -- dig +short web.svc-basics.svc.cluster.local
kubectl -n svc-basics exec client -- dig +short web-headless.svc-basics.svc.cluster.local

The first returns one address: the ClusterIP. The second returns three: the Pod IPs. And that is the whole trade-off in two commands.

With a ClusterIP, the client gets one stable address and kube-proxy picks a backend per connection. With headless, the client gets the full list and must choose — which is exactly what you want for a database driver that maintains a connection pool per replica, and exactly what you do not want for an HTTP client that resolves once at startup and caches the first answer forever.

Task 6 — NodePort adds a port; it replaces nothing

cat > svc-nodeport.yaml <<'YAML'
apiVersion: v1
kind: Service
metadata:
  name: web-np
  namespace: svc-basics
spec:
  type: NodePort
  selector:
    app: web
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
YAML

kubectl apply -f svc-nodeport.yaml
kubectl -n svc-basics get svc web-np

Read the PORT(S) column. It shows something like 8080:31274/TCP — two numbers, because a NodePort Service is a ClusterIP Service plus a node port:

kubectl -n svc-basics get svc web-np \
  -o jsonpath='clusterIP={.spec.clusterIP}{"\n"}nodePort={.spec.ports[0].nodePort}{"\n"}'

The ClusterIP is still allocated, still virtual, still the thing in-cluster clients should use. The node port is an additional entrance, allocated from the range 30000-32767 by default. You did not request a specific one, so the API server picked a free port and recorded it — which is why hard-coding nodePort in a manifest is a decision to manage a cluster-wide port registry by hand.

Reach it from outside the cluster:

# Substitute a node IP from: kubectl get nodes -o wide
NODE_IP=192.0.2.11
NODE_PORT=$(kubectl -n svc-basics get svc web-np -o jsonpath='{.spec.ports[0].nodePort}')

curl -s -o /dev/null -w '%{http_code}\n' "http://${NODE_IP}:${NODE_PORT}"

Now try a node that is not running any web Pod. It still answers. The node port is open on every node in the cluster regardless of where the Pods are, and kube-proxy on the receiving node forwards to a backend wherever it lives.

That convenience costs something worth knowing: the extra hop rewrites the source address, so the application sees the node’s address rather than the real client’s. externalTrafficPolicy: Local avoids the hop and preserves the client address, at the price of a node with no local backend refusing the connection instead of forwarding it. Neither is a default you should inherit without deciding.

Task 7 — A Service with no selector, and a slice you write yourself

This is the task that proves the model. Create a Service with no selector at all, then supply its backends by hand.

cat > svc-selectorless.yaml <<'YAML'
apiVersion: v1
kind: Service
metadata:
  name: legacy-db
  namespace: svc-basics
spec:
  ports:
    - name: http
      port: 8080
      targetPort: 80
      protocol: TCP
YAML

kubectl apply -f svc-selectorless.yaml
kubectl -n svc-basics get svc legacy-db
kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=legacy-db

The Service exists and has a ClusterIP. The EndpointSlice query returns nothing — not an empty slice, no slice at all. The controller only manages slices for Services that have a selector; with no selector it was never asked to look, and it creates nothing.

That distinction matters in an incident. An empty slice means the controller looked and found no matching Pods. No slice at all means the controller was never involved, and the backends are somebody’s manual responsibility.

Now be that somebody. Point the Service at a real Pod IP so you can prove traffic flows:

POD_IP=$(kubectl -n svc-basics get pod -l app=web \
  -o jsonpath='{.items[0].status.podIP}')

cat > slice-manual.yaml <<YAML
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: legacy-db-manual
  namespace: svc-basics
  labels:
    kubernetes.io/service-name: legacy-db
addressType: IPv4
ports:
  - name: http
    port: 80
    protocol: TCP
endpoints:
  - addresses:
      - ${POD_IP}
    conditions:
      ready: true
YAML

kubectl apply -f slice-manual.yaml

kubectl -n svc-basics exec client -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://legacy-db.svc-basics.svc.cluster.local:8080

200. A Service whose backend list you wrote by hand routes exactly like one the controller wrote. Three things follow:

• The kubernetes.io/service-name label is the only link between the two objects. Not an owner reference, not the name. Get the label wrong and the slice is inert with no error anywhere.
• This is the production pattern for a real external dependency — a managed database, a load balancer outside the cluster — where you want in-cluster clients to use a normal Service name and DNS entry rather than an IP scattered through every manifest.
• Nothing maintains it. The address you wrote is the address that stays, through the backend moving, failing or being replaced. A selectorless Service is a promise to keep the slice current by some other means.

Delete the manual slice and watch the difference between managed and unmanaged:

kubectl -n svc-basics delete endpointslice legacy-db-manual
kubectl -n svc-basics get endpointslices -l kubernetes.io/service-name=legacy-db

Nothing recreates it. Delete a web slice by comparison and it is back within a second, because that one has a controller behind it.

Validation

Save as validate.sh and run it. It exits non-zero if any layer is wrong, which is the point: a validation you can pass by squinting at output is not a validation.

#!/usr/bin/env bash
set -euo pipefail
NS=svc-basics

cip=$(kubectl -n "$NS" get svc web -o jsonpath='{.spec.clusterIP}')
case "$cip" in
  ''|None) echo "FAIL web has no ClusterIP"; exit 1 ;;
esac
echo "ok   web ClusterIP is $cip"

slice_port=$(kubectl -n "$NS" get endpointslices \
  -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')
if [ "$slice_port" != "80" ]; then
  echo "FAIL web slice port is $slice_port, expected the resolved targetPort 80"
  exit 1
fi
echo "ok   web slice records target port 80, not service port 8080"

ready=$(kubectl -n "$NS" get endpointslices -l kubernetes.io/service-name=web \
  -o jsonpath='{range .items[*].endpoints[*]}{.conditions.ready}{"\n"}{end}' \
  | grep -c true || true)
if [ "$ready" -ne 3 ]; then
  echo "FAIL web has $ready ready endpoints, expected 3"
  exit 1
fi
echo "ok   web has 3 ready endpoints"

hl=$(kubectl -n "$NS" get svc web-headless -o jsonpath='{.spec.clusterIP}')
if [ "$hl" != "None" ]; then
  echo "FAIL web-headless has a ClusterIP: $hl"
  exit 1
fi
echo "ok   web-headless is headless"

answers=$(kubectl -n "$NS" exec client -- \
  dig +short web-headless."$NS".svc.cluster.local | grep -c . || true)
if [ "$answers" -lt 3 ]; then
  echo "FAIL headless DNS returned $answers addresses, expected 3"
  exit 1
fi
echo "ok   headless DNS returns $answers Pod addresses"

np=$(kubectl -n "$NS" get svc web-np -o jsonpath='{.spec.ports[0].nodePort}')
if [ "$np" -lt 30000 ] || [ "$np" -gt 32767 ]; then
  echo "FAIL nodePort $np is outside the default range"
  exit 1
fi
echo "ok   web-np allocated nodePort $np"

for s in web web-headless web-np legacy-db; do
  code=$(kubectl -n "$NS" exec client -- \
    curl -s -o /dev/null -m 5 -w '%{http_code}' "http://$s.$NS.svc.cluster.local:8080")
  if [ "$code" != "200" ]; then
    echo "FAIL $s returned $code"
    exit 1
  fi
  echo "ok   $s returned 200"
done

Six separate claims, and they are deliberately not the same claim. The ClusterIP was allocated (the API server acted). The slice port is 80 (the controller resolved a named targetPort correctly). Three endpoints are ready (the selector matched and the probes passed). Headless has no ClusterIP but does have three DNS answers (removing the VIP did not remove the backends). The node port is inside the default range. And all four Services return 200 from a Pod, including the one whose backends you wrote by hand.

Expected Outcome

validate.sh exits 0, and you have both endpoint-condition triples captured: one draining Pod and one that has never been ready.

Troubleshooting

ImagePullBackOff on the client Pod. nicolaka/netshoot comes from Docker Hub and anonymous pulls are rate limited. Any image with a shell, curl and a resolver works instead; this course also uses registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7. Before depending on a substitute, ask it what it has: kubectl -n svc-basics exec client -- sh -c 'command -v curl wget dig nslookup'.

dig is not found in the client Pod. Use nslookup or getent hosts NAME. getent returns addresses without any DNS-specific formatting, which is often easier to script against.

The headless lookup returns one address, not three. Check that the Pods are Ready. Only ready endpoints get A records by default, so a headless Service in front of a half-started Deployment answers with a short list rather than an error.

curl to the Service on port 80 fails but 8080 works. Correct. port: 8080 is what the Service listens on; targetPort is what the Pod listens on. The Service does not answer on its targetPort.

The EndpointSlice PORTS column shows the Service port, not 80. Then targetPort was omitted, and it defaults to the same value as port. That default is the single most common Service misconfiguration, and Lab 13 builds it deliberately as a fault.

kubectl get endpointslices returns nothing for a Service that has a selector. Two possibilities that look identical: the selector matches no Pods, or you are querying the wrong label. Confirm with kubectl -n svc-basics get endpointslices --show-labels and compare against kubectl -n svc-basics get pods --show-labels.

The manual slice exists but legacy-db still returns nothing. Check the label spelling: kubernetes.io/service-name must match the Service name exactly. There is no validation for this and no event when it is wrong — the slice simply belongs to a Service that does not exist.

The NodePort answers from one node and not another. A host firewall on the non-answering node is the usual cause. kube-proxy programs every node, but it does not open host firewalls.

Cleanup

Everything this lab created lives in one namespace.

$ kubectl delete namespace svc-basics

kubectl get namespace svc-basics 2>&1 | grep -q NotFound && echo "namespace gone"
rm -f svc-basics.yaml svc-headless.yaml svc-nodeport.yaml \
      svc-selectorless.yaml slice-manual.yaml validate.sh

Deleting the namespace releases the ClusterIPs back to the allocator and the node port back to its range, so nothing needs freeing by hand. Nothing outside the namespace was created or modified: no node, no CNI setting, no kube-proxy configuration, and no object in kube-system.

Production notes

Name your ports, always. targetPort: http survives a container that moves from 80 to 8080; targetPort: 80 does not. With more than one port on a Service the name field stops being optional, and named ports are what make a Service readable in review six months later. This lab used a named targetPort specifically so you would see the resolved number appear in the slice.

Do not hard-code nodePort, and do not hard-code clusterIP. Both turn a cluster-wide allocator into a spreadsheet somebody has to maintain, and both fail at apply time with a conflict when two teams pick the same number. Let the API server allocate; read the value back when you need it.

A selectorless Service is infrastructure, not a manifest. Fronting an external database this way is a good pattern and gives in-cluster clients a normal DNS name. It is only good if something keeps the slice current and alerts when it cannot. A hand-written slice with a stale address routes traffic confidently into a hole.

preStop is not optional for anything serving traffic. Endpoint removal and container shutdown race, and the race is lost silently — as dropped requests during an otherwise successful rollout. A preStop sleep covering slice propagation across your nodes, with a terminationGracePeriodSeconds comfortably larger than it, is the fix. Measure it once on your own cluster rather than copying a number.

Changing a Service’s ports is a live change with no rollout. Patching port or targetPort takes effect as soon as kube-proxy on each node picks it up, with no Deployment restart and no gradual rollout to hide behind. It is reversible in seconds, which makes it a reasonable emergency change — but every existing client is affected at once, and there is no canary.

Scaling and rollouts churn the slice, not the Service. Anything watching your cluster’s API — a service mesh, a policy controller, an ingress controller — sees EndpointSlice writes on every readiness transition. On a large, busy Deployment that is a real load on the control plane, and it is why EndpointSlices replaced the single Endpoints object in the first place.

What You Learned

• One Service object drives three independent components. The API server allocates the ClusterIP, the EndpointSlice controller evaluates the selector, kube-proxy programs the data plane — and CoreDNS watches alongside them. Each leaves a separate artefact you can read on its own.
• port and targetPort are ports on different machines. The slice records the resolved targetPort, which makes the PORTS column the fastest confirmation of what the cluster actually resolved a named port to.
• The ClusterIP exists on no interface. It is a rewrite rule, not an address, which is why only traffic passing through kube-proxy can reach it.
• ready, serving and terminating are three different facts. A draining Pod is ready=false serving=true terminating=true; a Pod that never started is false/false/false. Graceful shutdown is built entirely on that difference.
• Headless removes the VIP and the kube-proxy hop, and nothing else. The selector still runs, the slice is still maintained, and load balancing becomes the client’s job whether the client knows it or not.
• NodePort adds an entrance; it does not replace the ClusterIP. Every node opens the port regardless of where the Pods are, and the extra hop rewrites the source address unless you change externalTrafficPolicy.
• A Service and its EndpointSlice are separate objects joined by one label. You can have a Service with no slice, and a slice you wrote by hand that routes perfectly. Nothing validates the label, and nothing health-checks what you wrote.