Services and ClusterIP — the stable virtual IP for a set of Pods

A Service is a stable virtual IP that fronts a set of Pods. The IP is allocated by the API server; the Endpoints or EndpointSlices are reconciled by the controller-manager; the kube-proxy on every node translates the ClusterIP to a Pod IP. The Service is the basis of service discovery: clients send traffic to the ClusterIP, and the Service route to the Pods. This lesson walks the Service, the ClusterIP, the endpoints, and the operational discipline.

Why a Service

Pods are ephemeral. A Pod’s IP can change with every restart. A Deployment that recreates Pods keeps the Pod spec consistent, but the Pod IPs are not stable. Clients cannot rely on Pod IPs.

A Service provides a stable virtual IP. The IP does not change as Pods come and go. The clients send traffic to the Service; the Service routes to the current set of Pods that match the selector.

flowchart LR
    A[Client] -->|ClusterIP: 10.96.0.10| B[Service]
    B -->|EndpointSlice| C[Pod 1: 10.244.1.5]
    B -->|EndpointSlice| D[Pod 2: 10.244.1.6]
    B -->|EndpointSlice| E[Pod 3: 10.244.2.5]

The Service is the abstraction; the EndpointSlice is the implementation; the kube-proxy is the data plane.

The Service manifest

A Service is a Kubernetes object:

apiVersion: v1
kind: Service
metadata:
  name: billing
  namespace: prod-app
spec:
  type: ClusterIP
  selector:
    app: billing
  ports:
    - name: http
      port: 80
      targetPort: 8080
      protocol: TCP

The fields:

• type: ClusterIP, NodePort, LoadBalancer, or ExternalName. The default is ClusterIP.
• selector: the labels that select the Pods. The Service routes to the Pods with matching labels.
• ports: the port mapping. The Service’s port is the client-facing port; the targetPort is the Pod’s port.

The ClusterIP

The ClusterIP is allocated by the API server when the Service is created. The IP comes from the --service-cluster-ip-range flag of the kube-apiserver. The default is 10.96.0.0/16.

kubectl get svc billing -o jsonpath='{.spec.clusterIP}'

10.96.0.10

The ClusterIP is a virtual IP: it does not exist on any interface. The kube-proxy on every node intercepts traffic to the ClusterIP and DNATs to a Pod IP.

The Endpoints and EndpointSlices

The Service’s Endpoints are the set of Pod IPs that match the selector. The controller-manager reconciles the Endpoints on every Service update, Pod creation, and Pod deletion.

The Kubernetes 1.33+ default is EndpointSlices:

kubectl get endpointslices -l kubernetes.io/service-name=billing

NAME             ADDRESSTYPE   PORTS   ENDPOINTS
billing-abc123   IPv4          8080    10.244.1.5,10.244.1.6,10.244.2.5

The EndpointSlice is the data plane’s view of the Service. The kube-proxy reads the EndpointSlice and programs the iptables or IPVS rules.

The kube-proxy translation

The kube-proxy on every node translates the ClusterIP to a Pod IP. The translation is a DNAT (destination network address translation):

Client -> 10.96.0.10:80 -> kube-proxy -> 10.244.1.5:8080

The kube-proxy programs the iptables or IPVS rules based on the EndpointSlices. The rules are updated when the EndpointSlices change.

The cluster operator can inspect the rules:

# iptables
iptables-save | grep 10.96.0.10

# ipvs
ipvsadm -Ln | grep 10.96.0.10

The output shows the DNAT rules that the kube-proxy has programmed.

The failure modes

The Service’s failure modes:

• Selector mismatch: the Service’s selector does not match any Pods. The EndpointSlice is empty; the Service has no backends. The fix is to verify the selector’s labels.
• Empty EndpointSlice: the Pods are not Ready; the EndpointSlice is empty. The fix is to check the Pod’s readiness probe.
• Port mismatch: the Service’s port does not match the Pod’s port. The fix is to verify the port mapping.
• kube-proxy not running: the iptables rules are not programmed; the ClusterIP is unreachable. The fix is to restart the kube-proxy.
• ClusterIP exhaustion: the API server’s —service-cluster-ip-range is exhausted. The fix is to add a larger range.

The operational discipline

The Service’s operational discipline:

• Document the Service’s selector. The operator must understand which Pods the Service routes to.
• Audit the EndpointSlice. The EndpointSlice is the Service’s backends; an empty EndpointSlice is the most common failure.
• Test the Service in staging. A misconfigured selector or port is the most common Service failure.
• Monitor the Service’s metrics. The Service’s metrics expose the backends and the traffic.
• Document the Service’s design. The Service is the cluster’s networking primitive; the documentation is the reference.

Quiz

Production discipline

• The Service is a stable virtual IP. The IP does not change as Pods come and go.
• The ClusterIP is virtual. The kube-proxy translates it to a Pod IP via iptables or IPVS.
• The EndpointSlice is the data plane’s view. The kube-proxy reads it and programs the rules.
• The selector is the Service’s contract. A selector mismatch is the most common Service failure.
• Document the Service’s selector. The operator must understand which Pods the Service routes to.
• Audit the EndpointSlice. The EndpointSlice is the Service’s backends.
• Test the Service in staging. A misconfigured selector or port is the most common Service failure.
• Monitor the Service’s metrics. The Service’s metrics expose the backends and the traffic.
• Document the Service’s design. The Service is the cluster’s networking primitive; the documentation is the reference.