EndpointSlice controller — Service-to-Pod traffic surface

The EndpointSlice controller translates Service selectors into EndpointSlices — sets of Pods that the Service should route to. This is the controller that powers every Service in the cluster. Without it, kube-proxy and CoreDNS have nothing to react to. This lesson walks the controller, the data structures, and the production patterns.

The controller in one sentence

The EndpointSlice controller watches Services and Pods; for each Service with a selector, it computes the matching Pods and writes EndpointSlices to the API server. The EndpointSlices are the authoritative source of “which Pods are part of this Service”.

flowchart LR
    S[Service] -->|selector| ES[EndpointSlice]
    ES -->|endpoints list| AS[API server]
    AS -->|watch| KP[kube-proxy]
    AS -->|watch| CD[CoreDNS]
    AS -->|watch| CIDR[clients via Service IP]

The Service itself does not contain endpoints; it delegates to EndpointSlices (the API object).

The legacy Endpoints and the modern EndpointSlices

Kubernetes 1.21+ treats the legacy Endpoints API as deprecated. The controller writes both Endpoints and EndpointSlices to preserve backward compatibility.

# The legacy API
kubectl get endpoints web -n prod

# The modern API
kubectl get endpointslices -l kubernetes.io/service-name=web -n prod

In production, kube-proxy and CoreDNS watch EndpointSlices (preferred) and ignore the legacy Endpoints.

The EndpointSlice data structure

An EndpointSlice contains:

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: web-abc123
  namespace: prod
  labels:
    kubernetes.io/service-name: web
addressType: IPv4
ports:
- name: http
  port: 80
  protocol: TCP
endpoints:
- addresses:
  - 10.0.1.20
  conditions:
    ready: true
  topology:
    kubernetes.io/hostname: cp-1
- addresses:
  - 10.0.1.21
  conditions:
    ready: true
  topology:
    kubernetes.io/hostname: cp-2

Each EndpointSlice represents up to 100 endpoints (a configurable limit). A Service with 500 Pods has 5 EndpointSlices.

The discovery process

sequenceDiagram
    autonumber
    participant C as EndpointSlice controller
    participant API
    C->>API: watch Services and Pods
    API-->>C: Service "web" created (selector: app=web)
    C->>API: list Pods matching selector
    API-->>C: Pods with app=web
    C->>C: build EndpointSlices
    C->>API: write EndpointSlice(s)
    API-->>C: 201 Created
    C->>C: continue watching (Pod changes update slices)
    API-->>C: Pod web-0 deleted
    C->>C: rebuild EndpointSlice
    C->>API: update EndpointSlice

The controller’s logic:

• Watch Services for selector changes.
• For each Service with a selector, watch Pods matching the selector.
• Compute the EndpointSlice (set of addresses, ports, conditions, topology).
• Write the EndpointSlice through the API.

The selector and readiness

The controller respects Pod readiness. A Pod that is not Ready is excluded from the EndpointSlice:

endpoints:
- addresses:
  - 10.0.1.20
  conditions:
    ready: false      # not Ready (Pending, Terminating, etc.)

kube-proxy and CoreDNS only route to ready: true endpoints. A Pod that is NotReady is removed from the service.

The reconciliation cycle

The controller’s reconcile loop:

1. List all Services with selectors.
2. For each Service, list matching Pods.
3. Compute current EndpointSlices vs observed.
4. Create / update / delete EndpointSlices to match.

The cycle runs:

• On watch events (immediate).
• Periodically (backstop).
• After API server reconnect (after a watch recovery).

The failure modes

Controller falls behind

workqueue_depth{name="endpoint_slice"} = 50 (high)

If the controller cannot keep up with watch events, the queue grows. Pods that should be removed from Service traffic may remain for a brief period (until reconciled).

# Force a reconcile
kubectl annotate service web -n prod \
  kubernetes.io/force-update-endpoint-slice="true"

The annotation forces the controller to re-list the Service.

EndpointSlices not updating

kubectl get endpointslices ... -o yaml shows stale endpoints

The controller is stalled; check the controller-manager logs for errors. A common cause is RBAC denied (the controller’s ServiceAccount lacks permissions).

The Slicing for large services

A Service with many Pods (say 1,000) has many EndpointSlices. The controller’s slicing:

Service "metrics"
  1,000 Pods matching
  10 EndpointSlices, each with 100 endpoints

A change to a single Pod updates a single EndpointSlice. The cost is bounded regardless of Service size.

The slice size defaults to 100 endpoints; tune via the controller flag:

--max-endpoints-per-slice=100

Larger slices reduce API objects but increase per-slice write size.

$ kubectl get endpointslices -l kubernetes.io/service-name=web -n prod -o json | jq '.items | length'

3

The metrics

- controller_reconcile_total{controller="endpointslice"}
- controller_reconcile_duration_seconds{controller="endpointslice"}
- workqueue_depth{controller="endpointslice"}

A growing work queue depth is the diagnostic.

The CoreDNS dependency

CoreDNS serves the Service DNS records. CoreDNS reads EndpointSlices via the API server’s watch and serves web.prod.svc.cluster.local with the EndpointSlice IPs.

A misbehaving EndpointSlice controller means:

• Pods are not in the Service.
• DNS records have stale IPs.
• kube-proxy has stale rules.
• The cluster is broken until reconciled.

Production discipline

The EndpointSlice controller is the cluster’s Service-to- Pod bridge. Operating it well is keeping Service traffic correct.

• Monitor the work-queue depth. A growing queue indicates the controller is falling behind.
• Verify Service Endpoints periodically. kubectl get endpointslices ... -o yaml confirms the selectors are matched.
• Use the --max-endpoints-per-slice setting. Default 100 is reasonable; tune for the cluster.
• Restart the controller if it stalls. Move the static pod aside to trigger a kubelet restart.
• Document the controller’s role. A runbook entry on Service troubleshooting should mention the EndpointSlice controller.
• Document selector changes. A change to a Service selector causes a rebuild of EndpointSlices.
• Stay on the modern API. Legacy Endpoints are deprecated; EndpointSlices are the path forward.

Quiz