Pod-to-Service — the Service abstraction and the kube-proxy dataplane

The Service abstraction is the cluster’s load balancer. The Service has a virtual IP that is distributed across the cluster’s Pods; the kube-proxy implements the Service dataplane. This lesson walks the Service abstraction, the kube-proxy’s three implementations, the EndpointSlice, and the operational patterns.

The Service abstraction

The Service is a Kubernetes object (apiVersion: v1, kind: Service) that exposes a set of Pods as a single virtual IP. The Service’s virtual IP is distributed across the cluster’s Pods by the kube-proxy.

apiVersion: v1
kind: Service
metadata:
  name: billing
spec:
  selector:
    app: billing
  ports:
    - port: 80
      targetPort: 8080

The Service’s virtual IP is the cluster’s IP for the Service. The Service’s Pods are the backends.

The Service’s lifecycle:

sequenceDiagram
    autonumber
    participant API as API server
    participant K as kube-proxy
    participant P as Pod

    API->>API: Service created
    API->>API: allocate cluster IP
    API->>K: watch Service
    K->>K: program dataplane
    API->>API: EndpointSlice created
    API->>K: watch EndpointSlice
    K->>K: update dataplane
    P->>K: connect to cluster IP
    K->>P: forward to backend Pod

The Service is the cluster’s load balancer. The kube-proxy implements the load balancing.

The kube-proxy’s implementations

The kube-proxy has three implementations:

• iptables: the default. The kube-proxy configures iptables rules to implement the Service. The rules distribute the traffic to the backends.
• IPVS: the alternative. The kube-proxy configures IPVS rules to implement the Service. The IPVS is faster than iptables for large clusters.
• eBPF: the modern. The kube-proxy configures eBPF programs to implement the Service. The eBPF is the fastest.

The three implementations are the cluster’s Service dataplane. The cluster operator chooses the implementation.

The iptables implementation

The iptables implementation:

# Substitute your own value before running:
SVC_IP=10.96.0.1

iptables -t nat -L -n -v | grep "$SVC_IP"

The output shows the iptables rules the kube-proxy has installed. The rules are:

-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp --dport 80 -j KUBE-SVC-XXX
-A KUBE-SVC-XXX -m statistic --mode random --probability 0.333 -j KUBE-SEP-YYY
-A KUBE-SEP-YYY -p tcp -j DNAT --to-destination 10.244.1.5:8080

The rules distribute the traffic to the backends. The statistic module performs the random load balancing.

The iptables implementation is the cluster’s default Service dataplane. The kube-proxy configures the iptables rules on every node.

The IPVS implementation

The IPVS implementation:

# Substitute your own values before running:
SVC_IP=10.96.0.1
SVC_PORT=80

ipvsadm -L -n -t "$SVC_IP:$SVC_PORT"

The output shows the IPVS rules the kube-proxy has installed. The IPVS implements the load balancing using the kernel’s IP Virtual Server.

The IPVS implementation is faster than iptables for large clusters. The IPVS has a smaller rule count; the IPVS uses the kernel’s IPVS hash table.

The eBPF implementation

The eBPF implementation:

bpftool map show

The output shows the eBPF maps the kube-proxy has installed. The eBPF programs implement the load balancing using the kernel’s eBPF.

The eBPF implementation is the fastest. The eBPF programs run in the kernel; the eBPF has a smaller overhead.

The EndpointSlice

The EndpointSlice is the cluster’s view of the Service’s backends:

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: billing-abc
  labels:
    kubernetes.io/service-name: billing
addressType: IPv4
endpoints:
  - addresses:
      - 10.244.1.5
    conditions:
      ready: true
    nodeName: node-1
ports:
  - port: 8080

The EndpointSlice is the cluster’s view of the Service’s Pods. The kube-proxy watches the EndpointSlice and updates the dataplane.

The EndpointSlice is the cluster’s mechanism for scaling the Service. The EndpointSlice has a limit of 100 endpoints per slice; the cluster creates multiple EndpointSlices for large services.

The Service’s DNS

The Service’s DNS is the cluster’s mechanism for discovering the Service’s IP:

nslookup billing.prod-app.svc.cluster.local

The output:

Name:   billing.prod-app.svc.cluster.local
Address: 10.96.0.1

The Service’s DNS is configured by the cluster’s DNS plugin (CoreDNS). The DNS plugin watches the Service and updates the DNS records.

The Service’s DNS is the cluster’s mechanism for discovering the Service. The Pod’s application uses the DNS to find the Service’s IP.

The Service’s virtual IP

The Service’s virtual IP is allocated by the cluster’s IPAM. The IP is from the cluster’s Service CIDR (default 10.96.0.0/12). The IP is unique within the cluster.

The Service’s virtual IP is the cluster’s stable IP for the Service. The IP does not change when the Pods are recreated.

The Service’s virtual IP is the cluster’s load balancer. The kube-proxy distributes the traffic to the backends.

The kube-proxy’s troubleshooting

The kube-proxy’s troubleshooting:

# Substitute the kube-proxy Pod on the node you are debugging:
KUBE_PROXY_POD=kube-proxy-7h2np

kubectl logs -n kube-system "$KUBE_PROXY_POD"

The kube-proxy’s logs show the dataplane’s status. The fix is to investigate the kube-proxy’s logs.

The kube-proxy’s troubleshooting:

# Substitute your own value before running:
SVC_IP=10.96.0.1

iptables -t nat -L -n -v | grep "$SVC_IP"

The iptables rules show the Service’s dataplane. The fix is to investigate the rules.

The kube-proxy’s troubleshooting:

# Substitute your own values before running:
SVC_IP=10.96.0.1
SVC_PORT=80

ipvsadm -L -n -t "$SVC_IP:$SVC_PORT"

The IPVS rules show the Service’s dataplane. The fix is to investigate the rules.

The Service’s operational patterns

The Service’s operational patterns:

• Use the DNS for Service discovery. The Pod’s application should use the DNS to find the Service.
• Use the right kube-proxy implementation. The cluster operator should choose the implementation that matches the cluster’s traffic.
• Monitor the kube-proxy’s metrics. The kube-proxy’s metrics expose the Service’s dataplane health.
• Audit the kube-proxy’s configuration. The kube-proxy’s configuration should be version- controlled; the audit catches the failures.

Quiz

Production discipline

• The Service is the cluster’s load balancer. The kube-proxy implements the Service dataplane.
• Choose the right kube-proxy implementation. The iptables is the default; the IPVS is for large clusters; the eBPF is the fastest.
• Use the EndpointSlice for scaling. The EndpointSlice has a limit of 100 endpoints per slice; the cluster creates multiple slices.
• Use the DNS for Service discovery. The Pod’s application should use the DNS to find the Service.
• Audit the kube-proxy’s configuration at every release. The kube-proxy’s configuration should be version-controlled; the audit catches the failures.
• Monitor the kube-proxy’s metrics. The kube-proxy’s metrics expose the Service’s dataplane health.
• Test the Service in non-production. A staging cluster that mirrors production is the right place to test the Service.
• Document the Service’s design. The Service is the cluster’s load balancer; the documentation is the cluster’s networking reference.