kube-proxy fundamentals — the Service dataplane

kube-proxy is the cluster’s Service dataplane. It watches the API for Services and EndpointSlices and programs the iptables, IPVS, or eBPF rules on every node. The kubelet is the orchestrator; the CNI is the Pod network; the kube-proxy is the Service network. This lesson walks the kube-proxy, the watch loop, and the operational discipline.

What kube-proxy does

kube-proxy is a DaemonSet that runs on every node. It watches the API for Services and EndpointSlices and programs the local dataplane rules:

sequenceDiagram
    autonumber
    participant API as Kubernetes API
    participant KP as kube-proxy
    participant DP as Dataplane (iptables/IPVS/eBPF)
    API->>KP: Service billing created
    KP->>DP: program DNAT 10.96.0.10 -> Pod IPs
    API->>KP: EndpointSlice updated
    KP->>DP: update DNAT rules

The kube-proxy is the bridge between the API and the dataplane. The API is the source of truth; the dataplane is what the kernel uses to forward traffic.

The watch loop

The kube-proxy uses an informer to watch the API:

// Pseudocode
informer := factory.Core().V1().Services().Informer()
informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc:    onServiceAdd,
    UpdateFunc: onServiceUpdate,
    DeleteFunc: onServiceDelete,
})
informer.Run(stopCh)

The informer caches the API objects and emits events on changes. The kube-proxy translates each event into a dataplane rule update.

The EndpointSlice informer runs in parallel:

endpointSliceInformer := factory.Discovery().V1().EndpointSlices().Informer()
endpointSliceInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc:    onEndpointSliceAdd,
    UpdateFunc: onEndpointSliceUpdate,
    DeleteFunc: onEndpointSliceDelete,
})

The kube-proxy is reactive: when the EndpointSlice changes, the kube-proxy updates the dataplane.

The three modes

The kube-proxy has three modes:

The mode is set by the --proxy-mode flag. The default in Kubernetes 1.34 is iptables; IPVS is opt-in; eBPF is provided by Cilium’s kube-proxy replacement.

The iptables mode

The iptables mode programs the Linux iptables rules on every node. The rules are DNAT rules that translate the ClusterIP to a Pod IP.

-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-ABC
-A KUBE-SVC-ABC -m statistic --mode random --probabilities 0.333 -j KUBE-SEP-1
-A KUBE-SVC-ABC -m statistic --mode random --probabilities 0.500 -j KUBE-SEP-2
-A KUBE-SVC-ABC -j KUBE-SEP-3

The rules form a chain: the ClusterIP is matched, the traffic is split randomly across the Pods. The iptables mode is simple and well understood; the trade-off is the O(n) rule count per Service.

The IPVS mode

The IPVS mode programs the IP Virtual Server (IPVS) rules. IPVS is a Linux kernel feature that provides load balancing at Layer 4.

ipvsadm -Ln

TCP  10.96.0.10:80 rr
  -> 10.244.1.5:8080    Masq    1    0    0
  -> 10.244.1.6:8080    Masq    1    0    0
  -> 10.244.2.5:8080    Masq    1    0    0

The IPVS mode supports multiple scheduling algorithms (round-robin, least connections, etc.) and is more efficient than iptables for large Services.

The eBPF mode

The eBPF mode is provided by Cilium’s kube-proxy replacement. The cilium-agent replaces the kube-proxy and programs eBPF maps for the Service dataplane.

The eBPF mode is faster than iptables and IPVS, with no kernel rule overhead. The trade-off is that the Cilium agent is required; the kube-proxy is not used.

The kube-proxy’s metrics

The kube-proxy exposes Prometheus metrics:

# Substitute your own value before running:
NODE_IP=192.0.2.11

curl http://"$NODE_IP":10249/metrics

kubeproxy_sync_proxy_rules_duration_seconds_bucket{le="0.001"} 100
kubeproxy_sync_proxy_rules_duration_seconds_bucket{le="0.01"}  200
kubeproxy_sync_proxy_rules_duration_seconds_count            250
kubeproxy_networkprogramming_latency_seconds_bucket{le="0.1"} 100

The metrics show the rule sync duration and the network programming latency. The cluster operator monitors these metrics for performance issues.

The failure modes

The kube-proxy’s failure modes:

• kube-proxy down: the dataplane rules are not updated. The fix is to restart the kube-proxy.
• Dataplane rule corruption: the iptables or IPVS rules are corrupted. The fix is to restart the kube-proxy.
• Watch loop broken: the kube-proxy is not receiving events. The fix is to check the API connectivity.
• High rule count: the iptables rule count is too high. The fix is to switch to IPVS or eBPF.
• EndpointSlice updates lost: the kube-proxy misses an update. The fix is to restart the kube-proxy.

The operational discipline

The kube-proxy’s operational discipline:

• Document the kube-proxy mode. The cluster operator must understand which mode is used.
• Audit the kube-proxy’s logs. The logs are the source of truth.
• Monitor the kube-proxy’s metrics. The metrics are the leading indicator.
• Test the kube-proxy in staging. The kube-proxy is critical infrastructure.
• Plan the kube-proxy’s evolution. The kube-proxy mode can be changed; the change is significant.
• Document the kube-proxy’s design. The kube-proxy is the cluster’s Service dataplane.

Quiz

Production discipline

• kube-proxy is the cluster’s Service dataplane. The cluster operator must treat it as critical infrastructure.
• Watch the kube-proxy’s logs. The logs are the source of truth.
• Monitor the kube-proxy’s metrics. The metrics are the leading indicator.
• Test the kube-proxy in staging. The kube-proxy is critical infrastructure.
• Plan the kube-proxy’s evolution. The kube-proxy mode can be changed; the change is significant.
• Document the kube-proxy’s design. The kube-proxy is the cluster’s Service dataplane; the documentation is the reference.
• Document the kube-proxy’s mode. The mode is the cluster’s Service networking implementation.
• Document the kube-proxy’s upgrade path. The upgrade is part of the cluster’s maintenance.