IPVS mode — high-performance Service load balancing

The kube-proxy IPVS mode programs the IP Virtual Server rules on every node. IPVS is a Linux kernel feature that provides load balancing at Layer 4. The performance is O(1) per packet; the mode scales better than iptables for large clusters. This lesson walks the IPVS mode, the scheduling algorithms, and the operational discipline.

The IPVS rule chain

The IPVS mode programs IP Virtual Server rules via ipvsadm. The rules are organised as a virtual server:

ipvsadm -Ln

TCP  10.96.0.10:80 rr
  -> 10.244.1.5:8080    Masq    1    0    0
  -> 10.244.1.6:8080    Masq    1    0    0
  -> 10.244.2.5:8080    Masq    1    0    0

The first line is the virtual server (ClusterIP:port). The lines below are the backends. The rr is the scheduling algorithm (round-robin).

flowchart LR
    A[ClusterIP: 10.96.0.10:80] --> B[IPVS rr]
    B -->|1| C[10.244.1.5:8080]
    B -->|2| D[10.244.1.6:8080]
    B -->|3| E[10.244.2.5:8080]

IPVS is implemented in the kernel; the packet is forwarded in O(1) regardless of the number of backends.

The scheduling algorithms

IPVS supports multiple scheduling algorithms:

The default is rr (round-robin). The cluster operator can configure the algorithm per Service via the Service.spec.sessionAffinity and Service.spec.ipvs.scheduler fields.

The IPVS mode’s configuration

The IPVS mode is configured by the --proxy-mode flag:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  scheduler: rr
  excludeCIDRs: []
  strictIP: false
  tcpTimeout: 0s
  tcpFinTimeout: 0s
  udpTimeout: 0s

The scheduler field is the default IPVS scheduling algorithm. The strictIP field controls whether the IPVS virtual server can bind to a non-local IP.

The IPVS and iptables interaction

The IPVS mode uses iptables for some functionality:

• Hairpin: the iptables rules enable hairpin traffic (Pod-to-self via the ClusterIP).
• Masquerading: the iptables rules perform masquerading for external traffic.
• Service NodePort: the iptables rules for NodePort are still in iptables.

The IPVS mode uses iptables for the features IPVS does not support. The cluster operator must understand the interaction.

The IPVS mode’s metrics

The kube-proxy exposes IPVS metrics:

# Substitute your own value before running:
NODE_IP=192.0.2.21   # a node running kube-proxy

curl "http://$NODE_IP:10249/metrics"

kubeproxy_ipvs_healthcheck_ping_threshold 3
kubeproxy_ipvs_healthcheck_ping_interval 1s
kubeproxy_ipvs_healthcheck_failures_counter 0

The metrics show the IPVS health check status. The cluster operator monitors these metrics for issues.

The failure modes

The IPVS mode’s failure modes:

• IPVS module not loaded: the IPVS kernel module is not loaded. The fix is to load the module.
• IPVS rule corruption: the IPVS rules are corrupted. The fix is to restart the kube-proxy.
• Scheduler mismatch: the scheduler is not supported by the kernel. The fix is to use a supported scheduler.
• Watch loop broken: the kube-proxy is not receiving events. The fix is to check the API connectivity.
• Hairpin not working: the Pod cannot reach itself via the ClusterIP. The fix is to update the iptables rules.

The operational discipline

The IPVS mode’s operational discipline:

• Document the kube-proxy mode. The cluster operator must understand which mode is used.
• Monitor the IPVS rule count. The rule count is the leading indicator of the cluster’s size.
• Test the kube-proxy in staging. The kube-proxy is critical infrastructure.
• Plan the mode’s evolution. The mode can be changed; the change is significant.
• Document the mode’s design. The IPVS mode is the cluster’s Service dataplane; the documentation is the reference.

Quiz

Production discipline

• IPVS scales better than iptables for large clusters. IPVS is O(1) per packet.
• Document the kube-proxy mode. The cluster operator must understand which mode is used.
• Monitor the IPVS rule count. The rule count is the leading indicator of the cluster’s size.
• Test the kube-proxy in staging. The kube-proxy is critical infrastructure.
• Plan the mode’s evolution. The mode can be changed; the change is significant.
• Document the mode’s design. The IPVS mode is the cluster’s Service dataplane; the documentation is the reference.
• Train the operations team on the IPVS diagnostics. The IPVS rules are the source of truth.
• Verify the IPVS kernel modules are loaded. The kernel modules are required for IPVS.