iptables mode — the legacy default and the rule explosion

The kube-proxy iptables mode is the legacy default. It programs the Linux iptables rules on every node. The rules form a chain: the ClusterIP matches, the traffic is split randomly across the Pods. The performance is O(n) per Service; the rule count grows linearly with the number of Services and Pods. This lesson walks the iptables mode, the rule chain, and the operational discipline.

The iptables rule chain

The iptables mode programs rules in the KUBE-SERVICES chain. The rules are organised as a chain:

-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-ABC
-A KUBE-SVC-ABC -m statistic --mode random --probabilities 0.333 -j KUBE-SEP-1
-A KUBE-SVC-ABC -m statistic --mode random --probabilities 0.500 -j KUBE-SEP-2
-A KUBE-SVC-ABC -j KUBE-SEP-3
-A KUBE-SEP-1 -s 10.244.1.5/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-1 -p tcp -j DNAT --to-destination 10.244.1.5:8080

The chain has three levels:

1. KUBE-SERVICES: matches the ClusterIP and the port. Jumps to the Service’s chain.
2. KUBE-SVC-XXX: the Service’s chain. Splits traffic randomly across the endpoints.
3. KUBE-SEP-XXX: the endpoint’s chain. Performs the DNAT to the Pod IP.

The random probability is set so that each endpoint gets an equal share of the traffic. The probabilities are computed based on the number of endpoints.

flowchart LR
    A[ClusterIP: 80] --> B[KUBE-SVC-ABC]
    B -->|1/3| C[KUBE-SEP-1: 10.244.1.5]
    B -->|1/3| D[KUBE-SEP-2: 10.244.1.6]
    B -->|1/3| E[KUBE-SEP-3: 10.244.2.5]

The traffic is split randomly across the endpoints. The iptables rule’s --probabilities is set to distribute the traffic evenly.

The rule count

The rule count grows with the number of Services and endpoints:

• Each Service clusterIP adds 1 rule in KUBE-SERVICES.
• Each Service adds 1 rule in KUBE-SVC-XXX.
• Each endpoint adds 1 rule in KUBE-SEP-XXX.

A Service with 100 endpoints adds 1 + 1 + 100 = 102 rules. A cluster with 100 Services and 100 endpoints per Service adds 10,200 rules.

The iptables rule count grows linearly with the number of Services and endpoints. The performance of the iptables dataplane is O(n) per packet, where n is the rule count.

The iptables mode’s configuration

The kube-proxy’s iptables mode is configured by the --proxy-mode flag:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: iptables
iptables:
  masqueradeBit: 14
  minSyncPeriod: 1s
  syncPeriod: 30s

The syncPeriod is the period at which the kube-proxy re-syncs the rules. The default is 30 seconds. The masqueradeBit is the iptables bit used for masquerading.

The kube-proxy’s metrics

The kube-proxy exposes Prometheus metrics:

# Substitute the node's address before running:
NODE=192.0.2.21

curl "http://$NODE:10249/metrics"

kubeproxy_sync_proxy_rules_duration_seconds_count 100
kubeproxy_networkprogramming_latency_seconds_bucket{le="0.1"} 250
iptables_rules_count 5000

The iptables_rules_count metric is the number of iptables rules. The cluster operator monitors this metric for the rule explosion.

The failure modes

The iptables mode’s failure modes:

• Rule explosion: the iptables rule count is too high. The fix is to switch to IPVS or eBPF.
• Rule corruption: the iptables rules are corrupted. The fix is to restart the kube-proxy.
• Slow sync: the sync duration is too high. The fix is to investigate the API server’s performance.
• Watch loop broken: the kube-proxy is not receiving events. The fix is to check the API connectivity.
• High CPU on the node: the iptables rule evaluation uses CPU. The fix is IPVS, whose hash table matches a ClusterIP in constant time instead of walking the chain for every packet.

The operational discipline

The iptables mode’s operational discipline:

• Document the kube-proxy mode. The cluster operator must understand which mode is used.
• Monitor the iptables rule count. A growing rule count is a leading indicator of the rule explosion.
• Switch to IPVS or eBPF for large clusters. The IPVS and eBPF modes scale better.
• Test the kube-proxy in staging. The kube-proxy is critical infrastructure.
• Plan the mode’s evolution. The mode can be changed; the change is significant.
• Document the mode’s design. The iptables mode is the cluster’s Service dataplane.

Quiz

Production discipline

• The iptables mode is the legacy default. The performance is O(n) per packet.
• Monitor the iptables rule count. A growing rule count is a leading indicator of the rule explosion.
• Switch to IPVS or eBPF for large clusters. The IPVS and eBPF modes scale better.
• Document the kube-proxy mode. The cluster operator must understand which mode is used.
• Test the kube-proxy in staging. The kube-proxy is critical infrastructure.
• Plan the mode’s evolution. The mode can be changed; the change is significant.
• Document the mode’s design. The iptables mode is the cluster’s Service dataplane; the documentation is the reference.
• Train the operations team on the iptables diagnostics. The iptables rules are the source of truth.