Bare metal LB problem — why cloud-provider LBs do not exist on-prem

Bare metal clusters cannot use cloud-provider load balancers. This lesson walks the problem, the alternatives, the design choices, and the discipline.

The problem

flowchart LR
    A[Service type LoadBalancer] --> B{Cloud provider?}
    B -->|AWS| C[AWS ELB created]
    B -->|GCP| D[GCP LB created]
    B -->|Azure| E[Azure LB created]
    B -->|Bare metal| F[No LB created]
    F --> G[Service stays Pending]

The problem:

• A Service of type LoadBalancer requests a cloud-provider load balancer.
• AWS, GCP, Azure create the LB.
• Bare metal has no cloud provider; the Service stays Pending forever.
• External clients have no IP to reach.

The alternatives

flowchart LR
    A[Alternatives] --> B[NodePort]
    A --> C[externalIPs]
    A --> D[MetalLB]
    A --> E[Cilium BGP]
    A --> F[F5 BIG-IP]
    A --> G[HAProxy]
    A --> H[Keepalived + IPVS]

The alternatives:

• NodePort. Every node exposes the Service on a port (30000-32767). External clients reach any node; kube-proxy routes to a Pod.
• externalIPs. The Service has externalIPs that are manually configured on nodes.
• MetalLB. L2 mode (ARP/NDP) or BGP mode.
• Cilium BGP. BGP mode (since Cilium 1.16+).
• F5 BIG-IP. Enterprise hardware/virtual load balancer.
• HAProxy. Open source, deployed externally.
• Keepalived + IPVS. HA pair with virtual IP.

The trade-offs

flowchart LR
    A[NodePort] --> B[+ Simple]
    A --> C["- High ports (30000+)"]
    A --> D[- Single-node bottleneck]
    E[externalIPs] --> F[+ Direct IP]
    E --> G[- Manual config]
    E --> H[- No failover]
    I[MetalLB L2] --> J["+ Simple, no router config"]
    I --> K[- Single-node bottleneck]
    L[MetalLB BGP] --> M[+ Multi-node]
    L --> N[- Requires router config]
    O[F5 BIG-IP] --> P[+ Enterprise features]
    O --> Q["- Cost, complexity"]

The trade-offs:

NodePort:

• Pros: Simple; no extra components.
• Cons: High ports (30000+); single-node bottleneck; no health-checked load balancing.

externalIPs:

• Pros: Direct IP; no extra components.
• Cons: Manual config; no failover; no load balancing.

MetalLB L2:

• Pros: Simple; no router config.
• Cons: Single-node bottleneck; slow failover.

MetalLB BGP:

• Pros: Multi-node; fast failover; ECMP.
• Cons: Requires router config; AS numbers.

F5 BIG-IP:

• Pros: Enterprise features (TLS offload, WAF).
• Cons: Cost ($10k+); operational complexity.

The design choices

flowchart TD
    A[Choose LB] --> B{Environment?}
    B -->|Development| C[NodePort]
    B -->|Production on-prem| D[MetalLB BGP]
    B -->|Production with F5| E[F5 BIG-IP]
    B -->|Cilium-based| F[Cilium BGP]
    A --> G{Need TLS offload?}
    G -->|Yes| E
    G -->|No| D
    A --> H{Need WAF?}
    H -->|Yes| E
    H -->|No| D

The design choices:

• Development. NodePort is sufficient.
• Production on-prem (small). MetalLB L2 is simple.
• Production on-prem (medium). MetalLB BGP with router cooperation.
• Production with Cilium. Cilium BGP integrates with the existing CNI.
• Production with enterprise features. F5 BIG-IP for TLS offload and WAF.

Quiz

The operational discipline

Bare metal LB in production rests on five non-negotiable elements:

• Choose the LB solution deliberately. Not by default.
• Test failover. A node failure must not disrupt traffic.
• Monitor LB health. The LB’s own metrics; alert on failures.
• Document the topology. IP pools, peers, routers.
• Plan the integration with the CNI. MetalLB and Cilium BGP interact; conflicts are possible.

Bare metal LB is production networking. Treat it with the same rigour as the CNI: test, monitor, document.