Metrics Server scaling — large clusters and high availability

The Metrics Server is a single replica for most clusters. For large clusters (1000+ nodes), the single replica is a bottleneck. This lesson walks the scaling, the bottlenecks, the HA, and the production patterns.

The single replica

The single replica is the production default:

spec:
  replicas: 1

The single replica is sufficient for most clusters. The Metrics Server is a passive listener; it does not lose data on restart.

The bottlenecks

The bottlenecks:

flowchart LR
    A[Kubelet] --> B[Metrics Server]
    B --> C[API server]
    B --> D[etcd]
    C --> E[HPA]
    C --> F[kubectl top]

The bottlenecks:

• Kubelet connections: the Metrics Server opens HTTP/2 connections to each kubelet.
• API server connections: the Metrics Server serves the metrics.k8s.io API via the API server.
• Memory: the Metrics Server stores the metrics in memory.

The scaling for large clusters

For large clusters (1000+ nodes):

spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: metrics-server
        args:
        - --metric-resolution=15s
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --max-http-client-idle-conns=100
        resources:
          requests:
            cpu: 1
            memory: 1Gi
          limits:
            cpu: 2
            memory: 2Gi

The parameters:

• --metric-resolution=15s: the collection interval.
• --max-http-client-idle-conns=100: the HTTP connection pool.

The connection pool

The HTTP connection pool:

args:
- --max-http-client-idle-conns=100

The default is 100 connections. For larger clusters, the pool may need to be increased.

The memory

The memory:

resources:
  requests:
    memory: 1Gi

The memory usage is proportional to the number of pods and nodes. For 1000 nodes, the memory is ~1 GB.

The HA pattern

The HA pattern is two replicas:

flowchart LR
    A[Kubelet 1] --> B[Metrics Server 1]
    A --> B2[Metrics Server 2]
    C[Kubelet 2] --> B
    C --> B2
    B --> D[API server]
    B2 --> D

The two replicas serve the same API. The API server load-balances the requests.

spec:
  replicas: 2

The horizontal scaling

The Metrics Server does not scale horizontally. The two replicas are the maximum.

flowchart LR
    A[Kubelet] --> B[Replica 1]
    A --> C[Replica 2]
    B --> D[API server]
    C --> D

The two replicas serve the API; the kubelet connections are doubled.

The KPI monitoring

The KPIs:

# Metrics Server scrape duration
metrics_server_scraper_duration_seconds

# Metrics Server memory usage
container_memory_working_set_bytes{pod=~"metrics-server-.*"}

# Metrics Server CPU usage
container_cpu_usage_seconds_total{pod=~"metrics-server-.*"}

The KPIs are the input for the alerts.

The alerts

The alerts:

- alert: MetricsServerDown
  expr: |
    up{job="metrics-server"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Metrics Server is down"

- alert: MetricsServerHighLatency
  expr: |
    metrics_server_scraper_duration_seconds > 10
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Metrics Server is slow"

The alerts are the early-warning system.

The troubleshooting

The troubleshooting:

# Check the pod status
kubectl describe pod -n kube-system -l k8s-app=metrics-server

# Check the logs
kubectl logs -n kube-system -l k8s-app=metrics-server

# Check the API Service
kubectl get apiservice v1beta1.metrics.k8s.io

# Check the metrics
kubectl top nodes

The troubleshooting is direct.

Cross-course references

• The HPA course (Part LXXXII) covers the integration.
• The Capacity Planning course (Part LXXXIV) covers the cluster sizing.
• The Prometheus course (Part LXXXVIII) covers the monitoring.

Quiz

Production discipline

• Use 1 replica for most clusters. Sufficient.
• Use 2 replicas for HA. For large clusters.
• Configure the connection pool. For kubelet connections.
• Configure the memory. Proportional to nodes/pods.
• Monitor the KPIs. The duration, the memory, the latency.
• Test the scaling. Verify the HPA still works.

The Metrics Server scaling is the cluster’s resource metrics. Operating it well is using the right replica count, the connection pool, and the memory limits, and monitoring the KPIs.