metrics.k8s.io and the Metrics Server — the default metrics source

The Metrics Server is the default metrics source for the HPA. It provides the metrics.k8s.io API, which exposes CPU and memory usage for pods and nodes. This lesson walks the Metrics Server, the deployment, the integration with HPA, and the failure modes.

The Metrics Server

The Metrics Server is a cluster add-on:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - name: metrics-server
        image: registry.k8s.io/metrics-server/metrics-server:v0.7.x
        args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s

The Metrics Server is a single replica. It is not HA; it is a passive aggregator.

flowchart LR
    A[Kubelet] --> B[Metrics Server]
    B --> C[metrics.k8s.io API]
    C --> D[HPA]
    C --> E[kubectl top]
    C --> F[Custom adapters]

The metrics collection

The Metrics Server collects metrics from the kubelets:

Metrics Server:
  - Periodically queries kubelet /metrics/resource
  - Stores aggregated metrics in memory
  - Exposes them via metrics.k8s.io API

The collection interval is 15 seconds (default). The metrics are aggregated across all nodes.

The kubectl top command

kubectl top nodes

NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
cp-1       500m         12%    2048Mi          26%
cp-2       450m         11%    1900Mi          25%
worker-1   800m         40%    4096Mi          52%
worker-2   600m         30%    3500Mi          45%

The output shows the CPU and memory usage per node.

kubectl top pods -A

NAMESPACE   NAME                              CPU(cores)   MEMORY(bytes)
default     nginx-1-abc                       100m         128Mi
default     nginx-2-def                       120m         130Mi
kube-system kube-apiserver-cp-1               500m         2048Mi

The output shows the CPU and memory usage per pod.

The metrics API

kubectl get --raw=/apis/metrics.k8s.io/v1beta1/nodes

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "items": [
    {
      "metadata": {"name": "cp-1"},
      "usage": {"cpu": "500m", "memory": "2048Mi"}
    }
  ]
}

The API exposes the metrics in JSON format.

The HPA integration

The HPA queries the metrics.k8s.io API:

sequenceDiagram
    participant HPA as HPA controller
    participant M as Metrics Server
    participant K as Kubelet
    loop every 15s
        HPA->>M: get pod metrics
        M->>K: get kubelet metrics
        K-->>M: CPU/memory
        M-->>HPA: aggregated metrics
        HPA->>HPA: compute replicas
        HPA->>HPA: update Deployment
    end

The Metrics Server is the bridge between the kubelet and the HPA.

The Helm installation

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server \
  --namespace kube-system \
  --set args="{--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname,--kubelet-use-node-status-port,--metric-resolution=15s}"

The Helm chart installs the Metrics Server.

The failure modes

The common failure modes:

Kubelet TLS

metrics-server: x509: certificate is valid for 10.0.1.10, not for 127.0.0.1

The Metrics Server cannot connect to the kubelet because the kubelet’s certificate does not include the IP address the Metrics Server is using. The fix is to specify --kubelet-preferred-address-types.

Metrics Server not running

error: Metrics API not available

The Metrics Server is not running. Check the pod status: kubectl get pods -n kube-system -l k8s-app=metrics-server.

High latency

metrics-server: scrapes took 30s, threshold 15s

The Metrics Server is slow to scrape. The cluster is too large or the kubelet is slow. Increase the --metric-resolution.

flowchart LR
    A[Metrics Server] --> B{Issue?}
    B -->|TLS| C[Check kubelet TLS]
    B -->|Not running| D[Check pod status]
    B -->|Latency| E[Increase resolution]

The TLS configuration

The Metrics Server requires the kubelet’s CA to be configured:

spec:
  containers:
  - name: metrics-server
    args:
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --kubelet-use-node-status-port
    - --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt

The --kubelet-certificate-authority flag specifies the CA bundle to trust the kubelet’s cert.

Cross-course references

• The Prometheus course covers custom metrics adapter.
• The Observability course covers metrics pipelines.
• The Helm course covers chart installation.

Quiz

Production discipline

• Deploy the Metrics Server. Required for HPA on resource metrics.
• Configure the kubelet TLS. The Metrics Server must trust the kubelet’s cert.
• Monitor the Metrics Server. Prometheus exposes its metrics.
• Tune the resolution. The default 15s is fine for most workloads.
• Test the HPA. Use load generation to verify the scaling.
• Document the configuration. The Metrics Server args, the HPA target.

The Metrics Server is the default metrics source. Operating it well is deploying it, configuring the TLS, and testing the HPA.