Eviction monitoring — metrics, alerts, and the kubelet's exposes

Monitoring eviction is the cluster’s primary defence against silent failure. The kubelet exposes the eviction metrics on its metrics endpoint; Prometheus scrapes the metrics; the operator alerts on the eviction rate. This lesson walks the metrics, the alerts, the dashboards, and the operational discipline.

The kubelet’s metrics endpoint

flowchart LR
    A["kubelet"] -->|HTTPS 10250| B["/metrics endpoint"]
    B --> C["Prometheus scrapes"]
    C --> D["PagerDuty alert"]
    B --> F["Grafana dashboards"]
    A --> G["node stats"]
    A --> H["container stats"]
    G --> B
    H --> B

The kubelet exposes its metrics on the HTTPS endpoint

The kubelet exposes its metrics on the HTTPS endpoint on port 10250:

# The kubelet's address; substitute your own node's IP:
NODE_IP=192.0.2.18

curl -k "https://$NODE_IP:10250/metrics" | grep -E "evictions|pressure"

The endpoint is the kubelet’s Prometheus metrics endpoint. The metrics are in the Prometheus format.

The kubelet’s metrics are scoped to the node. The metrics include the node’s resource usage, the container’s resource usage, and the eviction metrics.

The eviction metrics

The kubelet’s eviction metrics:

The metrics are counters and gauges. The counters increment on each eviction; the gauges reflect the current state.

The Prometheus scrape configuration

The Prometheus scrape configuration for the kubelet:

scrape_configs:
  - job_name: kubelet
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

The scrape config uses the Kubernetes API to discover the nodes; the metrics are scraped via the API server’s proxy. The scraper uses the cluster’s service account for authentication.

The alert rules

The Prometheus alert rules for the eviction:

groups:
  - name: eviction
    rules:
      - alert: NodeMemoryPressure
        expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.node }} has memory pressure"
          description: "Node {{ $labels.node }} has been in MemoryPressure for more than 5 minutes."

      - alert: NodeDiskPressure
        expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.node }} has disk pressure"
          description: "Node {{ $labels.node }} has been in DiskPressure for more than 5 minutes."

      - alert: NodePIDPressure
        expr: kube_node_status_condition{condition="PIDPressure",status="true"} == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.node }} has PID pressure"
          description: "Node {{ $labels.node }} has been in PIDPressure for more than 5 minutes."

      - alert: NodeEvictionRateHigh
        expr: rate(kubelet_pods_evicted_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.node }} is evicting pods at a high rate"
          description: "Node {{ $labels.node }} has evicted {{ $value }} pods per second over the last 5 minutes."

The alert rules cover the three pressure conditions and the eviction rate. The for: 5m clause gives the cluster time to recover the node.

The Grafana dashboard

The Grafana dashboard for the eviction:

panels:
  - title: Pod Evictions by Node
    type: graph
    datasource: prometheus
    targets:
      - expr: rate(kubelet_pods_evicted_total[5m])
        legend: "{{ node }}"
    yaxes:
      - unit: short

  - title: Memory Pressure by Node
    type: graph
    datasource: prometheus
    targets:
      - expr: kube_node_status_condition{condition="MemoryPressure",status="true"}
        legend: "{{ node }}"
    yaxes:
      - unit: short

  - title: Disk Pressure by Node
    type: graph
    datasource: prometheus
    targets:
      - expr: kube_node_status_condition{condition="DiskPressure",status="true"}
        legend: "{{ node }}"

  - title: PID Pressure by Node
    type: graph
    datasource: prometheus
    targets:
      - expr: kube_node_status_condition{condition="PIDPressure",status="true"}
        legend: "{{ node }}"

  - title: Memory Available by Node
    type: graph
    datasource: prometheus
    targets:
      - expr: kubelet_memory_available_bytes
        legend: "{{ node }}"
    yaxes:
      - unit: bytes

The dashboard shows the eviction rate, the pressure conditions, and the resource availability. The dashboard is the operator’s primary tool for monitoring the eviction.

The kubelet’s metrics exposure

The kubelet exposes its metrics on the HTTPS endpoint. The endpoint is authenticated by the kubelet’s client certificate. The Prometheus scraper uses the cluster’s service account.

The kubelet’s metrics exposure is configurable:

kubelet --authentication-token-webhook
        --authorization-mode=Webhook

The kube-apiserver’s node proxy forwards the metrics requests to the kubelet. The kubelet’s authentication is checked; the request is forwarded or rejected.

The production rule is to restrict the kubelet’s metrics exposure to the cluster’s Prometheus. The kubelet’s metrics endpoint should not be exposed to the public network.

The eviction’s slow churn

The eviction’s slow churn is the most common silent failure. A cluster that is evicting a few pods per day is a cluster that is silently losing capacity.

The diagnostic:

promtool query instant http://prometheus:9090 \
  'sum by (node) (rate(kubelet_pods_evicted_total[7d]))'

The query returns the eviction rate over the last 7 days. A non-zero rate is a cluster that is evicting.

The fix is to identify the cause:

• A workload that is over-committing memory.
• A workload that is leaking memory.
• A workload that is using more disk than expected.
• A workload that is leaking processes.

The production rule is to alert on the eviction rate; the silent eviction is the most common operational failure.

The alert’s runbook

The alert’s runbook should be documented:

runbook_url: "https://runbooks.example.com/node-eviction"

The runbook should include:

• The diagnostic steps.
• The common causes.
• The fix for each cause.
• The escalation path.

The runbook is the operator’s primary tool for responding to the alert. The production rule is to document the runbook; the undocumented runbook is the unaddressed runbook.

Quiz

Production discipline

• The kubelet’s metrics are the cluster’s signal. The metrics expose the eviction activity. The operator should monitor the metrics.
• The alert rules are the cluster’s response. The rules should fire on the pressure conditions and the eviction rate.
• The Grafana dashboard is the operator’s tool. The dashboard shows the eviction activity, the pressure conditions, and the resource availability.
• The runbook is the operator’s response. The runbook should be documented; the undocumented runbook is the unaddressed runbook.
• Audit the alerts at every release. The alerts should fire on the conditions; the audit catches the failures.
• Test the alerts in non-production. A staging cluster that mirrors production is the right place to test the alerts.