kube-state-metrics — the cluster's object metrics

kube-state-metrics (KSM) is the tool that generates metrics from Kubernetes API objects. The metrics are scraped by Prometheus; the metrics are the cluster’s object state. This lesson walks KSM, the metrics taxonomy, the deployment, and the integration with Prometheus.

What KSM does

KSM watches the API server for object state changes and generates metrics:

flowchart LR
    A[API server] --> B[KSM]
    B --> C[Metrics]
    C --> D[Prometheus]
    D --> E[Grafana]

KSM is a passive listener; it generates metrics from the API objects.

The metrics taxonomy

The KSM metrics taxonomy:

flowchart LR
    A[KSM metrics] --> B[Pod metrics]
    A --> C[Deployment metrics]
    A --> D[Node metrics]
    A --> E[Job metrics]
    A --> F[Service metrics]
    A --> G[DaemonSet metrics]
    A --> H[StatefulSet metrics]

Each object type has its metrics.

The pod metrics

The pod metrics:

kube_pod_info
kube_pod_status_phase
kube_pod_status_ready
kube_pod_status_scheduled
kube_pod_container_status_running
kube_pod_container_status_waiting
kube_pod_container_status_terminated
kube_pod_container_resource_requests
kube_pod_container_resource_limits
kube_pod_container_status_restarts_total
kube_pod_start_time
kube_pod_completion_time

The pod metrics describe the pod’s state.

The deployment metrics

The deployment metrics:

kube_deployment_status_replicas
kube_deployment_status_replicas_available
kube_deployment_status_replicas_unavailable
kube_deployment_status_replicas_updated
kube_deployment_status_observed_generation
kube_deployment_spec_replicas

The deployment metrics describe the deployment’s state.

The node metrics

The node metrics:

kube_node_info
kube_node_status_condition
kube_node_status_capacity
kube_node_status_allocatable
kube_node_spec_unschedulable

The node metrics describe the node’s state.

The deployment

KSM is deployed as a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.x
        ports:
        - name: http
          containerPort: 8080

The Deployment is a single replica. KSM is HA via the leader election (optional).

The Prometheus integration

The Prometheus integration:

# Prometheus scrape config
scrape_configs:
  - job_name: 'kube-state-metrics'
    static_configs:
      - targets: ['kube-state-metrics.monitoring:8080']

The Prometheus scrapes the KSM endpoint.

The metrics output

The metrics output:

# HELP kube_pod_status_phase The pods current phase.
# TYPE kube_pod_status_phase gauge
kube_pod_status_phase{namespace="default",pod="nginx-1-abc",phase="Running"} 1

The metrics are in OpenMetrics format.

The KSM labels

The KSM labels are the cluster’s label taxonomy:

kube_pod_info{namespace="default",pod="nginx-1-abc",node="worker-1",created_by_kind="Deployment",created_by_name="nginx"}

The labels include the namespace, the pod name, the node, the owner kind, and the owner name.

The cross-course references

• The Prometheus course (Part LXXXVIII) covers the metrics collection.
• The HPA course (Part LXXXII) uses the KSM metrics.
• The Cluster Autoscaler course (Part LXXXI) uses the KSM metrics.

Quiz

Production discipline

• Deploy KSM. Required for cluster-level observability.
• Configure Prometheus to scrape KSM. The metrics flow.
• Use the KSM metrics in dashboards. The cluster dashboards.
• Use the KSM metrics in alerts. The detect anomalies.
• Document the KSM metrics. The metric names, the labels.
• Test the KSM integration. Verify the metrics are scraped.

The KSM is the cluster’s object metrics. Operating it well is deploying it, configuring Prometheus, and using the metrics in dashboards and alerts.