KSM metrics taxonomy — the cluster's object state

The KSM metrics taxonomy is the cluster’s object state. There are ~50 metrics per object type, totaling hundreds of unique metrics. This lesson walks the taxonomy, the labels, the queries, and the dashboards.

The metrics taxonomy

The KSM metrics taxonomy:

flowchart LR
    A[KSM metrics] --> B[Pod]
    A --> C[Deployment]
    A --> D[StatefulSet]
    A --> E[DaemonSet]
    A --> F[ReplicaSet]
    A --> G[Job]
    A --> H[CronJob]
    A --> I[Node]
    A --> J[Service]
    A --> K[PersistentVolume]
    A --> L[PersistentVolumeClaim]
    A --> M[HorizontalPodAutoscaler]
    A --> N[Ingress]
    A --> O[ConfigMap]
    A --> P[Secret]

Each object type has its metrics.

The pod metrics

The pod metrics:

# Pod info (gauge)
kube_pod_info

# Pod phase (gauge)
kube_pod_status_phase

# Pod ready (gauge)
kube_pod_status_ready

# Pod scheduled (gauge)
kube_pod_status_scheduled

# Container status (gauge)
kube_pod_container_status_running
kube_pod_container_status_waiting
kube_pod_container_status_terminated

# Resource requests/limits (gauge)
kube_pod_container_resource_requests
kube_pod_container_resource_limits

# Restart count (counter)
kube_pod_container_status_restarts_total

# Time (gauge)
kube_pod_start_time
kube_pod_completion_time

# Labels (gauge)
kube_pod_labels

The pod metrics describe the pod’s lifecycle.

The deployment metrics

The deployment metrics:

# Status (gauge)
kube_deployment_status_replicas
kube_deployment_status_replicas_available
kube_deployment_status_replicas_unavailable
kube_deployment_status_replicas_updated
kube_deployment_status_observed_generation

# Spec (gauge)
kube_deployment_spec_replicas
kube_deployment_spec_paused

# Conditions (gauge)
kube_deployment_status_condition

The deployment metrics describe the deployment’s state.

The node metrics

The node metrics:

# Status (gauge)
kube_node_info
kube_node_status_condition
kube_node_status_capacity
kube_node_status_allocatable

# Spec (gauge)
kube_node_spec_unschedulable
kube_node_spec_taint
kube_node_spec_active
kube_node_status_addresses

The node metrics describe the node’s lifecycle.

The job metrics

The job metrics:

# Status (gauge)
kube_job_status_active
kube_job_status_succeeded
kube_job_status_failed
kube_job_status_completion_time

# Spec (gauge)
kube_job_spec_active_deadline_seconds
kube_job_spec_completions
kube_job_spec_parallelism

The job metrics describe the job’s state.

The HPA metrics

The HPA metrics:

# Status (gauge)
kube_horizontalpodautoscaler_status_current_replicas
kube_horizontalpodautoscaler_status_desired_replicas
kube_horizontalpodautoscaler_status_condition

# Spec (gauge)
kube_horizontalpodautoscaler_spec_min_replicas
kube_horizontalpodautoscaler_spec_max_replicas
kube_horizontalpodautoscaler_spec_target_metric

The HPA metrics describe the HPA’s state.

The metric labels

The KSM metrics include the labels:

kube_pod_info{
  namespace="default",
  pod="nginx-1-abc",
  node="worker-1",
  host_network="false",
  created_by_kind="Deployment",
  created_by_name="nginx",
  owner_kind="ReplicaSet",
  owner_name="nginx-7b9f8c5f6",
  pod_ip="10.0.1.20"
}

The labels include the namespace, the pod name, the node, and the owner.

The Prometheus queries

The Prometheus queries:

# Number of pods per namespace
count(kube_pod_info) by (namespace)

# Number of pods in Pending
sum(kube_pod_status_phase{phase="Pending"}) by (namespace)

# Number of ready pods
sum(kube_pod_status_ready{condition="true"}) by (namespace)

# Pod restart rate
rate(kube_pod_container_status_restarts_total[5m])

# Node pressure
kube_node_status_condition{condition="Ready",status="true"}

The queries are the input for the dashboards and alerts.

The dashboards

The dashboards:

flowchart LR
    A[KSM metrics] --> B[Prometheus]
    B --> C[Grafana]
    C --> D[Cluster dashboard]
    C --> E[Node dashboard]
    C --> F[Deployment dashboard]
    C --> G[Pod dashboard]

The dashboards are the output of the KSM metrics.

The alerts

The alerts:

# Prometheus alert
- alert: KubePodCrashLooping
  expr: |
    rate(kube_pod_container_status_restarts_total[10m]) * 60 * 5 > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is restarting"

The alerts are the input for the alerting.

Cross-course references

• The Prometheus course (Part LXXXVIII) covers the queries.
• The HPA course (Part LXXXII) uses the HPA metrics.
• The Cluster Autoscaler course (Part LXXXI) uses the KSM metrics.

Quiz

Production discipline

• Use the KSM metrics in dashboards. The cluster dashboards.
• Use the KSM metrics in alerts. The pod crash, the node pressure.
• Query by namespace and owner. The standard filters.
• Document the metric names. The metric taxonomy.
• Test the queries. Verify the metrics are correct.
• Review the alerts. The KSM-driven alerts.

The KSM metrics taxonomy is the cluster’s object state. Operating it well is using the metrics in dashboards and alerts, querying by namespace and owner, and documenting the metric names.