KSM Prometheus integration — the metrics flow

The KSM integration with Prometheus is the metrics flow. The PrometheusOperator provides the ServiceMonitor for auto-discovery. The static scrape config is the alternative. The recording rules and the alerts are the output. This lesson walks the integration, the patterns, and the verification.

The ServiceMonitor

The ServiceMonitor is the auto-discovery mechanism:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app: kube-state-metrics
spec:
  selector:
    matchLabels:
      app: kube-state-metrics
  endpoints:
  - port: http
    interval: 30s
    honorLabels: true

The ServiceMonitor selects the KSM Service and scrapes the HTTP endpoint.

The static scrape config

The static scrape config is the alternative:

scrape_configs:
  - job_name: 'kube-state-metrics'
    static_configs:
      - targets: ['kube-state-metrics.monitoring:8080']
    scrape_interval: 30s

The static config is the manual approach.

The scraping flow

sequenceDiagram
    participant P as Prometheus
    participant SM as ServiceMonitor
    participant SVC as KSM Service
    participant D as KSM Deployment
    P->>SM: discover ServiceMonitor
    SM->>P: add KSM Service to scrape config
    P->>SVC: scrape http://kube-state-metrics:8080/metrics
    SVC->>D: forward request
    D->>SVC: return metrics
    SVC->>P: return metrics
    P->>P: store the metrics

The scraping flow is the metrics path.

The recording rules

The recording rules:

# Prometheus recording rules
groups:
  - name: kube-state-metrics.rules
    rules:
      - record: kube_pod_status_ready_total
        expr: sum(kube_pod_status_ready{condition="true"}) by (namespace)

      - record: kube_pod_container_status_restarts_total
        expr: kube_pod_container_status_restarts_total

      - record: cluster:node_cpu:ratio
        expr: |
          sum(kube_node_status_allocatable_cpu_cores) by (cluster)
          -
          sum(rate(container_cpu_usage_seconds_total[5m])) by (cluster)

The recording rules pre-compute the metrics.

The alerts

The alerts using KSM metrics:

# Prometheus alert
groups:
  - name: kube-state-metrics.alerts
    rules:
      - alert: KubePodCrashLooping
        expr: |
          rate(kube_pod_container_status_restarts_total[10m]) * 60 * 5 > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"

      - alert: KubeDeploymentReplicasMismatch
        expr: |
          kube_deployment_spec_replicas
          !=
          kube_deployment_status_replicas_available
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has mismatched replicas"

The alerts are the SLO-driven inputs.

The kube-prometheus-stack

The kube-prometheus-stack is the canonical deployment:

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set kubeStateMetrics.enabled=true

The chart includes:

• Prometheus.
• Alertmanager.
• Grafana.
• kube-state-metrics.
• node_exporter.
• The ServiceMonitors.
• The recording rules.
• The alerts.

The chart is the production deployment.

The verification

The verification:

# Verify the ServiceMonitor
kubectl get servicemonitor -n monitoring kube-state-metrics

# Verify the targets
kubectl port-forward -n monitoring prometheus-xxx 9090
# Open http://localhost:9090/targets
# Verify kube-state-metrics is in the targets

# Verify the metrics
promtool query instant http://prometheus:9090 kube_pod_info

The verification confirms the integration.

The Grafana dashboards

The Grafana dashboards:

flowchart LR
    A[KSM metrics] --> B[Grafana]
    B --> C[Cluster dashboard]
    B --> D[Node dashboard]
    B --> E[Deployment dashboard]

The dashboards are the visualization.

The cross-course references

• The Prometheus course (Part LXXXVIII) covers the metrics.
• The Prometheus Operator course covers the auto-discovery.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Use the ServiceMonitor. The Prometheus Operator pattern.
• Configure the recording rules. The pre-computed metrics.
• Configure the alerts. The SLO-driven alerts.
• Use the kube-prometheus-stack. The canonical deployment.
• Verify the integration. The targets, the metrics, the rules.
• Document the integration. The ServiceMonitor, the rules, the alerts.

The KSM Prometheus integration is the metrics flow. Operating it well is via the ServiceMonitor, with the recording rules and the alerts, and the kube-prometheus-stack.