KSM vs cAdvisor — different scopes of metrics

KSM and cAdvisor are two different metrics sources. KSM generates object metrics (state); cAdvisor generates resource metrics (CPU, memory). The two are complementary. This lesson walks the difference, the metrics sources, the metrics types, and the production patterns.

The two metrics sources

flowchart LR
    A[Cluster] --> B[KSM]
    A --> C[cAdvisor]
    B --> D[Object metrics]
    C --> E[Resource metrics]
    D --> F[Prometheus]
    E --> F

The two sources are independent.

The KSM metrics

The KSM metrics are object metrics:

# Pod state
kube_pod_status_phase
kube_pod_status_ready
kube_pod_container_status_running

# Deployment state
kube_deployment_status_replicas
kube_deployment_status_replicas_available

# Node state
kube_node_status_condition
kube_node_status_capacity

The KSM metrics describe the cluster’s objects.

The cAdvisor metrics

The cAdvisor metrics are resource metrics:

# Container CPU usage
container_cpu_usage_seconds_total

# Container memory usage
container_memory_working_set_bytes
container_memory_rss

# Container network
container_network_receive_bytes_total
container_network_transmit_bytes_total

# Container filesystem
container_fs_usage_bytes
container_fs_limit_bytes

The cAdvisor metrics describe the workloads’ resource usage.

The metrics types

The KSM metrics types:

kube_pod_status_phase: gauge (1 if phase = X, 0 otherwise)
kube_deployment_status_replicas: gauge
kube_pod_container_status_restarts_total: counter

The cAdvisor metrics types:

container_cpu_usage_seconds_total: counter
container_memory_working_set_bytes: gauge
container_network_receive_bytes_total: counter

The two have different metric types.

The metrics scope

The KSM metrics scope:

flowchart LR
    A[KSM] --> B[Pod]
    A --> C[Deployment]
    A --> D[Node]
    A --> E[Job]
    A --> F[HPA]
    A --> G[Service]
    A --> H[PersistentVolume]

The cAdvisor metrics scope:

flowchart LR
    A[cAdvisor] --> B[Container]
    A --> C[Pod]
    A --> D[Node]

The cAdvisor scope is narrower.

The metrics source

The KSM metrics source:

KSM watches the API server for object state changes.
The metrics are generated from the API objects.

The cAdvisor metrics source:

cAdvisor is embedded in the kubelet.
The metrics are generated from the cgroup filesystem.

The two have different sources.

The combined metrics

The combined metrics:

# Pod CPU usage (cAdvisor)
rate(container_cpu_usage_seconds_total{pod="nginx-1-abc"}[5m])

# Pod phase (KSM)
kube_pod_status_phase{pod="nginx-1-abc",phase="Running"}

# Pod readiness (KSM)
kube_pod_status_ready{pod="nginx-1-abc",condition="true"}

The combined metrics provide a complete view.

The Prometheus queries

The Prometheus queries:

# Pod CPU usage as a percentage of the request
rate(container_cpu_usage_seconds_total[5m])
/
kube_pod_container_resource_requests{resource="cpu"}

# Pod count
count(kube_pod_info)

# Pod ready
sum(kube_pod_status_ready{condition="true"}) by (namespace)

The queries combine the two metrics.

The dashboards

The dashboards combine the two:

flowchart LR
    A[KSM metrics] --> B[Cluster dashboard]
    C[cAdvisor metrics] --> B
    B --> D[Total CPU usage]
    B --> E[Total memory usage]
    B --> F[Pod count]
    B --> G[Pod ready count]

The dashboards use both metrics.

The alerts

The alerts:

# KSM alert: pod crash
- alert: KubePodCrashLooping
  expr: rate(kube_pod_container_status_restarts_total[10m]) > 0
  for: 5m

# cAdvisor alert: CPU usage high
- alert: ContainerCPUUsageHigh
  expr: |
    rate(container_cpu_usage_seconds_total[5m])
    /
    kube_pod_container_resource_requests{resource="cpu"} > 0.9
  for: 10m

The alerts use both metrics.

Cross-course references

• The Prometheus course (Part LXXXVIII) covers the queries.
• The Metrics Server course (Part LXXXVII) covers the resource metrics.
• The HPA course (Part LXXXII) uses the combined metrics.

Quiz

Production discipline

• Use KSM for object metrics. The state.
• Use cAdvisor for resource metrics. The usage.
• Combine the two in dashboards. The complete view.
• Combine the two in alerts. The detection.
• Document the combination. The queries, the labels.
• Test the queries. Verify the metrics are correct.

The KSM and cAdvisor are complementary metrics sources. Operating it well is using both, combining the queries, and documenting the metrics.