Cluster utilization analysis — the metrics of capacity

Cluster utilization analysis is the metric-driven view of the cluster’s capacity. The Prometheus metrics, the efficiency ratio, and the per-workload usage are the inputs for capacity decisions. This lesson walks the analysis, the metrics, the efficiency ratio, and the production patterns.

The metrics

The cluster utilization metrics come from Prometheus:

flowchart LR
    A[kube-state-metrics] --> B[Prometheus]
    C[cAdvisor] --> B
    D[Metrics Server] --> B
    B --> E[Utilization analysis]

The metrics sources are:

• kube-state-metrics: the cluster’s resource requests per workload.
• cAdvisor: the per-container resource usage.
• Metrics Server: the per-pod CPU and memory usage.

The key metrics

The key metrics for utilization analysis:

# Cluster CPU efficiency
sum(rate(container_cpu_usage_seconds_total[5m])) by (cluster)
/
sum(kube_node_status_allocatable_cpu_cores) by (cluster)

# Cluster memory efficiency
sum(container_memory_working_set_bytes) by (cluster)
/
sum(kube_node_status_allocatable_memory_bytes) by (cluster)

# Per-workload CPU usage
sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace, pod)

# Per-workload memory usage
sum(container_memory_working_set_bytes) by (namespace, pod)

The metrics are the input for the analysis.

The efficiency ratio

The efficiency ratio is the actual usage / the requests:

# CPU efficiency
sum(rate(container_cpu_usage_seconds_total[5m]))
/
sum(kube_pod_container_resource_requests{resource="cpu"})

The efficiency ratio is the basis for the analysis:

• < 30%: the workload is over-requested. The CPU requests are higher than the actual usage.
• 30-70%: the workload is well-sized. The requests match the usage.
• > 70%: the workload is under-requested. The CPU requests are lower than the actual usage; the workload is CPU-throttled or about to be.

The memory efficiency

The memory efficiency is similar:

# Memory efficiency
sum(container_memory_working_set_bytes)
/
sum(kube_pod_container_resource_requests{resource="memory"})

The memory efficiency is the input for the memory decisions.

The per-workload analysis

The per-workload analysis identifies the underutilized and overutilized workloads:

# Per-namespace CPU usage
sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace)

# Per-namespace CPU requests
sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)

The output:

NAMESPACE   CPU USAGE   CPU REQUEST   EFFICIENCY
default     2.5         5             50%
kube-system 1.0         4             25%
monitoring  0.5         2             25%

The default namespace is at 50% efficiency; the kube-system and monitoring namespaces are at 25%.

The capacity decisions

The capacity decisions are based on the analysis:

flowchart LR
    A[Utilization analysis] --> B{Efficiency?}
    B -->|< 30%| C[Right-size: reduce requests]
    B -->|30-70%| D[OK: keep current]
    B -->|> 70%| E[Right-size: increase requests]

The decisions are per workload.

The right-sizing

The right-sizing is the VPA’s role:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa
spec:
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: nginx
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi

The VPA adjusts the requests based on the actual usage.

The scaling decisions

The scaling decisions are based on the cluster’s overall utilization:

Cluster CPU efficiency: 80%
Cluster nodes: 5
Cluster CPU capacity: 40 vCPU
Cluster CPU usage: 32 vCPU

Decision: scale up the cluster (add 1-2 nodes)

The scaling decisions are the cluster’s growth.

The cost optimization

The cost optimization is the cluster’s right-sizing:

Cluster memory efficiency: 30%
Cluster memory capacity: 160Gi
Cluster memory usage: 48Gi

Decision: scale down the cluster (remove 1-2 nodes)

The cost optimization is the cluster’s efficiency.

The monitoring

The utilization analysis is automated:

# Prometheus alert
- alert: KubeClusterHighCPUUsage
  expr: |
    sum(rate(container_cpu_usage_seconds_total[5m]))
    /
    sum(kube_node_status_allocatable_cpu_cores) > 0.85
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Cluster CPU efficiency > 85%"

- alert: KubeClusterLowCPUUsage
  expr: |
    sum(rate(container_cpu_usage_seconds_total[5m]))
    /
    sum(kube_node_status_allocatable_cpu_cores) < 0.30
  for: 1h
  labels:
    severity: info
  annotations:
    summary: "Cluster CPU efficiency < 30%; consider scaling down"

The alerts are the early-warning system.

Cross-course references

• The VPA course (Part LXXXIII) covers the right-sizing.
• The HPA course (Part LXXXII) covers the scaling.
• The Prometheus course (Part LXXXVIII) covers the metrics.

Quiz

Production discipline

• Monitor the cluster efficiency. Prometheus exports the metrics.
• Right-size the workloads. Use VPA for the right-sizing.
• Scale the cluster. Use Cluster Autoscaler for the scaling.
• Alert on the efficiency. Prometheus alerts on the high/low efficiency.
• Document the decisions. The right-sizing, the scaling, the rationale.
• Review the utilization. Quarterly review of the efficiency.

The utilization analysis is the cluster’s diagnostic. Operating it well is monitoring the metrics, computing the efficiency, and acting on the decisions.