Cluster-level vs workload-level observability

Cluster observability has two scopes: cluster-level (the cluster’s health, the API server, the kubelet, etcd) and workload-level (the application’s behavior). The signals, the tools, and the operators differ. This lesson walks the two scopes, the signals, the tools, and the production patterns.

The two scopes

flowchart LR
    A[Observability] --> B[Cluster-level]
    A --> C[Workload-level]
    B --> D[API server, kubelet, etcd]
    C --> E[Application behavior]

The two scopes are different. The cluster-level is the infrastructure; the workload-level is the application.

The cluster-level signals

The cluster-level signals:

# API server request rate
apiserver_request_total

# Kubelet runtime operations
kubelet_runtime_operations_total

# etcd disk fsync
etcd_disk_wal_fsync_duration_seconds

# Node CPU usage
node_cpu_usage_seconds_total

The cluster-level signals are emitted by the cluster components.

The workload-level signals

The workload-level signals:

# Application HTTP requests
http_requests_total

# Application latency
http_request_duration_seconds

# Application error rate
http_errors_total

# Database query duration
db_query_duration_seconds

The workload-level signals are emitted by the application.

The cluster-level tools

The cluster-level tools:

flowchart LR
    A[Cluster] --> B[Prometheus]
    A --> C[event-exporter]
    A --> D[kubelet logs]
    B --> E[Grafana]
    C --> E
    D --> E

• kube-state-metrics: the cluster’s resource requests.
• cAdvisor: the per-container resource usage.
• node_exporter: the host’s metrics.
• Prometheus: the cluster’s metrics database.
• event-exporter: the cluster’s events to S3.

The workload-level tools

The workload-level tools:

flowchart LR
    A[Application] --> B[Prometheus]
    A --> C[Promtail/Fluent Bit]
    A --> D[OpenTelemetry]
    B --> E[Grafana]
    C --> E
    D --> E

• Application metrics: the application’s Prometheus metrics endpoint.
• Promtail/Fluent Bit: the application’s stdout collector.
• OpenTelemetry SDK: the application’s trace emitter.
• Grafana: the application’s dashboards.

The cluster-level operators

The cluster-level operators are the cluster administrators:

• Cluster SRE: the cluster’s health, the API server, the kubelet.
• Network engineer: the CNI, the network policies.
• Storage engineer: the CSI, the storage classes.

The cluster-level operators care about the cluster’s health.

The workload-level operators

The workload-level operators are the application developers:

• Application developer: the application’s behavior.
• Application SRE: the application’s SLOs, the alerts.
• Product owner: the application’s business metrics.

The workload-level operators care about the application’s behavior.

The cluster-level dashboards

The cluster-level dashboards:

# Cluster CPU usage
sum(rate(container_cpu_usage_seconds_total[5m])) by (node)

# Node memory usage
sum(container_memory_working_set_bytes) by (node)

# Pod count
count(kube_pod_info) by (namespace)

# API server request rate
apiserver_request_total

The cluster-level dashboards are operator-facing.

The workload-level dashboards

The workload-level dashboards:

# HTTP request rate
rate(http_requests_total[5m])

# HTTP request latency (p99)
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# HTTP error rate
rate(http_errors_total[5m])

# Database query latency
histogram_quantile(0.99, rate(db_query_duration_seconds_bucket[5m]))

The workload-level dashboards are developer-facing.

The division of responsibility

The division of responsibility:

The division is per scope.

The cross-cutting concerns

The cross-cutting concerns:

• Networking: the CNI metrics, the network policies.
• Storage: the CSI metrics, the PVC latencies.
• Security: the audit logs, the admission logs.

The cross-cutting concerns are shared between the two scopes.

Cross-course references

• The kube-state-metrics course (Part LXXXVI) covers the cluster’s metrics.
• The Prometheus course (Part LXXXVIII) covers the metrics database.
• The Loki course (Part LXXXIX) covers the logs.

Quiz

Production discipline

• Configure both scopes. Cluster-level and workload-level.
• Identify the owners. Cluster SRE for cluster; developers for workloads.
• Use the canonical tools. Prometheus, Loki, Jaeger, event-exporter.
• Configure the dashboards. Per scope.
• Configure the alerts. Per scope.
• Document the division. The owners, the tools, the dashboards.

The two scopes are the cluster’s observability. Operating it well is configuring both, identifying the owners, and using the canonical tools.