Cluster observability — the four pillars

Cluster observability is the discipline of making the cluster’s behavior visible. The four pillars are metrics, logs, traces, and events. Each pillar has its signal, its tool, and its storage. This lesson walks the four pillars, the signals, the tools, and the production patterns.

The four pillars

flowchart LR
    A[Cluster observability] --> B[Metrics]
    A --> C[Logs]
    A --> D[Traces]
    A --> E[Events]
    B --> F[Prometheus]
    C --> G[Loki/ELK]
    D --> H[Jaeger/Tempo]
    E --> I[Kubernetes events]

Each pillar is a different angle on the cluster’s behavior.

The metrics pillar

The metrics pillar is the time-series numerical data:

# Cluster CPU usage
sum(rate(container_cpu_usage_seconds_total[5m]))

# Pod count
count(kube_pod_info)

# HTTP request rate
rate(http_requests_total[5m])

The metrics are collected by Prometheus; the storage is Prometheus’s time-series database.

The logs pillar

The logs pillar is the unstructured text:

2026-08-16T10:00:00.000Z INFO  request_handler.go:42 request received: GET /api/users
2026-08-16T10:00:00.100Z INFO  request_handler.go:55 response sent: 200 OK in 100ms

The logs are collected by Promtail or Fluent Bit; the storage is Loki or Elasticsearch.

The traces pillar

The traces pillar is the request journey:

sequenceDiagram
    participant C as Client
    participant A as API
    participant B as Backend
    participant D as Database
    C->>A: GET /api/users
    A->>B: forward request
    B->>D: SELECT * FROM users
    D-->>B: result
    B-->>A: response
    A-->>C: 200 OK

The traces are collected by OpenTelemetry; the storage is Jaeger or Tempo.

The events pillar

The events pillar is the Kubernetes events:

Type     Reason            Age   From               Message
----     ------            ----  ----               -------
Normal   Scheduled         5m    default-scheduler  Successfully assigned nginx-1-abc to worker-1
Normal   Pulling           5m    kubelet            Pulling image nginx:1.25
Normal   Pulled            5m    kubelet            Successfully pulled image nginx:1.25
Normal   Created           5m    kubelet            Created container nginx
Normal   Started           5m    kubelet            Started container nginx

The events are collected by event-exporter; the storage is the cluster’s events API.

The signal collection

The collection is the discipline of gathering the signals:

flowchart LR
    A[Cluster] --> B[Collection]
    B --> C[Metrics: Prometheus]
    B --> D[Logs: Promtail/Fluent Bit]
    B --> E[Traces: OTLP]
    B --> F[Events: event-exporter]
    C --> G[Storage]
    D --> G
    E --> G
    F --> G

The collection is the input for the storage.

The signal storage

The storage is the discipline of persisting the signals:

The storage is per signal.

The signal query

The query is the discipline of asking the right question:

# Metrics: query the time-series database
sum(rate(http_requests_total[5m])) by (status)

# Logs: query the log database
{service="nginx"} |= "error"

# Traces: query the trace database
trace_id = "abc123"

# Events: query the events API
kubectl get events --field-selector type=Warning

The query is the input for the dashboard or the alert.

The cross-course references

The Observability course covers the tools in detail.

• The Prometheus course (Part LXXXVIII) covers the metrics.
• The Loki course (Part LXXXIX) covers the logs.
• The Jaeger course (Part XC) covers the traces.
• The Events course (Part XCI) covers the events.

Quiz

Production discipline

• Configure all four pillars. Metrics, logs, traces, events.
• Use the canonical tools. Prometheus, Loki, Jaeger, event-exporter.
• Configure the storage. Short-term and long-term.
• Configure the collection. Per workload.
• Configure the dashboards. Grafana integrates the four pillars.
• Document the observability setup. Per workload.

The four pillars are the cluster’s observability. Operating it well is configuring all four, using the canonical tools, and integrating them in dashboards.