Observability maturity — the progression of the discipline

The observability maturity model is the progression of the discipline. Level 1 is reactive monitoring; Level 5 is continuous observability. Each level builds on the previous. This lesson walks the levels, the signals, the alerts, and the production patterns.

The five levels

flowchart LR
    A[Level 1: Reactive] --> B[Level 2: Metrics]
    B --> C[Level 3: Traces]
    C --> D[Level 4: SLOs]
    D --> E[Level 5: Continuous]

Each level adds capabilities.

Level 1: Reactive monitoring

The Level 1 cluster has basic logs:

- Logs: stdout to ephemeral storage
- Metrics: none
- Traces: none
- Alerts: based on logs (kubectl logs)
- SLOs: none

The cluster is observable only by reading the logs.

Level 2: Metrics + alerts

The Level 2 cluster has metrics:

- Logs: stdout to Loki
- Metrics: Prometheus + Thanos
- Traces: none
- Alerts: based on metrics (Prometheus rules)
- SLOs: none

The cluster is observable via metrics and alerts.

Level 3: Distributed tracing

The Level 3 cluster has traces:

- Logs: stdout to Loki
- Metrics: Prometheus + Thanos
- Traces: Jaeger / Tempo
- Alerts: based on metrics and traces
- SLOs: none

The cluster is observable via metrics, logs, and traces.

Level 4: SLOs/SLIs

The Level 4 cluster has SLOs:

- Logs: stdout to Loki
- Metrics: Prometheus + Thanos
- Traces: Jaeger / Tempo
- Alerts: based on SLOs (burn rate)
- SLOs: defined per workload

The cluster is observable via SLOs; the alerts are SLO-driven.

Level 5: Continuous observability

The Level 5 cluster has continuous observability:

- Logs: stdout to Loki
- Metrics: Prometheus + Thanos
- Traces: Jaeger / Tempo
- Alerts: based on SLOs (burn rate)
- SLOs: defined per workload
- Continuous: new workloads inherit the stack
- Automation: alerts, dashboards, SLOs are automated

The cluster is observable via the full stack; new workloads inherit the observability.

The progression

The progression is per cluster and per workload:

flowchart LR
    A[Cluster 1: Level 2] --> B[Cluster 1: Level 3]
    B --> C[Cluster 1: Level 4]
    C --> D[Cluster 1: Level 5]
    E[Cluster 2: Level 1] --> F[Cluster 2: Level 2]

The progression is incremental; each cluster progresses independently.

The signals per level

The signals per level:

Each level adds signals.

The alerts per level

The alerts per level:

Each level adds alert sophistication.

The production patterns

The production patterns per level:

flowchart LR
    A[Level 1] --> B[Logs to Loki]
    B --> C[Metrics + alerts]
    C --> D[Distribute tracing]
    D --> E[SLOs]
    E --> F[Continuous]

Each level adds capabilities.

The assessment

The assessment is the current state:

# Cluster 1: Level 3 (metrics, logs, traces)
# Cluster 2: Level 2 (metrics, logs)
# Cluster 3: Level 1 (logs only)

The assessment is the input for the progression plan.

The cross-course references

• The Prometheus course (Part LXXXVIII) covers the metrics.
• The Loki course (Part LXXXIX) covers the logs.
• The SRE course covers the production discipline.

Quiz

Production discipline

• Assess the current maturity. The baseline.
• Plan the progression. The roadmap.
• Add signals incrementally. Metrics, logs, traces, SLOs.
• Add alerts incrementally. Threshold, SLO, burn rate.
• Document the maturity. The current level, the plan.
• Review the maturity quarterly. The progression check.

The observability maturity is the cluster’s progression. Operating it well is assessing the current state, planning the progression, and adding signals incrementally.