The Production Observability Topology

The production topology puts collectors between workloads and backends. Grafana and Alertmanager close the loop.

What it is

A precise definition scoped to production operations.

Why a sysadmin cares

Production framing.

How it works

The mental model.

example_setting: value

How to configure it

Real configuration examples with annotated options.

promtool check config /etc/prometheus/prometheus.yml

How to validate it

Commands the operator runs to confirm the configuration is live and correct.

How it can fail

Failure modes: silent misconfiguration, crash on load, performance regression, permissions failure, schema / version drift.

How to troubleshoot it

Diagnostic order: was it working before? what does the service view say? what does the platform view say? form hypothesis, find evidence, test, validate.

The Production Topology in One Diagram

The full observability topology at production scale:

graph TB
    subgraph Workload
        W1[Linux Host]
        W2[Container]
        W3[Service]
    end
    
    subgraph Collector Layer
        W1 --> nExp[node_exporter]
        W2 --> cAdv[cAdvisor]
        W3 --> OTelC[OTel Collector]
    end
    
    subgraph Backend Layer
        nExp --> Prom[Prometheus]
        cAdv --> Prom
        OTelC --> Prom
        OTelC --> Loki[Loki]
        OTelC --> Tempo[Tempo]
    end
    
    subgraph Presentation
        Prom --> AM[Alertmanager]
        AM --> R[Receivers]
        Prom --> Grafana[Grafana]
        Loki --> Grafana
        Tempo --> Grafana
    end

The canonical production observability topology. Each layer is covered in detail in its respective Part.

Security implications

The topic has security implications wherever the relevant component exposes an HTTP endpoint, an authentication layer, or a credential.

Performance implications

Performance implications come from cardinality, scrape / push interval, rule size, retention, and query cost.