The Production Observability Topology · Observability

A new joiner on the SRE team opens the platform dashboard. Every panel is green. She clicks the checkout-service panel and follows the link to the underlying metric, then to the application logs, then to a single request trace. The trace loads in two seconds. She pivots from the slow span to the database host metrics, then to the alertmanager silence history. She has the root cause in eleven minutes. None of this is magic. Every hop she took was designed in advance, on a whiteboard, with names.

The production observability topology is the whiteboard. It is the set of decisions that say: workloads produce signals here, collectors fan-in here, long-term storage lives here, alerts leave here, humans read here. Get the topology wrong and the panels can be green while the service is on fire. Get it right and the panels are honest about both health and investigation depth.

What the topology is

The production observability topology is the placement and wiring of every component that turns workload behaviour into operator decisions. It is five layers.

                    +---------------------------------+
                    |       Presentation (humans)     |
                    |  Grafana 11.x, Alertmanager UI  |
                    +---------------+-----------------+
                                    |
                    +---------------v-----------------+
                    |       Alerting (machines)       |
                    |  Alertmanager, receivers, Pager |
                    +---------------+-----------------+
                                    |
                    +---------------v-----------------+
                    |       Storage (queries)         |
                    |  Prometheus TSDB | Loki | Tempo |
                    +---------------+-----------------+
                                    |
                    +---------------v-----------------+
                    |       Collection (fan-in)       |
                    |  OTel Collector / Alloy         |
                    |  node_exporter, cAdvisor        |
                    +---------------+-----------------+
                                    |
                    +---------------v-----------------+
                    |       Workload (signals)        |
                    |  Hosts, containers, services     |
                    +---------------------------------+

Workload layer. Linux hosts, Docker containers, container orchestrators, and the services that run inside them. The workload layer owns producing signals — emitting metrics, writing logs, sending spans.
Collection layer. Processes that pull from or receive from the workload. node_exporter and cAdvisor expose host and container metrics; the OpenTelemetry Collector (or Grafana Alloy, the OTel-compatible Grafana distribution) receives traces, logs, and OTLP metrics.
Storage layer. The places signals rest. Prometheus keeps metrics in a local TSDB; Loki indexes log streams and stores compressed chunks in object storage; Tempo stores trace blocks in object storage.
Alerting layer. Alertmanager groups, routes, and deduplicates alerts from Prometheus (and from Loki, Tempo, and Grafana-managed alerts) and ships them to receivers.
Presentation layer. Grafana queries the storage layer and Alertmanager and renders dashboards. Humans read here.

Every layer has a single primary responsibility. The collection layer never queries; the storage layer never scrapes; the presentation layer never stores long-term. When a layer reaches outside its lane, the topology quietly rots.

Why a sysadmin cares

Three failure shapes appear when the topology is unstated or unloved:

The green dashboard with a dead service. A team scrapes everything through Prometheus, never sets up probes against the service itself, and the service has been 503-ing for ten minutes. The host metrics are fine because the host is fine. The dashboard is green because nothing on the dashboard asks the question that would have surfaced the 503.
The collector that ate the host. A single OTel Collector runs as a central gateway on a 4 GB VM. The workload generates 40 000 spans per second. The Collector drops to its knees; the SDKs buffer and then drop; the team concludes “tracing is unreliable” and disables it. The topology was wrong; the tracing tooling was fine.
The alert that nobody owns. A team has five notification channels wired into Alertmanager, three of which no human watches. Alerts land in a chat room, sit there, and time out. The on-call never sees the page. The topology has a layer that nobody looks at.

The topology is the contract between the operators and the platform. When the contract is explicit, the failure modes above become impossible to introduce without notice. When the contract is implicit, the failure modes appear one at a time over a year.

How it works

The mental model is “data flows from workloads to operators, with explicit hops that each do one thing well.” Each hop has a protocol, a failure mode, and an owner.

                    +-------------------+
   Linux host ------->  node_exporter  ---+
                                         |
                                         v
                                     +---+-----+      +-----------+
                                     |  OTel  |----->| Prometheus |
                                     |Collector|----->|  (scrape) |
                                     |(agent)  |--+   +-----+-----+
   Container ----->  cAdvisor ---+     +---+-----+         |
                                 |         |          scrape :9090
                                 v         v               |
                            +----+---+ +---+----+      +---v---+
                            |  OTLP | | OTLP   |      |rules + |
                            | metrics| |traces  |      |alerts |
                            +---+---+ +---+----+      +---+---+
                                |        |               |
                                |        v               v
                                |   +----+----+    +-----------+
                                |   |  Tempo  |    |Alertmanager|
                                |   +---------+    +-----+-----+
                                v                      |
                          +-----+-----+                v
                          |   Loki   |             receivers
                          +---------+               (PagerDuty, Slack)
                                |
                                v
                          +-----+-----+        +-----------+
                          | Grafana  |<------>| Alerting UI|
                          +-----------+        +-----------+

A metric scrapes: workload exposes :9100, Prometheus pulls /metrics over HTTP, samples land in memory-mapped blocks, rules evaluate, alerts fire.

A log ships: Promtail (legacy) or the OTel filelog receiver tails log files, the agent pushes to Loki over HTTP, Loki indexes labels and stores compressed chunks in object storage.

A trace sends: the application SDK exports OTLP/gRPC to the agent, the agent batches and forwards to the gateway, the gateway fans out to Tempo over OTLP, Tempo writes blocks to object storage.

Each path has its own latency budget. Metric scrapes are pull-based and tolerate up to one scrape-interval of staleness. Log shipping is push-based and tolerates seconds of buffering. Trace export is push-based and tolerates milliseconds — drop on backpressure is the correct behaviour for traces.

How to configure it

The topology is configured as five configuration files, one per layer. Below is the production shape for each, annotated.

# /etc/prometheus/prometheus.yml  -- storage layer (metrics)
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    region: eu-west-1
    env: production

# Alerting layer wired into the storage layer.
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager.observability.svc:9093']

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['host-a:9100', 'host-b:9100']

  - job_name: otel-collector
    static_configs:
      - targets: ['otel-gateway:8889']

The scrape interval, external labels, and alertmanager wiring are the topology decisions. Each target that appears in scrape_configs is a node in the topology diagram.

# /etc/otelcol/config.yaml  -- collection layer (agent on host)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317
    tls:
      insecure: false
      cert_file: /etc/otelcol/certs/client.crt
      key_file: /etc/otelcol/certs/client.key

service:
  pipelines:
    metrics: { receivers: [otlp], exporters: [otlp/gateway] }
    traces:  { receivers: [otlp], exporters: [otlp/gateway] }
    logs:    { receivers: [otlp], exporters: [otlp/gateway] }

The agent owns one receiver (loopback) and one exporter (the gateway). Everything heavier — tail sampling, redaction, fan-out — belongs on the gateway, not the agent.

# /etc/loki/loki-config.yaml  -- storage layer (logs)
common:
  ring:
    kvstore:
      store: memberlist
  replication_factor: 3

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      chunks: tsdb
      index: tsdb

storage_config:
  aws:
    s3: s3://eu-west-1/loki-prod
    bucketnames: loki-prod

Loki uses memberlist for its hash ring and an S3-compatible bucket for chunks. The replication factor of 3 is the durability decision; the bucket name is the tenancy decision.

How to validate it

# READ-ONLY: each layer is up.
curl -fsS http://prometheus:9090/-/ready      # "ready"
curl -fsS http://loki:3100/ready             # "ready"
curl -fsS http://tempo:3200/ready            # "ready"
curl -fsS http://alertmanager:9093/-/ready   # "ready"
curl -fsS http://grafana:3000/api/health     # {"database":"ok"}

# CONFIGURATION: promtool confirms the metrics topology is parseable.
promtool check config /etc/prometheus/prometheus.yml

# CONFIGURATION: the OTel Collector prints the rendered config.
otelcol-contrib validate --config=/etc/otelcol/config.yaml

# READ-ONLY: end-to-end signal flow on each pipeline.
curl -fsS 'http://prometheus:9090/api/v1/query?query=up' | jq '.data.result[0].metric'
# {"__name__":"up","instance":"host-a:9100","job":"node"}

logcli query '{job="node"}' --since=1m --limit=1
# 2026-08-13T10:14:02Z  host-a  level=info  msg="scrape ok"

# READ-ONLY: a trace round-trip.
curl -fsS 'http://tempo:3200/api/search?tags=service.name=checkout' | jq '.traces | length'
# 47

# CONFIGURATION: reload each layer without dropping state.
curl -fsS -X POST http://prometheus:9090/-/reload
curl -fsS -X POST http://loki:3100/ingester/ring?mode=READONLY

A clean validation: every /ready returns 200, promtool check passes, otelcol validate passes, and at least one signal of each type is queryable from a live target.

How it can fail

The most expensive topology failure modes, in order of how often they appear in incident reviews.

The collector becomes the bottleneck. A central gateway ingests 80 000 spans/s from 600 agents. The gateway runs at 100% CPU; the memory_ballast is too small; the receivers start returning ResourceExhausted. Symptom: SDK-side errors in the application logs, drop metrics climbing in otelcol_exporter_sent_failed, dashboards going dark for the noisiest service.
Object storage is unreachable. Loki and Tempo both write to S3. S3 has an authentication rotation; the IAM role has not been granted s3:GetObject on the new path. Symptom: writes fail with 403 AccessDenied, ingester logs pile up, queries return resource_exhausted instead of data.
Network partition between collector and backend. A firewall change blocks port 4317 between the OTel Collector and Tempo. Symptom: trace export drops to 0% for the affected subnet; host and container metrics still arrive because they scrape on a different path.
Alertmanager wired to the wrong Prometheus. Two Prometheus instances run for HA. Alertmanager is configured to receive from prom-A:9093 only. A kill -9 of prom-A silences all alerting until the next config reload of Alertmanager picks up prom-B. Symptom: every panel still green but no alerts fire for an hour.
Grafana data source pointed at the staging backend. A config-as-code merge deploys https://loki-staging:3100 to production Grafana. Symptom: production dashboards show staging traffic for the four hours the misconfiguration persists.
Time skew between layers. The collectors run on hosts that are ten minutes behind NTP. Prometheus scrapes them anyway. Symptom: rules that compare against wall-clock time (rate windows, recording rules evaluated over for: 5m) produce wrong values; alerts fire late or never.

How to troubleshoot it

The diagnostic order is “is each layer up?”, “can each layer talk to the next?”, “is each layer doing what its configuration says?”.

Start at the top. Confirm Grafana can reach the data sources (/api/datasources/proxy/uid/<uid>/api/health). If Grafana is healthy, the storage layer is reachable; the problem is below or above this hop.
Check the receivers. Hit each backend’s /ready. A non-ready response tells you the storage layer cannot accept writes; queries may still work against cached blocks.
Check the exporters. otelcol_exporter_sent_spans and _sent_metrics should be monotonically increasing. A flat line means the gateway is dropping or the backend is rejecting.
Check the agents. From the gateway host, curl each agent’s local OTLP receiver port (127.0.0.1:4318/v1/traces). A refused connection means the agent is down or its network namespace is wrong.
Check the workloads. From each host, curl http://127.0.0.1:9100/metrics | head confirms the exporter is up. A failed scrape here is a workload-layer problem; the collector cannot fix it.
Reproduce at the lowest layer first. A trace that is missing in Tempo: confirm the application SDK is configured to export at all (OTEL_TRACES_EXPORTER=otlp), then confirm the agent’s OTLP listener is bound, then confirm the gateway can be reached from the agent’s network, then confirm Tempo accepts the export. Walk up from the bottom.

Security implications

Every hop is an attack surface. Each endpoint that accepts a scrape or a push should be either loopback-only or behind mTLS. node_exporter bound to 0.0.0.0:9100 is a credential leak waiting to happen.
Tenant separation at the boundary. Loki and Tempo use the X-Scope-OrgID header to identify tenants. A misconfigured data source proxy that strips or rewrites the header is a cross-tenant data leak.
Credential rotation. Each layer that uses IAM roles, bearer tokens, or basic auth has its own rotation cadence. The topology document should record which credential is used by which hop, and which team owns it.
Network segmentation. The collector-to-backend link should not traverse the public network. The production pattern is a dedicated observability VLAN or a private subnet with security groups limited to the relevant ports.

Performance implications

Hop count is a budget. Each hop adds CPU, latency, and a place to lose data. The topology should minimise hops for the hot path. The exemplar (workload -> agent -> gateway -> backend) is the minimum that gives independent failure domains.
Cross-AZ traffic is expensive. A central gateway that receives from every AZ and ships to every backend adds cross-AZ bytes. Topology-aware routing (agents in the same AZ as the gateway, gateway in the same AZ as the backend) is the optimisation.
Cardinality multiplies at every hop. A label added by the collector travels to the backend and into the TSDB. Topology decisions that look free (“let’s add a region label at the gateway”) can cost real money at the storage layer.

Production guidance

Document the diagram. One diagram in the runbook repo, one per environment, reviewed when any hop changes.
Own each layer. Name a team and a primary on-call for each layer. Unowned layers rot first.
Size the queue, not the drop. The right answer to back- pressure is a longer sending_queue with disk-backed overflow, not aggressive dropping. Dropping is the failure shape; queueing is the topology decision.
Treat config as code. Every layer’s configuration lives in git, goes through CI, and deploys by pull-request merge. No click-ops on the dashboards of any component.

Verification

You should now be able to answer:

What are the five layers of the production observability topology, and what does each layer own?
Which component fails when the symptom is “traces go dark for one subnet but metrics keep flowing”?
Why is “one OTel Collector per host” sometimes correct and sometimes wrong?
What is the canonical scope of work for each layer: who scrapes, who queries, who alerts?