Reference Architecture Anatomy · Observability

At 03:12 an on-call engineer is staring at a wall of green panels. CPU is fine. Memory is fine. Disk is fine. The payment service is returning 503s for 30% of requests, and the only signal that says so is the user-facing error-rate panel. The cards behind that panel are noise. The architecture that produced them is the problem, not the data.

A reference architecture is not a brand. It is a set of layer contracts. When the contracts are clear, every component has a reason to exist and a way to fail loudly. When the contracts are fuzzy, the components collapse into each other, the system becomes hard to reason about, and the on-call engineer ends up where the engineer above started.

What it is

A reference architecture for observability is a documented arrangement of five layers — workload, collector, backend, presentation, and operations — with a single clearly-named contract between each adjacent pair. The canonical stack covered by this course is Prometheus 2.55.x for metrics, Loki 3.x for logs, Tempo for traces, Grafana 11.x for presentation, and Grafana Alloy or the OpenTelemetry Collector 0.110.x for the collector layer. Alertmanager is treated as a backend-adjacent component that owns the delivery contract.

The reference architecture is not a finished product. It is a template. The right selection for a given environment depends on scale, security boundary, and team ownership. The wrong selection is “we downloaded everything, wired it up, and called it a platform.”

Why a sysadmin cares

Production observability is the property that lets an operator answer the question “what is happening right now?” from telemetry without shipping new code. The reference architecture exists to make that property durable. A team that adopts the architecture will not have to re-design it every time a new service joins the fleet. A team that ignores it will.

The four failure shapes that show up in stacks without a reference architecture:

The monolith collector. One OTel Collector accepts every workload and forwards to every backend. When it dies, every signal dies. When it backpressures, every signal stalls.
The bespoke exporter. Every team writes its own scrape format. Dashboards become archaeology.
The orphaned dashboard. A Grafana instance exists. The dashboards do not have owners. The on-call engineer opens one and finds a query that no longer parses.
The alert that pages without context. The alert manager fires. The receiving engineer does not know which service, which host, or which dependency is implicated. The page is paid for to start the investigation.

The reference architecture does not prevent all of these. It makes each one recognisable and gives the operator a vocabulary to talk about it.

How it works

The five layers with the contract at each boundary:

  +------------+      +-----------+      +---------+      +-----------+
  | WORKLOAD   |      | COLLECTOR |      | BACKEND |      | PRESENTATION
  |            |      |           |      |         |      |
  | Apps       | ---> |           | ---> |         | ---> |
  | Containers |      |           |      |         |      |
  | Hosts      |      |           |      |         |      |
  +------------+      +-----------+      +---------+      +-----------+
        |                   |                |                  |
        | instrumentation   | telemetry       | query            | view
        | endpoint          | shipping        |                  | + alert
        |                   |                 |                  |
        v                   v                 v                  v
  /metrics, /logs,    OTLP, Prometheus   Prometheus, Loki,   Grafana,
  OTLP gRPC/HTTP     remote_write,       Tempo, Mimir        Alertmanager
  (PUSH or PULL)     syslog, beats      (PULL over HTTP)    (PUSH)

Each layer has a single responsibility:

Workload — produce telemetry. The rest of the architecture is downstream.
Collector — receive, transform, batch, attribute, and forward. The collector is the only place that knows the topology of the backends.
Backend — store one signal well. Prometheus for metrics, Loki for logs, Tempo for traces. (Mimir is the horizontally-scalable Prometheus-compatible choice.)
Presentation — render, alert, and annotate. Grafana and Alertmanager.
Operations — the people, the runbooks, and the change discipline that turn the other four into something a team can actually run.

The contracts at each boundary are where the architecture breaks or holds:

Workload → collector. HTTP pull (Prometheus text format on /metrics) or OTLP push (gRPC or HTTP, the modern default).
Collector → backend. Prometheus remote_write for metrics, Loki HTTP push for logs, OTLP for traces.
Backend → presentation. Prometheus HTTP query API, Loki query language (LogQL), Tempo query API.
Presentation → operations. Grafana dashboards, Alertmanager notifications, SLO reports.

Under the hood

What the production stack actually looks like in files and processes:

  +----------------------- HOST A (application) ----------------------+
  |                                                                   |
  |  app PID 1042                                                    |
  |    emits OTLP spans over UDS to 127.0.0.1:4317                    |
  |    exposes :9100 /metrics via prometheus client_golang            |
  |    writes JSON logs to stdout                                     |
  |                                                                   |
  |  node_exporter PID 1051                                           |
  |    exposes :9100/metrics (host)                                   |
  |                                                                   |
  |  alloy PID 1060 (Grafana Alloy, agent mode)                      |
  |    receives OTLP :4317                                            |
  |    scrapes :9100/metrics for host and app                         |
  |    tails /var/log/containers/app.log                              |
  |    forwards to Prometheus remote_write, Loki push, Tempo OTLP     |
  |                                                                   |
  +-------------------------------------------------------------------+

  +--------------------- BACKEND NODES (separate) --------------------+
  |                                                                   |
  |  prometheus (or mimir)        :9090   query, /api/v1/write       |
  |  loki                         :3100   distributor, /loki/api     |
  |  tempo                        :3200   /api/traces, /api/search   |
  |  alertmanager                 :9093   /api/v2/alerts, /api/v2/silences
  |                                                                   |
  +-------------------------------------------------------------------+

  +----------------- PRESENTATION (Grafana 11.x) --------------------+
  |                                                                   |
  |  grafana                     :3000   provisioned dashboards,     |
  |                                       provisioned datasources,    |
  |                                       unified alerting            |
  |                                                                   |
  +-------------------------------------------------------------------+

The Alloy process on each host is the agent in agent topology. An alternative is gateway topology, where a small agent on the host batches telemetry and forwards to a central collector tier that fans out to the backends. Gateway topology centralises the routing config; agent topology distributes the failure domain.

How to configure it

The minimum viable reference architecture per host is one Grafana Alloy config that ships all three signals to the canonical backends. Real annotated example:

// /etc/alloy/config.alloy
// Agent-mode Grafana Alloy. One per host.

prometheus.scrape "host_metrics" {
  targets = [{
    __address__ = "localhost:9100",
    job         = "node",
  }]
  forward_to      = [prometheus.relabel.host_metrics.receiver]
  scrape_interval = "15s"
  job_name        = "node"
}

prometheus.scrape "app_metrics" {
  targets = [{
    __address__ = "localhost:9100",
    job         = "app",
    __metrics_path__ = "/metrics",
  }]
  forward_to      = [prometheus.relabel.app_metrics.receiver]
  scrape_interval = "15s"
  job_name        = "app"
}

prometheus.relabel "host_metrics" {
  forward_to = [prometheus.remote_write.default.receiver]
  rule {
    target_label = "instance"
    replacement  = constants.hostname
  }
  rule {
    target_label = "cluster"
    replacement  = "prod-eu-west-1"
  }
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://prometheus.monitoring.svc:9090/api/v1/write"
    basic_auth {
      username = "alloy"
      password = sys.env("PROM_REMOTE_WRITE_PASSWORD")
    }
  }
}

loki.source.file "container_logs" {
  targets    = discovery.relabel.container_logs.targets
  forward_to = [loki.process.tenant_a.receiver]
}

loki.process "tenant_a" {
  forward_to = [loki.write.default.receiver]
  stage.labels {
    values = { "cluster" = "prod-eu-west-1", "app" = "" }
  }
}

loki.write "default" {
  endpoint {
    url       = "http://loki.monitoring.svc:3100/loki/api/v1/push"
    tenant_id = "tenant-a"
  }
}

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.monitoring.svc:4317"
    tls {
      insecure = true
    }
  }
}

discovery.relabel "container_logs" {
  targets = [
    {__path__ = "/var/log/containers/*.log", "container" = "", "namespace" = "", "pod" = ""},
  ]
}

The same topology in OpenTelemetry Collector form uses the receivers, processors, exporters stanza; the routing contract is identical.

The backend side, server-side, is configured via the prometheus.yml, loki-config.yaml, and Tempo config files. The collector is the right place to enforce cardinality caps and attribution labels because the collector is the only component that sees every host.

How to validate it

CONFIGURATION — validate the configs before reload.

promtool check config /etc/prometheus/prometheus.yml
# SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config

promtool check rules /etc/prometheus/rules/*.yml
# SUCCESS: /etc/prometheus/rules/services.yml is valid

alloy fmt /etc/alloy/config.alloy > /tmp/alloy.restyled
diff /etc/alloy/config.alloy /tmp/alloy.restyled
# alloy fmt normalises formatting; diff before applying

loki -config.file=/etc/loki/loki-config.yaml -verify-config
# (Loki 3.x: process exits 0 with "config valid" on success)

tempo -config.file=/etc/tempo/tempo.yaml -verify-config
# (Tempo: process exits 0 with "config valid")

READ-ONLY — confirm the stack is live.

curl -sf http://prometheus.monitoring.svc:9090/-/ready
# Prometheus is Ready.

curl -sf http://loki.monitoring.svc:3100/ready
# ready

curl -sf http://tempo.monitoring.svc:3200/ready
# ready

curl -sf http://alertmanager.monitoring.svc:9093/-/ready
# Alertmanager is ready.

READ-ONLY — confirm an end-to-end trace is arriving.

curl -sf "http://tempo.monitoring.svc:3200/api/search?tags=service.name%3Dcheckout&limit=1" | jq .
# { "traces": [{ "traceID": "8f4ab1e3c2d9...", "rootServiceName": "checkout" }] }

READ-ONLY — confirm a metric is arriving from the expected collector.

curl -sf 'http://prometheus.monitoring.svc:9090/api/v1/query?query=up{job="node"}' | jq '.data.result[0]'
# { "metric": { "job": "node", "instance": "host-01.prod" },
#   "value": [1723651200, "1"] }

READ-ONLY — confirm logs are queryable from the host where the incident is reported.

logcli --addr=http://loki.monitoring.svc:3100 \
  query '{cluster="prod-eu-west-1", app="checkout"} |~ "error"'
# 2026-08-13T03:12:14Z {cluster="prod-eu-west-1", app="checkout"} \
#   level=error msg="payment timeout" trace_id=8f4ab1e3c2d9 ...

How it can fail

The failure modes that the reference architecture makes recognisable. Each one has a recognisable symptom.

Cardinality explosion at the collector. A relabel rule injects a label with unbounded values (request UUID, user ID, raw pod name). Prometheus memory rises steadily. Symptom: OOM of the Prometheus process, or prometheus_tsdb_head_series diverging from forecast.
Mis-routed signal. A collector forwards to the wrong backend. Logs intended for the EU cluster end up in the US region. Symptom: query returns empty for a known-firing service.
Contract drift between versions. OTel Collector 0.110.x drops a processor that the upgrade target renamed. The pipeline starts on boot but emits no data. Symptom: scrapes succeed but up == 0 for every target.
Backpressure from a slow backend. The collector buffers fills. The exporter drops batches. Symptom: gaps in metric series, longer trace latencies, queues visible in the otelcol_exporter_queue_size metric.
Stale scrapes behind a load balancer. The collector connects to a L7 LB that drops connections from the collector’s host. Symptom: intermittent scrape failures, then steady at 0.
Dashboard renders against a missing datasource UID. The datasource is renamed. Every panel with a hard-coded UID shows “datasource not found.” Symptom: dashboards that “work” but show no data.

How to troubleshoot it

The diagnostic order for “the stack is not producing telemetry I expect”:

Was it working before? Recent change in any layer. If not, the architecture is the suspect only if the architecture was recently changed.
Read the service view. Each component exposes its own health endpoint. /-/ready is the canonical first check.
Read the platform view. up in Prometheus, queue sizes in the collector, query latency in the backends. The health of the platform is the health of the pipeline.
Trace from the destination backward. If the dashboard is empty, the query is the first place to look. If the query returns data, the panel is the next place. If the panel renders nothing, the datasource is the issue. If the datasource is healthy, the collector is the issue. If the collector is healthy, the workload is the issue.
Form a hypothesis. Find evidence. Test. Validate. Pick one of the failure modes above and check the symptom. Do not iterate through them all.

Security implications

The reference architecture exposes ten HTTP endpoints in the canonical layout. Each one needs a security policy:

node_exporter :9100 is host-local. Bind to the loopback unless the network is trusted.
Collector receivers accept OTLP from workloads. mTLS is the correct choice in production. The default of “plaintext on the cluster network” is a posture, not a security control.
prometheus /api/v1/write accepts remote_write from collectors. Authentication is mandatory. The course recommends basic_auth with a credential retrieved from a secret manager, not a static file.
loki /api/v1/push and Tempo OTLP are similar.
Grafana has its own RBAC, but the signal backends should not be readable from the public internet. Network policy or ingress allowlist is the rule.
Alertmanager notification integrations (Slack, PagerDuty, email) hold credentials. The secret store is the authoritative source.

The security chapter of the course covers all of these in detail. The right mental model for the reference architecture is: every component is a service that needs its own identity, its own network policy, and its own audit trail.

Performance implications

Performance is dominated by five knobs:

Cardinality. The number of unique label combinations stored. Cardinality explosion is the most common way an observability stack destroys itself. The collector is the place to enforce it.
Scrape / push interval. 15s is the canonical default. 5s is the floor for most workloads. 60s is acceptable for long-tail metrics. Pushing to 1s raises cost by an order of magnitude.
Retention. Prometheus: 15 days default, 30 days for production. Loki: 30 days index, 365 days for compliance. Tempo: 7 days is common, 14 days for clusters with slower feedback cycles.
Compression. Loki and Tempo both have storage compression knobs. The defaults are reasonable; raising them trades CPU for disk.
Query cost. Grafana panels that re-evaluate every 5s against a Loki range query can destroy the backend. The right pattern is max_over_time aggregations on the backend side, then a fast query on the dashboard side.

Production guidance

Validate before applying. promtool check config, alloy fmt, loki -verify-config. Each of these is fast and catches the most common class of mistakes.
Test in non-production. A stack that works in staging with one host and one scrape target is not the same stack that works in production with five hundred. Test the full topology.
Provision the dashboards and datasources. Hand-edited Grafana is a liability. The right pattern is YAML in a repository, scp or kubectl apply to deploy.
Run a blackbox exporter. The four exporters described above (node, cAdvisor, app, alloy) all describe the system from inside. A blackbox exporter (HTTP, TCP, ICMP) describes it from outside. Both views are needed.
Trace exemplars into metrics. The metric that says “checkout latency p99” should have an exemplar link to a span. The on-call engineer then clicks from the metric to a trace without typing a trace ID.

Verification

You should now be able to answer:

What are the five layers of the canonical reference architecture, and what is the responsibility of each?
What is the contract at the boundary between the workload and the collector?
What is the difference between agent and gateway collector topology, and when is each one right?
Why is the collector the right place to enforce cardinality caps?
What is the failure mode when a dashboard references a datasource UID that no longer exists?