Reference Architecture

A team adopts the reference observability stack: agents on hosts, a gateway in the middle, Prometheus + Loki + Tempo at the back, Grafana at the front. They wire it together with mTLS, push the configuration through CI, and stand up Alertmanager with a real receiver. Six months in, the team grows from 50 to 250 hosts. The gateway becomes the bottleneck. The Loki ingester starts dropping chunks. The reference architecture does not scale by magic; it scales by deliberate choices that map to the workload shape.

The reference architecture is the set of those choices, in one diagram, with sizing rules per environment. It is not a recipe to be followed verbatim; it is a template that the operator adapts to the fleet.

What the reference architecture is

The reference architecture is the canonical production observability topology, sized for a small-to-medium fleet, ready to scale to a large fleet without redesign. It is the sum of the lessons in Part LXVI drawn on a single canvas.

                    +----------------------------+
                    |   Git repository (config)  |
                    |   per-environment overlays |
                    +-------------+--------------+
                                  |  CI/CD
                                  v
   +-------------+      +---------+----------+      +-------------+
   |  Workloads  |----->|  OTel Collector     |----->|  Prometheus |
   |  (hosts,    |      |  agent (per host)   |      |  TSDB local |
   |  containers,|      +---------+----------+      |  + remote    |
   |  services)  |                |                 |  write       |
   +-------------+                |                 +-------------+
                                  v
                          +-------+--------+
                          |  OTel Collector |
                          |  gateway (3x)   |
                          +---+----+----+---+
                              |    |    |
                              v    |    v
              +---------------+ | +--+--------------+
              |               | | |                 |
        +-----v-----+   +-----v-v-v+    +-----------v---+
        |   Loki    |   |   Tempo     |  |  Alertmanager  |
        |  (3x ingester, |  (3x ingester, |  (2x)        |
        |  S3 chunks)   |  S3 blocks)    |               |
        +-------------+   +--------------+   +-----+------+
              |                |                  |
              +-------+--------+                  |
                      v                           v
                 +----+----+                +------+-------+
                 | Grafana |                |  Receivers   |
                 | (2x HA) |<-------------->| (Slack, PD)  |
                 +---------+                +--------------+

The picture has six layers, each owned by a single team.

• Workload layer. Linux hosts, Docker containers, and application services. Owns signal production.
• Collection layer (agent). One OTel Collector per host (or per pod). Owns host-local sources, per-host batching, the queue that absorbs gateway outages.
• Collection layer (gateway). A central OTel Collector behind a load balancer. Owns fleet-wide processing: tail sampling, PII redaction, the final batch before fan-out.
• Storage layer. Prometheus (TSDB on NVMe, remote_write to Mimir/Thanos), Loki (TSDB index, chunks in S3), Tempo (blocks in S3). Owns the long-term home of every signal.
• Alerting layer. Alertmanager with grouped, routed alerts to PagerDuty, Slack, and the on-call rotation.
• Presentation layer. Grafana with provisioned data sources, provisioned dashboards, and RBAC wired into the identity provider.

Beneath all six layers, two cross-cutting concerns:

• Configuration-as-code. The git repository holds every configuration file. CI applies the per-environment overlay. A pull request merge deploys.
• Secret rotation. The secret store (Vault or a cloud equivalent) holds every credential. Certificates rotate every 60 days; tokens rotate every 30 days.

Why a sysadmin cares

The reference architecture is what the postmortem looks like when the team has it and what the postmortem looks like when the team does not.

1. The team that has the reference. A 02:00 incident. The on-call opens Grafana, follows the trace to the database host metrics, confirms the disk pressure, follows the runbook, escalates to the storage team. Total time-to-resolution: 22 minutes. The reference architecture let the on-call know where to look without rediscovering the topology.
2. The team that does not. A 02:00 incident. The on-call opens Grafana, sees a green dashboard, opens another dashboard, sees another green panel, opens Slack, asks “where do we look for X?” Three people spend 40 minutes answering the question. Total time-to-resolution: 90 minutes. The topology is implicit; the cost is the implicit work the on-call has to do.
3. The team that has the reference but grew past it. A 6-month-in team grows from 50 to 250 hosts. The gateway becomes the bottleneck. The Loki ingester starts dropping. The reference architecture is in the runbook repo but the sizing is wrong. The fix is not a topology redesign; the fix is gateway horizontal scaling and an ingester resize.

The reference architecture is the canvas on which growth is planned. A team that has the canvas plans growth by adding boxes; a team that does not plans growth by arguing about which box is which.

How it works

Each component in the reference plays one role. The mental model is “every box does one thing, every hop has one protocol, every layer has one owner.”

The data plane

A metric scrape: workload exposes /metrics; the OTel agent or node_exporter serves it; Prometheus pulls on scrape_interval: 15s; samples land in the head block; rules evaluate; alerts fire into Alertmanager.

A log line: the OTel filelog receiver tails the file; the agent batches and pushes to the gateway; the gateway fans out to Loki; Loki indexes the labels and stores the chunk in S3.

A trace span: the application SDK exports OTLP/gRPC to the agent on 127.0.0.1:4317; the agent forwards to the gateway on the cluster network; the gateway applies the tail-sampling and PII-redaction processors; the gateway fans out to Tempo; Tempo writes the block to S3.

The control plane

Configuration-as-code: the git repository holds prometheus.yml, the OTel agent and gateway configs, the Loki schema config, the Tempo storage config, the Grafana data sources and dashboards, and the Alertmanager routing tree. A pull request to any of these files triggers CI: promtool check config, otelcol validate, logcli config check, tempo-cli config validate, and a Grafana provisioning dry-run. A merge deploys.

Secret rotation: Vault issues short-lived certificates and tokens. The OTel Collector reads the client cert and key from Vault at startup and refreshes on a schedule. The Grafana data source reads the bearer token from Vault at startup. Rotation is a Vault policy; the application never sees the secret directly.

Per-environment sizing

The same topology, sized three ways:

The rule of thumb: the gateway is the first thing to scale horizontally; the storage backends scale by adding ingesters and increasing replication factor; the presentation layer is the last to scale because Grafana is cheap.

How to configure it

The reference architecture does not have one configuration file; it has a directory of configuration files, one per component, with per-environment overlays.

/observability-config/
  base/
    prometheus.yml
    otel-agent.yaml
    otel-gateway.yaml
    loki-config.yaml
    tempo-config.yaml
    alertmanager.yml
    datasources.yaml
    dashboards/
      checkout-service.json
  environments/
    dev.yaml
    staging.yaml
    prod-eu-west-1.yaml
    prod-us-east-1.yaml

The base/ directory holds the canonical configuration. The environments/ directory holds the per-environment overlay. CI applies the overlay on the top of the base to produce the deployed configuration.

A representative per-environment overlay:

# environments/prod-eu-west-1.yaml
prometheus:
  external_labels:
    cluster: prod-eu-west-1
    env: production
  retention:
    time: 30d
    size: 200GB

otel_gateway:
  replicas: 3
  memory_ballast: 2GB

loki:
  schema:
    from: 2026-01-01
    store: tsdb
    object_store: s3
  storage:
    s3:
      bucket: loki-prod-eu-west-1
      region: eu-west-1
  retention:
    period: 744h

tempo:
  storage:
    s3:
      bucket: tempo-prod-eu-west-1
      region: eu-west-1
  retention:
    block_retention: 744h

alertmanager:
  receivers:
    - name: pagerduty-prod
      pagerduty_configs:
        - service_key_file: /etc/alertmanager/secrets/pagerduty-prod.key

The overlay names the environment-specific values: external labels, replica counts, retention periods, bucket names, and the receiver configuration. Everything else is inherited from base/.

The CI pipeline that validates the rendered configuration:

# CONFIGURATION: render and validate the overlay for one environment.
yq eval-all '. as $item ireduce ({}; . * $item)' \
  base/prometheus.yml environments/prod-eu-west-1.yaml \
  | promtool check config /dev/stdin

# CONFIGURATION: validate the OTel Collector config.
yq eval-all '. as $item ireduce ({}; . * $item)' \
  base/otel-gateway.yaml environments/prod-eu-west-1.yaml \
  | otelcol-contrib validate --config=/dev/stdin

# CONFIGURATION: validate the Loki config.
yq eval-all '. as $item ireduce ({}; . * $item)' \
  base/loki-config.yaml environments/prod-eu-west-1.yaml \
  | logcli config check --config=/dev/stdin

The overlay render is a deliberate step. Without it, the per-environment differences live in the operator’s head and the configuration-as-code promise is broken.

How to validate it

# READ-ONLY: every layer is ready.
for endpoint in \
  prometheus:9090/-/ready \
  loki:3100/ready \
  tempo:3200/ready \
  alertmanager:9093/-/ready \
  grafana:3000/api/health ; do
    curl -fsS http://$endpoint || echo "FAIL: $endpoint"
done

# READ-ONLY: the agent can reach the gateway.
curl -fsS http://otel-agent:8888/metrics | grep exporter_sent_spans

# READ-ONLY: the gateway can reach the backends.
for backend in loki tempo prometheus ; do
  curl -fsS http://otel-gateway:8889/metrics \
    | grep "exporter_sent.*$backend" || echo "FAIL: $backend"
done

# READ-ONLY: the data source health checks pass in Grafana.
for ds in prometheus loki tempo ; do
  uid=$(curl -fsS -u admin:admin http://grafana:3000/api/datasources/name/$ds \
    | jq -r .uid)
  curl -fsS -u admin:admin \
    http://grafana:3000/api/datasources/uid/$uid/health \
    || echo "FAIL: $ds"
done

# CONFIGURATION: a synthetic trace, log, and metric round-trip.
# (Application-level; the SDK does the work.)
./integration-test.sh prod-eu-west-1

A clean validation: every layer is /ready, the agent’s exports are monotonically increasing, the gateway’s per-backend exports match, every Grafana data source is healthy, and the synthetic round-trip returns within the expected latency.

How it can fail

The reference architecture fails as a system when one layer starves another or when a boundary is misconfigured. The most expensive system-level failure modes:

1. Gateway becomes the bottleneck. The gateway ingests 80,000 spans/s from 250 agents and ships to three backends. The gateway memory_limiter kicks in; the sending_queue fills; agents’ queues fill; the SDKs start dropping. Symptom: every dashboard for the noisiest service goes dark; otelcol_processor_refused climbs; an incident review declares “tracing is unreliable.”
2. Loki ingester drops chunks. Loki ingester memory is exhausted; the WAL fills; new chunks are rejected. Symptom: loki_ingester_wal_bytes at 95% of disk; loki_request_duration_seconds for the write path spikes; queries for the last 10 minutes return empty.
3. Prometheus remote_write queue overflow. Prometheus remote_write to Mimir falls behind; the in-process queue fills past its capacity; samples are dropped at the sender. Symptom: prometheus_remote_storage_failed_samples climbs; Mimir shows gaps for the affected series.
4. Grafana data source mis-pointed. A Grafana provisioning merge deploys the staging URL as the production data source. Symptom: production dashboards render staging data; the alert fires from a staging spike; a customer escalation follows.
5. Alertmanager wired to one of two Prometheus replicas. A kill -9 of the primary Prometheus silences all alerting until the next Alertmanager config reload picks up the replica. Symptom: every panel is green for an hour but no alerts fire.
6. Secret store outage. Vault becomes unreachable; the OTel Collector and Grafana cannot refresh their credentials; the existing credentials expire or the SDK refuses to start. Symptom: telemetry stops flowing because no collector can authenticate to the backends.

How to troubleshoot it

The diagnostic order for the reference architecture is “is each layer up?”, “is each layer forwarding?”, “is each boundary correct?”.

1. Start at the top. Grafana’s data source health checks confirm the storage layer is reachable from Grafana.
2. Check the storage layer. Each backend’s /ready and its write-path metrics. A non-ready storage layer is the bottleneck; the queue at the layer above absorbs it.
3. Check the gateway. otelcol_exporter_sent_* per backend. A flat metric means the gateway is not exporting; the gateway logs tell you why.
4. Check the agents. From each host, curl http://127.0.0.1:8888/metrics | grep exporter_sent_spans. A flat metric on the agent means the agent is not exporting; the agent logs tell you why.
5. Check the workloads. From each host, curl http://127.0.0.1:9100/metrics | head for node_exporter; check the OTel SDK environment variables for the application. A failed scrape here is a workload-layer problem.
6. Reproduce at the lowest layer first. A missing trace: confirm the SDK is configured, the agent’s receiver is bound, the gateway is reachable, Tempo accepts the export. Walk up from the bottom.

Security implications

• Every boundary has an answer. The host boundary is the trust boundary (no protocol authentication needed); the cluster-network boundary is mTLS; the public-network boundary is TLS at the reverse proxy. A team that removes a boundary by accident makes the architecture weaker at that boundary.
• mTLS on every collector hop. The agent-to-gateway hop and the gateway-to-backend hop are both mTLS. Bearer tokens are an acceptable alternative but they are a secret that must be rotated.
• The secret store is part of the topology. Vault or a cloud equivalent issues certificates and tokens; the application never sees the secret directly. The secret store’s availability is a dependency of every layer that authenticates.
• Per-environment secrets. Production credentials never appear in a non-production overlay. The CI pipeline asserts that a secret path tagged prod is never referenced by a non-prod overlay.

Performance implications

• The gateway is the first thing to scale. Adding a fourth gateway replica halves the per-replica load. The storage backends scale by adding ingesters; the presentation layer is the last to scale.
• Cardinality multiplies at every tier. A label added at the agent travels to the gateway, to the backends, and into the TSDB. The reference architecture does not fix cardinality; it amplifies the cost of a bad label.
• Cross-AZ bytes are expensive. A gateway that fans out across AZ boundaries pays the cross-AZ bandwidth cost on every export. Topology-aware routing keeps the gateway in the same AZ as the backends it serves.

Production guidance

• Adopt the reference architecture on day one. A team that adopts it later pays the migration cost in misconfiguration incidents. A team that adopts it on day one pays the cost in a single afternoon of diagramming.
• Treat the diagram as code. The reference architecture lives in the runbook repository. A pull request that adds a box or moves a hop gets reviewed by every team that owns a layer.
• Size per environment, not per fleet. Dev, staging, and production share the topology; they differ in the sizing overlay. The dev overlay is a single replica of everything; the production overlay is three replicas and full retention.
• Plan growth by adding boxes, not by arguing. When the fleet grows past the current sizing, the answer is one more gateway, one more ingester, one more Grafana. The topology does not change.

Verification

You should now be able to answer:

• Which component is the first to scale horizontally when the fleet grows, and why?
• What is the right shape for per-environment sizing in the configuration-as-code repository?
• Where does the secret store belong in the topology, and what happens when it is unreachable?
• Why is the reference architecture a picture of a healthy platform rather than a recipe to follow verbatim?

Quiz