Lab: Capstone Production Observability Estate

Objective

By the end of this lab you will have the five layers of the reference architecture running on one host, carrying two environments through a single Loki with an enforced tenant boundary, and you will have proved every hop with a command that crosses it. Then you will break the estate five times on purpose. For each break you write down what you expect to see before you look, and afterwards you record which layer noticed. Two of the five produce no symptom anywhere in the platform. Finding out which two is the point of the lab.

Architecture

Nine containers, five layers, one host. Every arrow is a hop you will later cut.

  WORKLOAD        checkout-prod              checkout-staging
                  :8000 /metrics             :8000 /metrics
                  /var/log/app/*.log         /var/log/app/*.log
                       |     |                    |     |
        logs ----------+     |       logs --------+     |
                       v     |                    v     |
  COLLECTION      otel-agent-prod            otel-agent-staging
  (agent)         filelog -> otlp            filelog -> otlp
                       |                          |
                       | :4317                    | :14317
                       v                          v
  COLLECTION      +------------------------------------------+
  (gateway)       |            otel-gateway                  |
                  |  logs/prod  -> Loki tenant "prod"        |
                  |  logs/staging -> Loki tenant "staging"   |
                  |  traces/prod  -> Tempo                   |
                  +--------+--------------------+------------+
                           |                    |
  STORAGE            +-----v----+          +----v----+     +------------+
                     |   Loki   |          |  Tempo  |     | Prometheus |
                     | 2 tenants|          | 1 tenant|     |   TSDB     |
                     +-----+----+          +----+----+     +-----+------+
                           |                    |            |    | scrape
                           |                    |            |    +-----> everything
  ALERTING                 |                    |            v
                           |                    |        +---------------+
                           |                    |        | Alertmanager  |
                           |                    |        +-------+-------+
                           |                    |                |
  PRESENTATION        +----v--------------------v----------------v----+
                      |                  Grafana                      |
                      |  prom-estate / loki-prod / loki-staging /     |
                      |  tempo-estate                                 |
                      +-----------------------------------------------+

The structural decision worth staring at: metrics and logs travel different paths. Prometheus pulls from the workload directly; logs push through two collector tiers. That is not an accident of this lab, it is how a Prometheus-based estate is shaped, and it is why the first drill produces a dashboard that is entirely green while half the telemetry is on the floor.

Requirements

• A disposable Linux host with Docker Engine 28.x and Compose v2, 4 GB RAM and 10 GB free disk. The lab creates nine containers and six named volumes.
• curl and jq.
• Ports 3000, 3100, 3200, 8888, 8889, 8898, 9090, 9093 and 4318 free on loopback. Task 1 checks. Every published port binds 127.0.0.1, so nothing in this lab is reachable from the network.
• No out-of-band access requirement. Nothing here touches SSH, the host firewall, or the primary interface.
• Roughly 45 minutes of wall clock inside the 180. Several checks use [5m] rate windows, and a half-filled window gives numbers that move while you read them.

Scenario

You have inherited a platform that grew one component at a time. Prometheus was first, then somebody added Loki for a specific incident, then a Tempo that two people know how to query. Each piece works. Nobody has ever drawn the whole thing, nobody can say what happens when one piece stops, and the last two incidents both included the sentence “but the dashboard was green”.

The remit is not to add a component. It is to build the estate once, from files, with the environment boundary made explicit — and then to find out, before an incident does, which failures this platform can report about itself.

Tasks

Task 1: Capture the starting state

LAB="$HOME/rb-obs-capstone"
mkdir -p "$LAB"/{prometheus/rules,alertmanager,loki,tempo,otel,grafana/provisioning/datasources,gen}
cd "$LAB"

# Every port this lab publishes must be free on loopback.
for p in 3000 3100 3200 8888 8889 8898 9090 9093 4318; do
  if ss -ltn "sport = :$p" 2>/dev/null | grep -q LISTEN; then
    echo "PORT $p IS IN USE - resolve before continuing"
  fi
done | tee ports.pre-lab

docker ps --format '{{.Names}}' | tee containers.pre-lab

A port conflict here surfaces four tasks later as a container that will not start, with an error from the docker-proxy layer that names the port and not the cause. Two seconds now.

Task 2: Write the workload

gen/gen.py — one image, two environments, driven entirely by environment variables. It emits RED metrics on :8000 and writes one JSON log line per request to a file the agent will tail:

#!/usr/bin/env python3
"""Synthetic service for the capstone estate.

Emits the two RED series a service dashboard needs (a request counter labelled
by status, and a duration histogram) and writes a matching JSON log line per
request. Nothing here is measured from real work - the point is a telemetry
source with a knob for the error rate, not a realistic application.
"""
import json
import os
import random
import time

from prometheus_client import Counter, Histogram, start_http_server

SERVICE = os.environ.get("SERVICE", "checkout")
ENVIRONMENT = os.environ.get("ENVIRONMENT", "prod")
ROUTE = os.environ.get("ROUTE", "/checkout")
RATE = float(os.environ.get("RATE", "20"))          # requests per second
FAIL_RATIO = float(os.environ.get("FAIL_RATIO", "0.005"))
LOG_PATH = os.environ.get("LOG_PATH", "/var/log/app/app.log")

# The `env` label is NOT emitted here. Prometheus attaches it as a target
# label in Task 4, and a metric that carries its own `env` would collide:
# with honor_labels off (the default) the target label wins and the exposed
# one is renamed to `exported_env`, which is confusing rather than wrong.
REQUESTS = Counter(
    "http_server_requests_total",
    "Requests handled, by outcome.",
    labelnames=("service", "route", "status"),
)
DURATION = Histogram(
    "http_server_request_duration_seconds",
    "Time spent handling a request.",
    labelnames=("service", "route"),
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
)


def main():
    start_http_server(8000)
    interval = 1.0 / RATE
    with open(LOG_PATH, "a", buffering=1) as log:
        while True:
            failed = random.random() < FAIL_RATIO
            status = "500" if failed else "200"
            seconds = random.uniform(0.4, 1.8) if failed else random.uniform(0.02, 0.2)

            REQUESTS.labels(SERVICE, ROUTE, status).inc()
            DURATION.labels(SERVICE, ROUTE).observe(seconds)

            log.write(json.dumps({
                "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
                "level": "error" if failed else "info",
                "service": SERVICE,
                "env": ENVIRONMENT,
                "route": ROUTE,
                "status": status,
                "duration_ms": round(seconds * 1000, 1),
                "msg": "request failed" if failed else "request served",
            }) + "\n")

            time.sleep(interval)


if __name__ == "__main__":
    main()

gen/Dockerfile:

FROM python:3.12-slim
RUN pip install --no-cache-dir prometheus_client==0.21.0
COPY gen.py /app/gen.py
CMD ["python", "-u", "/app/gen.py"]

Task 3: Write the collection tier

Two configurations, and the difference between them is the collector strategy. The agent owns one host-local source and one destination. The gateway owns everything that needs context the agent does not have.

otel/agent.yaml — the same file serves both agents; the include path and the gateway endpoint come from environment variables the Compose file sets:

# otel/agent.yaml
receivers:
  filelog:
    include: ['${env:LOG_GLOB}']
    start_at: beginning
    operators:
      - type: json_parser

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    timeout: 5s
    send_batch_size: 1024
  resource:
    attributes:
      - key: service.name
        value: '${env:SERVICE_NAME}'
        action: upsert
      - key: deployment.environment
        value: '${env:ENVIRONMENT}'
        action: upsert

exporters:
  otlp/gateway:
    endpoint: '${env:GATEWAY_ENDPOINT}'
    tls:
      insecure: true
    sending_queue:
      enabled: true
      queue_size: 5000

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/gateway]

otel/gateway.yaml — one receiver per tenant, one exporter per tenant, and the redaction that has to live in exactly one place:

# otel/gateway.yaml
receivers:
  otlp/prod:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
  otlp/staging:
    protocols:
      grpc: { endpoint: 0.0.0.0:14317 }

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    timeout: 5s
    send_batch_size: 4096
  attributes/redact:
    actions:
      - key: user.email
        action: delete
      - key: http.request.header.authorization
        action: delete

exporters:
  otlphttp/loki-prod:
    endpoint: http://loki:3100/otlp
    headers:
      X-Scope-OrgID: prod
  otlphttp/loki-staging:
    endpoint: http://loki:3100/otlp
    headers:
      X-Scope-OrgID: staging
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  pipelines:
    logs/prod:
      receivers: [otlp/prod]
      processors: [memory_limiter, attributes/redact, batch]
      exporters: [otlphttp/loki-prod]
    logs/staging:
      receivers: [otlp/staging]
      processors: [memory_limiter, attributes/redact, batch]
      exporters: [otlphttp/loki-staging]
    traces/prod:
      receivers: [otlp/prod]
      processors: [memory_limiter, attributes/redact, batch]
      exporters: [otlp/tempo]

Task 4: Write the storage tier

prometheus/prometheus.yml — the scrape list is the topology diagram, one target per box, including the platform’s own components:

# prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    estate: capstone-lab

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  # Workload layer. The env label is applied here because a single
  # Prometheus serves both environments; see the tenancy note below.
  - job_name: checkout
    static_configs:
      - targets: ['checkout-prod:8000']
        labels: { env: prod }
      - targets: ['checkout-staging:8000']
        labels: { env: staging }

  # Collection layer - the collectors' own internal telemetry.
  - job_name: otel-agent
    static_configs:
      - targets: ['otel-agent-prod:8888', 'otel-agent-staging:8888']

  - job_name: otel-gateway
    static_configs:
      - targets: ['otel-gateway:8888']

  # Storage, alerting and presentation layers.
  - job_name: platform
    static_configs:
      - targets: ['loki:3100', 'tempo:3200', 'alertmanager:9093', 'grafana:3000']

prometheus/rules/platform.yml — one service alert and three that watch the platform itself:

# prometheus/rules/platform.yml
groups:
  - name: service
    rules:
      - alert: CheckoutErrorRateHigh
        expr: |
          sum by (env, service) (rate(http_server_requests_total{status=~"5.."}[5m]))
            / sum by (env, service) (rate(http_server_requests_total[5m])) > 0.05
        for: 2m
        labels:
          severity: page
        annotations:
          summary: 'checkout error ratio above 5% in {{ $labels.env }}'

  - name: platform
    rules:
      # The only failure in this lab that every layer notices.
      - alert: TargetDown
        expr: up == 0
        for: 1m
        labels:
          severity: page
        annotations:
          summary: '{{ $labels.job }} target {{ $labels.instance }} is down'

      # Prometheus can evaluate a rule and still fail to deliver it.
      - alert: AlertDeliveryFailing
        expr: rate(prometheus_notifications_errors_total[5m]) > 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: 'Prometheus cannot deliver alerts to Alertmanager'

      # Rule evaluation falling behind is how alerts arrive late.
      - alert: RuleEvaluationSlow
        expr: prometheus_rule_group_last_duration_seconds > 10
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: 'rule group {{ $labels.rule_group }} takes over 10s to evaluate'

loki/loki.yaml — auth_enabled: true is the whole tenancy story. Every read and every write now needs a tenant header, and a request without one is rejected rather than quietly filed under a default:

# loki/loki.yaml
auth_enabled: true

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: warn

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  # Required for OTLP ingestion: attributes that are not promoted to labels
  # are stored as structured metadata.
  allow_structured_metadata: true
  reject_old_samples: false

analytics:
  reporting_enabled: false

tempo/tempo.yaml:

# tempo/tempo.yaml
server:
  http_listen_port: 3200
  log_level: warn

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317

ingester:
  # Short blocks so a span pushed during the lab is flushed within it.
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 1h

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

Task 5: Write the alerting and presentation tiers

alertmanager/alertmanager.yml. The receiver is deliberately empty:

# alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'env']
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 1h
  receiver: 'null'

receivers:
  # A receiver with no configs accepts the alert and sends it nowhere. In
  # production this is PagerDuty or an on-call rotation; here it is the end of
  # the road, and that gap is the point of the note below.
  - name: 'null'

grafana/provisioning/datasources/estate.yaml — four data sources. The two Loki entries have the same URL and differ only in the tenant header:

# grafana/provisioning/datasources/estate.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    uid: prom-estate
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      timeInterval: 15s

  - name: Loki (prod)
    uid: loki-prod
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false
    jsonData:
      httpHeaderName1: 'X-Scope-OrgID'
    secureJsonData:
      httpHeaderValue1: 'prod'

  - name: Loki (staging)
    uid: loki-staging
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false
    jsonData:
      httpHeaderName1: 'X-Scope-OrgID'
    secureJsonData:
      httpHeaderValue1: 'staging'

  - name: Tempo
    uid: tempo-estate
    type: tempo
    access: proxy
    url: http://tempo:3200
    editable: false

Give Grafana an admin password from a file rather than an environment variable, so the credential never appears in docker inspect or the process table:

mkdir -p "$LAB/secrets"
openssl rand -base64 24 | tr -d '=/+' > "$LAB/secrets/grafana_admin"
chmod 0600 "$LAB/secrets/grafana_admin"

Task 6: Write the compose file and start the estate

# compose.yaml
name: rb-obs-capstone

x-gen: &gen
  build: ./gen
  restart: unless-stopped
  volumes:
    - applogs:/var/log/app

x-agent: &agent
  image: otel/opentelemetry-collector-contrib:0.110.0
  command: ['--config=/etc/otelcol/agent.yaml']
  restart: unless-stopped
  volumes:
    - ./otel/agent.yaml:/etc/otelcol/agent.yaml:ro
    - applogs:/var/log/app:ro

services:
  checkout-prod:
    <<: *gen
    environment:
      SERVICE: checkout
      ENVIRONMENT: prod
      RATE: '20'
      FAIL_RATIO: '0.005'
      LOG_PATH: /var/log/app/checkout-prod.log

  checkout-staging:
    <<: *gen
    environment:
      SERVICE: checkout
      ENVIRONMENT: staging
      RATE: '5'
      # Interpolated so Task 7 can raise it from the shell without editing
      # this file. Compose substitutes at `up`, not at container start.
      FAIL_RATIO: '${STAGING_FAIL_RATIO:-0.02}'
      LOG_PATH: /var/log/app/checkout-staging.log

  otel-agent-prod:
    <<: *agent
    environment:
      LOG_GLOB: /var/log/app/checkout-prod.log
      SERVICE_NAME: checkout
      ENVIRONMENT: prod
      GATEWAY_ENDPOINT: otel-gateway:4317
    # The collector image carries the binary and nothing else - no shell, so
    # `docker compose exec` cannot help you. Publish the telemetry port and
    # read it from the host instead.
    ports:
      - '127.0.0.1:8888:8888'

  otel-agent-staging:
    <<: *agent
    environment:
      LOG_GLOB: /var/log/app/checkout-staging.log
      SERVICE_NAME: checkout
      ENVIRONMENT: staging
      GATEWAY_ENDPOINT: otel-gateway:14317
    ports:
      - '127.0.0.1:8898:8888'

  otel-gateway:
    image: otel/opentelemetry-collector-contrib:0.110.0
    command: ['--config=/etc/otelcol/gateway.yaml']
    restart: unless-stopped
    volumes:
      - ./otel/gateway.yaml:/etc/otelcol/gateway.yaml:ro
    ports:
      - '127.0.0.1:4318:4318'
      - '127.0.0.1:8889:8888'

  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=6h'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prom-data:/prometheus
    ports:
      - '127.0.0.1:9090:9090'

  alertmanager:
    image: prom/alertmanager:v0.28.1
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    restart: unless-stopped
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - am-data:/alertmanager
    ports:
      - '127.0.0.1:9093:9093'

  loki:
    image: grafana/loki:3.3.0
    command: -config.file=/etc/loki/loki.yaml
    restart: unless-stopped
    volumes:
      - ./loki/loki.yaml:/etc/loki/loki.yaml:ro
      - loki-data:/loki
    ports:
      - '127.0.0.1:3100:3100'

  # Tempo runs as uid 10001 and its storage paths do not exist in the image,
  # so the named volume is created owned by root and Tempo cannot write to it.
  tempo-init:
    image: grafana/tempo:2.6.0
    user: root
    entrypoint: ['/bin/sh', '-c']
    command: ['chown -R 10001:10001 /var/tempo']
    volumes:
      - tempo-data:/var/tempo

  tempo:
    image: grafana/tempo:2.6.0
    command: -config.file=/etc/tempo/tempo.yaml
    restart: unless-stopped
    depends_on:
      tempo-init:
        condition: service_completed_successfully
    volumes:
      - ./tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - '127.0.0.1:3200:3200'

  grafana:
    image: grafana/grafana:11.3.0
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
      GF_USERS_ALLOW_SIGN_UP: 'false'
      GF_AUTH_ANONYMOUS_ENABLED: 'false'
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./secrets/grafana_admin:/run/secrets/grafana_admin:ro
      - grafana-data:/var/lib/grafana
    ports:
      - '127.0.0.1:3000:3000'

volumes:
  applogs:
  prom-data:
  am-data:
  loki-data:
  tempo-data:
  grafana-data:

Validate before you converge. docker compose config catches the YAML mistakes; the component validators catch the ones inside the mounted files:

cd "$LAB"
docker compose config --quiet && echo compose-ok

# --entrypoint is required: these images start their server binary, so the
# arguments of a plain `docker run` are read as server flags, not as a tool
# name. Mount the tree at the path prometheus.yml references, or the
# rule_files glob resolves to nothing and the check passes vacuously.
docker run --rm -v "$LAB/prometheus:/etc/prometheus:ro" \
  --entrypoint /bin/promtool prom/prometheus:v2.55.1 \
  check config /etc/prometheus/prometheus.yml

docker run --rm -v "$LAB/prometheus:/etc/prometheus:ro" \
  --entrypoint /bin/promtool prom/prometheus:v2.55.1 \
  check rules /etc/prometheus/rules/platform.yml

docker run --rm -v "$LAB/alertmanager:/etc/alertmanager:ro" \
  --entrypoint /bin/amtool prom/alertmanager:v0.28.1 \
  check-config /etc/alertmanager/alertmanager.yml

$ docker compose up -d --build

Give it five minutes before measuring anything.

docker compose ps --format 'table {{.Name}}\t{{.Status}}'

Task 7: Prove every hop

The rule for this section: each command must cross the hop it is testing. A check that reads the same layer it is asserting about proves only that the layer can talk to itself.

# Layer by layer, bottom up. Workload: the exposition exists at all.
docker compose exec -T prometheus wget -qO- http://checkout-prod:8000/metrics \
  | grep -c '^http_server_requests_total'

# Storage (metrics): Prometheus reached every box in the diagram.
curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up' \
  | jq -r '.data.result[] | "\(.value[1])  \(.metric.job)  \(.metric.instance)"' | sort

Every line must start with 1. A 0 here is a hop that is already broken, and fixing it now is cheaper than reading it as a drill result later.

# Collection: the agents are exporting and the gateway is receiving. Read the
# collectors' own telemetry endpoints directly rather than asserting a series
# name - the internal metric names carry a _total suffix in some collector
# releases and not in others, and grep tells you which you have.
curl -s http://localhost:8888/metrics | grep -E '^otelcol_exporter_sent_log_records'
curl -s http://localhost:8889/metrics | grep -E '^otelcol_receiver_accepted_log_records'

# Storage (logs), prod tenant. The header is not optional: auth_enabled is on.
curl -sG http://localhost:3100/loki/api/v1/query_range \
  -H 'X-Scope-OrgID: prod' \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode 'limit=5' | jq '.data.result | length'

# The same query for staging, which must return its own lines...
curl -sG http://localhost:3100/loki/api/v1/query_range \
  -H 'X-Scope-OrgID: staging' \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode 'limit=5' | jq '.data.result | length'

# ...and a tenant that has never been written to, which must return nothing.
curl -sG http://localhost:3100/loki/api/v1/query_range \
  -H 'X-Scope-OrgID: nobody' \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode 'limit=5' | jq '.data.result | length'

The third query returning 0 is the tenant boundary. Same URL, same query, same Loki — a different header, and no data. That is the whole mechanism, and it is why the boundary lives in the request rather than in a label.

Now the trace hop. Push one span into the gateway’s prod OTLP receiver and read it back out of Tempo, which proves gateway and Tempo in one motion:

TRACE_ID=0af7651916cd43dd8448eb211c80319c
SPAN_ID=b7ad6b7169203331
START_NS="$(date +%s)000000000"
END_NS="$(( $(date +%s) + 1 ))000000000"

curl -s -X POST http://localhost:4318/v1/traces \
  -H 'Content-Type: application/json' \
  -d "{\"resourceSpans\":[{\"resource\":{\"attributes\":[
        {\"key\":\"service.name\",\"value\":{\"stringValue\":\"checkout\"}}]},
      \"scopeSpans\":[{\"spans\":[{
        \"traceId\":\"$TRACE_ID\",\"spanId\":\"$SPAN_ID\",
        \"name\":\"POST /checkout\",\"kind\":2,
        \"startTimeUnixNano\":\"$START_NS\",\"endTimeUnixNano\":\"$END_NS\"}]}]}]}"

sleep 10
curl -s "http://localhost:3200/api/traces/$TRACE_ID" | jq '.batches | length'

Finally the alerting and presentation layers. Raise the staging error rate above the rule’s threshold and follow the alert all the way to Alertmanager:

STAGING_FAIL_RATIO=0.5 docker compose up -d --force-recreate checkout-staging

Wait out the for: 2m plus the rate window — about five minutes — then:

# Prometheus thinks the rule is firing.
curl -s http://localhost:9090/api/v1/alerts \
  | jq -r '.data.alerts[] | "\(.labels.alertname) \(.labels.env // "-") \(.state)"'

# Alertmanager actually received it. This is the hop that matters.
curl -s http://localhost:9093/api/v2/alerts \
  | jq -r '.[] | "\(.labels.alertname) \(.status.state)"'

# Presentation: Grafana can reach each data source, using the admin password
# from the file rather than from a shell variable in your history.
GF_PW="$(cat "$LAB/secrets/grafana_admin")"
for uid in prom-estate loki-prod loki-staging tempo-estate; do
  printf '%s: ' "$uid"
  curl -s -u "admin:$GF_PW" "http://localhost:3000/api/datasources/uid/$uid" | jq -r '.name'
done

Task 8: The topology drill

For each drill: read the diagram, write down your prediction before running anything, then run it. The value is entirely in the gap between the two.

Keep a log:

DRILL="$LAB/drill.md"
printf '# Topology drill\n\n| # | hop cut | predicted | observed | caught by |\n|---|---|---|---|---|\n' > "$DRILL"

Drill 1: the gateway stops

$ docker compose stop otel-gateway

Wait three minutes, then look at the estate the way an on-call would — from the top:

# What a metrics dashboard sees.
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (env) (rate(http_server_requests_total[5m]))' \
  | jq -r '.data.result[] | "\(.metric.env) \(.value[1])"'

# What the log tier sees: nothing new arriving.
curl -sG http://localhost:3100/loki/api/v1/query_range \
  -H 'X-Scope-OrgID: prod' \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" \
  --data-urlencode 'limit=5' | jq '.data.result | length'

# Where the evidence actually is: the agent's own export counters.
curl -s http://localhost:8888/metrics | grep -E '^otelcol_exporter_(sent|send_failed)_log_records'

The metrics query is perfectly healthy, because metrics never went through the gateway. Logs stopped. The up == 0 rule fires for the otel-gateway job, which is the only reason the platform notices at all — and it noticed because somebody thought to put the collector in scrape_configs. An estate that scrapes only its workloads is completely blind to this.

Restart it and watch the agent’s queue drain:

docker compose start otel-gateway
sleep 60
curl -s http://localhost:8888/metrics | grep -E '^otelcol_exporter_(sent|send_failed)_log_records'

The sending_queue on the agent is what turned a gateway outage into a delay instead of a hole. Note the queue is in memory here: it survives a gateway restart, not an agent restart.

Drill 2: the pipeline loses its tenant

Edit otel/gateway.yaml and delete the headers: block from otlphttp/loki-prod, then recreate the gateway:

docker compose up -d --force-recreate otel-gateway
sleep 90

# The write path reports nothing wrong.
docker compose logs --tail 20 otel-gateway | grep -ci error

# The prod tenant stops receiving. It does not error - it just stops.
curl -sG http://localhost:3100/loki/api/v1/query_range \
  -H 'X-Scope-OrgID: prod' \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode "start=$(( $(date +%s) - 60 ))000000000" | jq '.data.result | length'

# Staging, whose exporter still has its header, is unaffected. That contrast
# is the diagnostic: one pipeline, not the platform.
curl -sG http://localhost:3100/loki/api/v1/query_range \
  -H 'X-Scope-OrgID: staging' \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode "start=$(( $(date +%s) - 60 ))000000000" | jq '.data.result | length'

No alert fires. up is 1 everywhere. The collector is exporting successfully — it is exporting to a tenant nobody reads. Restore the header and recreate the gateway before continuing.

Drill 3: alerts evaluate but do not deliver

$ docker compose stop alertmanager

sleep 180

# Prometheus is certain the alert is firing.
curl -s http://localhost:9090/api/v1/alerts | jq -r '.data.alerts[].state' | sort | uniq -c

# And equally certain it cannot deliver it.
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(prometheus_notifications_errors_total[5m])' \
  | jq -r '.data.result[].value[1]'

This is the failure that looks least like a failure. Every dashboard is correct, every rule is right, and no page is sent. AlertDeliveryFailing and the up == 0 on the platform job both catch it — but both of them are alerts, and alerts are the thing that is broken. In production the watcher for this hop lives somewhere else: a second Prometheus, a blackbox probe, or a dead-man’s-switch alert that fires always and pages when it stops arriving.

docker compose start alertmanager

Drill 4: the right dashboard, the wrong environment

Change httpHeaderValue1 on the loki-prod data source from prod to staging in grafana/provisioning/datasources/estate.yaml, then:

docker compose restart grafana
sleep 20

GF_PW="$(cat "$LAB/secrets/grafana_admin")"
curl -s -u "admin:$GF_PW" http://localhost:3000/api/datasources/uid/loki-prod \
  | jq '{name, url, jsonData}'

Grafana reports the data source as healthy, because it is: the URL resolves, Loki answers, rows come back. They are staging’s rows. Nothing in the API response shows the tenant, because the value lives in secureJsonData and is never returned.

This drill produces no symptom anywhere in the platform. The only defences are upstream of it: the provisioning file in version control, a review that reads diffs of secureJsonData, and a CI check that asserts each environment’s data source carries its own tenant. Restore the file and restart Grafana.

Drill 5: a target goes away

$ docker compose stop checkout-staging

sleep 90
curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up == 0' \
  | jq -r '.data.result[] | "\(.metric.job) \(.metric.instance)"'
curl -s http://localhost:9093/api/v2/alerts | jq -r '.[].labels.alertname' | sort -u

# Restore the generator to its normal error ratio, undoing Task 7's trigger.
docker compose up -d --force-recreate checkout-staging

A dead target is the easy case, and every monitoring system in the world catches it. Note how little it resembles drills 2 and 4 — and note that most monitoring is built as though drill 5 is the shape all failures take.

Validation

Run this after restoring the estate. It walks all five layers in order and prints one line per claim:

cd "$LAB"
GF_PW="$(cat "$LAB/secrets/grafana_admin")"
NOW="$(date +%s)"

echo "== layer 1: every box in the diagram is scraped and up"
curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up' \
  | jq -r '.data.result[] | select(.value[1] != "1") | "DOWN \(.metric.instance)"' \
  | grep . || echo "all targets up"

echo "== layer 2: both environments are producing metrics"
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (env) (rate(http_server_requests_total[5m])) > 0' \
  | jq -r '.data.result[] | "\(.metric.env) ok"'

echo "== layer 3: both tenants are receiving logs, and a third is not"
for t in prod staging nobody; do
  n=$(curl -sG http://localhost:3100/loki/api/v1/query_range \
        -H "X-Scope-OrgID: $t" \
        --data-urlencode 'query={service_name="checkout"}' \
        --data-urlencode "start=$(( NOW - 300 ))000000000" \
        --data-urlencode 'limit=5' | jq '.data.result | length')
  echo "tenant $t streams=$n"
done

echo "== layer 4: alerts evaluate and are delivered"
curl -s http://localhost:9090/api/v1/alerts | jq -r '.data.alerts | length' \
  | xargs -I{} echo "prometheus firing/pending: {}"
curl -s http://localhost:9093/api/v2/alerts | jq -r 'length' \
  | xargs -I{} echo "alertmanager holds: {}"

echo "== layer 5: presentation reaches every backend"
for uid in prom-estate loki-prod loki-staging tempo-estate; do
  printf '%s ' "$uid"
  curl -s -o /dev/null -w '%{http_code}\n' -u "admin:$GF_PW" \
    "http://localhost:3000/api/datasources/uid/$uid"
done

The expected shape: layer 1 prints all targets up; layer 3 prints non-zero for prod and staging and 0 for nobody; layer 5 prints 200 four times.

Expected Outcome

• Nine containers running, all Prometheus targets up, six named volumes.
• Both environments visible in Prometheus by the env label, and in Loki by tenant — two different mechanisms, deliberately, for two different backends.
• A Loki query against an unwritten tenant returns nothing while the same query against prod returns lines.
• One span pushed through the gateway is retrievable from Tempo by trace id.
• A firing alert is visible in both /api/v1/alerts and Alertmanager’s /api/v2/alerts.
• A completed drill.md with five rows, of which two have “caught by: nothing”.

Troubleshooting

docker compose up fails on a port. Compare against ports.pre-lab. Every publish in this file is 127.0.0.1-bound, so the conflict is another loopback listener, most often a previous run of this lab that was not cleaned up.

Tempo restarts with a permission error on /var/tempo. The tempo-init container did not complete. docker compose logs tempo-init will say why; until it exits successfully, depends_on holds Tempo back and the symptom is a container that never starts rather than one that crashes.

Loki returns HTTP 401 on a query. auth_enabled: true and no X-Scope-OrgID header. This is the correct behaviour and it is the reason to enable it: a missing tenant is refused rather than silently mapped to a default tenant where the data would be findable by anyone.

A Loki query returns 200 with an empty result and you expected lines. Three causes, in cost order: the time range (start defaults to one hour ago), the tenant header, and the stream selector. Confirm which by asking Loki what labels it holds for that tenant: curl -sG http://localhost:3100/loki/api/v1/labels -H 'X-Scope-OrgID: prod'.

The agent shows no accepted log records. The filelog receiver found no file matching LOG_GLOB. Check that the generator is writing: docker compose exec -T checkout-prod ls -l /var/log/app. A generator that crashed on start leaves an empty directory and an agent with nothing to do, which looks identical to a broken agent.

CheckoutErrorRateHigh never fires. The rule needs for: 2m on top of a [5m] rate window: allow seven minutes. Confirm the ratio is actually above the threshold by running the alert’s own expression in /api/v1/query before assuming the rule is wrong.

Cleanup

cd "$LAB"
docker compose down -v --remove-orphans --rmi local

# Confirm nothing of the estate survives.
docker volume ls --filter 'name=rb-obs-capstone' --format '{{.Name}}'
docker ps --format '{{.Names}}' | diff - containers.pre-lab && echo "container list restored"

cd "$HOME" && rm -rf "$LAB"

Production notes

Mapping this to a real change window, the estate is four independent changes, not one:

Storage tier first, alone. Prometheus, Loki and Tempo can run for days with nothing writing to them. Rollback is stopping them; blast radius is zero because nothing depends on them yet.

Collection tier second. Agents before gateway, and the agent’s sending_queue is what makes that order safe: agents that cannot reach a gateway queue rather than drop. This is the step where you find out whether your firewall rules match your diagram.

Alerting third, and route everything to a test receiver for the first week. An Alertmanager wired to the real on-call rotation on day one turns every configuration mistake into a page.

Presentation last. Grafana is the only tier whose failure is purely a degradation — dashboards down, telemetry still being collected. Putting it last means every earlier tier has been proved by command line before anyone judges it by a dashboard.

Three things this lab simplifies. There is no TLS on any hop: in production the collector-to-backend hop is mTLS with the backend authorising writers by client certificate, and the Grafana-to-backend hop carries a bearer token from a secret store rather than a value in a provisioning file. Every component is a single replica: the reference architecture puts three gateways behind a load balancer and two Alertmanagers in a cluster, because the gateway and the alerting tier are the two places where one instance is a single point of failure for the whole estate. And retention is six hours on a local disk — production Prometheus keeps 15 to 30 days locally and streams the long tail to object-storage-backed remote storage.

What You Learned

• Metrics and logs do not share a path, so they do not share a failure. Drill 1 cut every log in the estate and left a metrics dashboard that was entirely, honestly green. Any statement of the form “monitoring is up” that is not qualified by which signal is a statement about one path.
• Scraping your own platform is not optional. The only reason the gateway outage was detectable at all is that the collectors were in scrape_configs. The components you do not scrape are the components that fail silently.
• The tenant boundary lives in the request, not in the data. One Loki, one URL, one query — and a header that decides whether you get prod, staging, or nothing. That design keeps tenancy out of the label set, where it would multiply cardinality by the tenant count.
• Two of the five drills produced no symptom at all. A pipeline exporting to an unread tenant, and a data source pointed at the wrong environment, are both indistinguishable from correct operation from inside the platform. Their defences are configuration in version control, review, and CI — not monitoring.
• The layer that catches a failure is rarely the layer that has it. The gateway outage was caught by Prometheus. The Alertmanager outage was caught by an alert, which is exactly the wrong watcher. Write down, for each component, what would notice if it stopped — and if the answer is “itself”, you have found the next piece of work.