Capstone stage 4 — wiring observability to an SLO · Docker & Containers

Stage 3 left the stack serving traffic. Stage 4 makes it answer questions: is it working, for whom, how badly is it failing, and where.

The trap in this stage is building a beautiful observability stack that nobody consults. The defence is to work backwards from a single number — the SLO — and add only the telemetry that either measures it or explains a violation of it.

The SLO first

Service:        api.example.com
SLI:            proportion of HTTP requests that return a non-5xx
                status within 300 ms, measured at the edge
SLO:            99.5% over a rolling 28 days
Error budget:   0.5% of 28 days of requests
                at 400 rps = ~4.8 million allowed bad requests
                or, in time terms, 3 h 22 m of total outage

Measured at the edge, not in the application: a request the API never received still failed for the user. That choice decides where the metric comes from, which decides what stage 4 scrapes.

What gets scraped

# prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    env: production
    host: capstone-1

scrape_configs:
  - job_name: caddy
    static_configs:
      - targets: ['caddy:2019']

  - job_name: api
    static_configs:
      - targets: ['api:8080']
    metrics_path: /metrics

  - job_name: web
    static_configs:
      - targets: ['web:8080']
    metrics_path: /metrics

  - job_name: worker
    static_configs:
      - targets: ['worker:8080']
    metrics_path: /metrics

  - job_name: postgres
    static_configs:
      - targets: ['postgres-exporter:9187']

  - job_name: redis
    static_configs:
      - targets: ['redis-exporter:9121']

  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: cadvisor
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: docker
    static_configs:
      - targets: ['172.17.0.1:9323']

The last job is the one usually missing. Stage 1 set "metrics-addr": "127.0.0.1:9323" in daemon.json, which exposes the daemon’s own metrics — build counts, container state transitions, API request latency. Prometheus is in a container, so it reaches the host through the bridge gateway rather than through 127.0.0.1.

`127.0.0.1` inside a container is the container

metrics-addr: 127.0.0.1:9323 binds the daemon’s metrics endpoint to the host’s loopback. A Prometheus container scraping 127.0.0.1:9323 scrapes its own loopback, finds nothing, and reports the target as down with connection refused — which reads as “the daemon metrics are not enabled”.

Options, in order of preference:

Scrape the bridge gateway address (172.17.0.1 by default; confirm with docker network inspect bridge).
Use host.docker.internal with an extra_hosts: ["host.docker.internal:host-gateway"] entry on the Prometheus service, which survives a bridge subnet change.
Bind the daemon metrics to the bridge address rather than loopback — less good, because it is then reachable from every container.

The same mistake catches node-exporter configurations and any attempt to scrape a service running directly on the host.

Read-only / Safeevery target up— The gate for the scrape configuration. Any target not 'up' is a gap in the picture, whatever the dashboards show.

curl -fsS http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.labels.job)\t\(.health)\t\(.lastError)"'
curl -fsS http://localhost:9090/api/v1/targets \
  | jq -e '[.data.activeTargets[] | select(.health != "up")] | length == 0' \
  && echo 'OK: all targets up'

caddy	up	
api	up	
web	up	
worker	up	
postgres	up	
redis	up	
node	up	
cadvisor	up	
docker	up	
OK: all targets up

Illustrative output

One label ties the three pillars together

Metrics, logs and traces are only useful together if they agree on the name of a thing. The capstone uses service everywhere, and stage 1’s log-opts.labels setting is what makes it work on the log side.

# each service in compose.yml
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        labels: "com.docker.compose.service"

# promtail / alloy scrape config — relabels the Compose service into `service`
scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 15s
    relabel_configs:
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

A log shipper with the Docker socket has host root

docker_sd_configs needs the socket to discover containers, and any container with the socket mounted is root-equivalent on the host.

Mount it read-only (/var/run/docker.sock:/var/run/docker.sock:ro) and understand that :ro prevents nothing meaningful — the Docker API is served over that socket and POST /containers/create is a read of the socket file as far as the kernel is concerned. :ro blocks writing to the file, not calling the API.

The real mitigations are a socket proxy that allow-lists only the API paths the shipper needs, or discovering containers from the filesystem rather than the API. For a capstone the socket mount is acceptable if it is written down as an accepted risk with the mitigation named — and your stage 5 hardening audit will flag it, which is the correct outcome.

Read-only / Safethe same request in all three pillars— A trace ID from a response header should find the trace, the logs, and the latency sample. This is the correlation test.

TRACE=$(curl -fsS -D - -o /dev/null https://api.example.com/v1/orders \
  | awk '/^traceparent:/ {split($2,a,"-"); print a[2]}')
echo "trace: $TRACE"
curl -fsS "http://localhost:3200/api/traces/$TRACE" | jq -r '.batches | length'
curl -fsS --get http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode "query={service=\"api\"} |= \"$TRACE\"" \
  | jq -r '.data.result | length'

trace: 4bf92f3577b34da6a3ce929d0e0e4736
1
3

Illustrative output

If that returns a trace and no logs, the log pipeline is not injecting the trace ID and the correlation is decorative. Fix it now — during an incident is the wrong time to discover that the “click through to logs” link does nothing.

Alerting from the budget

Static thresholds produce alerts that are either too noisy or too late. Burn-rate alerting asks a different question: at the current error rate, how fast is the budget being consumed?

# rules.yml
groups:
  - name: api-slo
    rules:
      - record: job:api_request:error_ratio_5m
        expr: |
          sum(rate(caddy_http_requests_total{host="api.example.com",code=~"5.."}[5m]))
          /
          sum(rate(caddy_http_requests_total{host="api.example.com"}[5m]))

      - record: job:api_request:error_ratio_1h
        expr: |
          sum(rate(caddy_http_requests_total{host="api.example.com",code=~"5.."}[1h]))
          /
          sum(rate(caddy_http_requests_total{host="api.example.com"}[1h]))

      - alert: ApiErrorBudgetFastBurn
        expr: |
          job:api_request:error_ratio_5m > (14.4 * 0.005)
          and
          job:api_request:error_ratio_1h > (14.4 * 0.005)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "API burning error budget 14.4x — full budget gone in ~2 days"

      - alert: ApiErrorBudgetSlowBurn
        expr: |
          job:api_request:error_ratio_1h > (6 * 0.005)
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "API burning error budget 6x — budget gone in ~5 days"

The two magic numbers are not magic:

budget window        = 28 days
fast burn multiple   = 14.4   -> exhausts the 28-day budget in 2 days
slow burn multiple   =  6     -> exhausts it in about 5 days
error ratio trigger  = multiple x (1 - SLO) = 14.4 x 0.005 = 7.2%

The fast rule requires both the 5-minute and the 1-hour windows to be burning. The short window makes it react quickly; the long window stops a 90-second blip from paging anyone. That pairing is the whole technique.

Read-only / Safethe rules loaded and the budget remaining— Confirms Prometheus accepted the rules and shows how much of the 28-day budget is left.

promtool check rules rules.yml
curl -fsS http://localhost:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[] | select(.type=="alerting") | "\(.name)\t\(.state)"'
curl -fsS --get http://localhost:9090/api/v1/query \
  --data-urlencode 'query=1 - (job:api_request:error_ratio_1h / 0.005)' \
  | jq -r '.data.result[0].value[1]'

Checking rules.yml
SUCCESS: 4 rules found
ApiErrorBudgetFastBurn	inactive
ApiErrorBudgetSlowBurn	inactive
0.9993

Illustrative output

The stage 4 gate

An alert that has never fired is an untested alert.

# Force a burn and confirm the fast rule fires, then confirm it clears.
docker compose stop api
sleep 180
curl -fsS http://localhost:9090/api/v1/alerts \
  | jq -r '.data.alerts[] | "\(.labels.alertname) \(.state)"'
docker compose start api

Expected: ApiErrorBudgetFastBurn firing while api is down, and no alerts a few minutes after it returns. Run this in a maintenance window on the capstone, and record the transcript — “we tested the pager” is a claim that decays and needs a date attached.

Why cadvisor and node-exporter are both required

They answer adjacent questions and neither substitutes for the other.

cadvisor reads the cgroup tree and reports per-container resource use: container_memory_working_set_bytes, container_cpu_cfs_throttled_periods_total, container_fs_usage_bytes. This is where the capacity lessons’ numbers come from, per container, over time. Without it you cannot tell which container consumed the host.

node-exporter reports the host: total memory, load, disk usage per filesystem, network counters, nf_conntrack utilisation, and the entropy and time-sync state. Without it you cannot tell whether the host itself is the problem.

The failure mode of having only cadvisor is a fleet where every container looks fine and the host is at 100% disk. The failure mode of having only node-exporter is knowing the host is at 100% disk with no way to attribute it. The capacity review needs both series, which is why stage 4’s target list includes both and the gate requires both to be up.

What happens if Prometheus runs out of disk?

Prometheus is a stateful service and its failure is quiet in a particularly unhelpful way.

As /prometheus fills, the TSDB fails to write new blocks and to compact existing ones. Prometheus keeps serving queries over what it already has, so dashboards render and look plausible — while the most recent data is missing or truncated. Alert rules evaluate against incomplete data and can silently stop firing.

Three defences, all cheap:

Set an explicit retention (--storage.tsdb.retention.time=15d) and size the volume from it. The capstone’s disk model in stage 1 included 15 days of Prometheus data for exactly this reason.
Alert on Prometheus itself, using up{job="prometheus"} from a second instance or a dead-man’s-switch alert that must fire continuously and pages when it stops. A monitoring system that monitors only other things cannot report its own failure.
Watch prometheus_tsdb_compactions_failed_total and prometheus_tsdb_head_series. A rising failure count is the earliest signal, well before the dashboards degrade.

Write the SLO and the error budget down before configuring anything, including where the SLI is measured.
Add every target, including cadvisor, node-exporter and the Docker daemon through the bridge gateway.
**Standardise on one service label** across metrics, logs and traces, and prove correlation with a single trace ID.
Write burn-rate rules with paired short and long windows; validate them with promtool check rules.
Set Prometheus retention and size the volume from it, and add a dead-man alert for the monitoring stack itself.
Gate: every target up, one trace ID found in all three pillars, and a recorded test in which the fast-burn alert fired and cleared.

The SLO first

What gets scraped

One label ties the three pillars together

Alerting from the budget

The stage 4 gate

Sanity check