Application RED

At 14:32 UTC the synthetic checkout probe in us-east-1 reports p95 of 1.8 seconds against an SLO of 500 ms. The 5xx rate is within band. The synthetic probe is not alone: customers in the support channel are reporting that checkout takes forever. The on-call engineer opens Grafana, sees a wall of green panels, and has no answer. The dashboard was optimised for uptime, not for investigation.

The RED method is what closes that gap. It is the smallest set of metrics that describes the user-visible behaviour of an application service: Rate, Errors, Duration. Three numbers per service, per route, per status class. If those three numbers are correct, the on-call engineer can answer the question “is the service healthy for users?” in one panel. If even one of them is missing, the investigation has to fall back to logs or traces, which is slower.

What it is

RED is a micro-services methodology. It applies to the application layer, not the host or the network. For each service that the user calls directly, RED emits three metrics:

• Rate — requests per second, broken down by route and status class. A counter, monotonically increasing, resumed in PromQL with rate().
• Errors — the rate of failed requests, where “failed” is defined by the service. For an HTTP service, the conventional definition is status_code >= 500. For a message consumer, the definition is the count of messages that ended in a DLQ.
• Duration — the latency distribution of every request, not just the average. A histogram, usually seconds, with buckets chosen so that histogram_quantile(0.95, ...) and histogram_quantile(0.99, ...) are both meaningful.

The optional fourth metric is Saturation — in-flight requests, queue depth — which is covered in the saturation-from-application lesson. RED contains the three above; USE and the saturation dimension extend it.

The method originated with Tom Wilkie at Weaveworks in 2018 and is the canonical companion to Brendan Gregg’s USE method, which applies to the host layer. The two are complementary, not interchangeable.

Why a sysadmin cares

The single characteristic of a well-instrumented service is that the on-call engineer can answer the question “is this service healthy for users, and if not, since when?” in under thirty seconds. RED is the minimum data set that lets that answer happen from metrics alone, without opening a trace.

Three operational problems disappear when RED is in place:

1. The unknown failure mode. A deployment that introduces a 3% regression on the checkout path. The RED rate panel shows the same total; the duration panel shows p95 up by 200 ms. The engineer has evidence by phase 2 of the investigation.
2. The “is it me or them” question. The on-call sees the checkout 5xx rate at 0.4%. The dependency 5xx rate (from the payment-svc side) is 0.4%. The dependency is the cause. Without per-dependency RED, this question is open for the first ten minutes of every incident.
3. The “is it slow or broken” question. A 5xx alert is binary — the request failed. A duration alert is gradient — the request succeeded but slowly. RED is the only methodology that surfaces both shapes with the same data set.

How it works

The mental model is small: every request hits a service, the service records three numbers, the three numbers are scraped into Prometheus, and a dashboard joins them to a single service-row:

  user request
      |
      v
  +---------+
  | service |  -- rate  -->  http_server_request_duration_seconds_count
  +---------+  -- duration -> http_server_request_duration_seconds_bucket
      |        -- errors -->  same counter, status_code in 5xx
      v
  response (one of: 2xx, 4xx, 5xx)

The three metrics are derived from the same underlying instrument. The OpenTelemetry semantic convention for HTTP servers is to emit one histogram (http.server.request.duration) whose _count series is the rate and whose _bucket series is the duration distribution. Per-status breakdown is achieved with the http.response. status_code attribute on the histogram. This collapses three metrics into one, with the same data set.

The cardinality budget is the constraint. The http.response.status_code attribute adds at most a handful of values (200, 201, 400, 401, 404, 500, 502, 503). The http.route attribute (the templated path, e.g. /users/{id}/orders) is bounded by the number of routes in the service, typically under 100. The combination is acceptable. The trap is the un-templated URL — if the attribute is the literal request path (/users/7291/orders), the cardinality is unbounded and the histogram explodes. That is the most common RED failure mode in production.

Under the hood

How to configure it

The instrument is configured in three places: the SDK at service startup, the Collector pipeline, and the Prometheus scrape config. The three must agree on the metric and label names; the SDK is the source of truth.

1. The SDK at service startup

A Python FastAPI service using OpenTelemetry 0.110.x:

# app.py -- RED instrumentation
import time
from fastapi import FastAPI, Request
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import (
    PeriodicExportingMetricReader,
)
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
    OTLPMetricExporter,
)
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "checkout",
    "service.version": "1.4.2",
    "deployment.environment": "prod",
})

# SEVERITY: CONFIGURATION -- sets the OTLP endpoint at startup
reader = PeriodicExportingMetricReader(
    exporter=OTLPMetricExporter(
        endpoint="otel-collector.observability.svc:4317",
        insecure=True,
    ),
    export_interval_millis=10_000,
)

provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)

meter = metrics.get_meter("checkout", "1.4.2")

# Single histogram; rate and errors come from rate(..._count)
duration = meter.create_histogram(
    "http.server.request.duration",
    unit="s",
    description="Wall-clock duration of inbound HTTP requests",
)

active = meter.create_up_down_counter(
    "http.server.active_requests",
    unit="1",
    description="In-flight HTTP requests currently being served",
)

app = FastAPI()

@app.middleware("http")
async def red_middleware(request: Request, call_next):
    attrs = {
        "http.request.method": request.method,
        "http.route": request.scope.get("route", "unmatched"),
        "url.scheme": request.url.scheme,
    }
    active.add(1, attrs)
    start = time.perf_counter()
    status = 500
    try:
        response = await call_next(request)
        status = response.status_code
        return response
    finally:
        elapsed = time.perf_counter() - start
        full_attrs = {**attrs, "http.response.status_code": status}
        duration.record(elapsed, full_attrs)
        active.add(-1, attrs)

The http.route attribute uses the templated path (/users/{id}/orders), not the literal path. The auto-instrumentation packages for FastAPI, Flask, and Django all set this correctly. The most common production bug is the middleware that records the raw request.url.path instead.

2. The Collector pipeline

The Collector receives the OTLP stream and forwards to Prometheus remote-write. The relevant pipeline section:

# /etc/otelcol-contrib/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # SEVERITY: CONFIGURATION -- drops the high-cardinality
  # http.url attribute (the raw URL) while keeping route.
  attributes/remove_http_url:
    actions:
      - key: http.url
        action: delete
      - key: url.full
        action: delete
      - key: http.target
        action: delete
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
    tls:
      insecure: true
    resource_to_telemetry_conversion:
      enabled: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [attributes/remove_http_url, batch]
      exporters: [prometheusremotewrite]

The attributes/remove_http_url processor is the guardrail against the most common RED failure mode. Even if the SDK leaks the raw URL, the Collector drops it before it reaches Prometheus.

3. The Prometheus scrape / recording rule

RED is usually promoted to a recording rule so the dashboard panels are pre-computed:

# /etc/prometheus/rules/red.rules
groups:
- name: red
  interval: 30s
  rules:
  - record: service:request_rate:rate5m
    expr: |
      sum by (service_name, http_route) (
        rate(http_server_request_duration_seconds_count[5m])
      )

  - record: service:error_rate:rate5m
    expr: |
      sum by (service_name, http_route) (
        rate(http_server_request_duration_seconds_count{
          http_response_status_code=~"5.."
        }[5m])
      )

  - record: service:request_duration:p95
    expr: |
      histogram_quantile(0.95,
        sum by (service_name, http_route, le) (
          rate(http_server_request_duration_seconds_bucket[5m])
        )
      )

The promtool validator confirms the rule syntax:

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/red.rules

Expected output:

SUCCESS: /etc/prometheus/rules/red.rules

How to validate it

Three checks confirm RED is live and correct end-to-end.

1. The metric is scraped. The Prometheus targets page shows the service as up and the metric as present:

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=up{job="checkout"}'

Expected output:

{"status":"success","data":{"resultType":"vector",
 "result":[{"metric":{"job":"checkout",
  "instance":"checkout-7d4b-abcde:8080"},
  "value":[1755000030,"1"]}]}}

The 1 confirms the scrape is succeeding.

2. The rate and the error count are present. The raw counter for the checkout service:

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=sum%20by%20(http_response_status_code)%20(http_server_request_duration_seconds_count{service_name%3D"checkout"})'

Expected output:

http_response_status_code
200  1820341
400    12044
404     2381
500      211
502       18
503      104

The 5xx totals are small relative to 2xx, which is the healthy shape. A 5xx rate above 1% of total is a signal of misconfiguration or a regression; check the work queue.

3. The duration histogram has its expected buckets. The buckets are spread so that p95 and p99 are both meaningful for an interactive service:

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.95%2C%20sum%20by%20(le)%20(rate(http_server_request_duration_seconds_bucket{service_name%3D"checkout"%2C%20http_route%3D"/checkout"}[5m])))'

Expected output:

{"value":[1755000030,"0.181"]}

A value of 0.181 is 181 ms. The SLO for checkout is 500 ms. The panel is well inside the SLO.

How it can fail

Five specific failure shapes appear regularly in RED instrumentation:

1. Un-templated path in the http.route attribute. The middleware records request.url.path instead of the templated path. Every distinct user ID is a new histogram series. Prometheus memory climbs; the histogram is unusable. Symptom: rate of http_server_request_duration_seconds_count series growth per hour, OOMKilled Prometheus.
2. Rate and error counter desynchronised. Two separate counters for rate and errors, advanced by different code paths. A handler exception that increments the error counter but not the rate counter skews the error fraction above 1.0. Symptom: the error-fraction panel shows NaN or >1.0 during an incident.
3. Histogram buckets borrowed from another service. A Go service that talks to a database using the database default buckets (microseconds). The p95 of an HTTP request lands in [+Inf, +Inf] because every bucket up to 1 second is empty. The panel is flat. Symptom: histogram_quantile(0.95, ...) returns NaN or the value of the previous bucket.
4. The status code is recorded from the wrong layer. A reverse proxy that records 200 on the response, but the application raised before serialising. The 5xx is emitted by the proxy but the application counter reports 200. The error fraction understates by an order of magnitude. Symptom: service:error_rate:rate5m is 0.01% during an incident that customer reports put at 30%.
5. The active_requests counter is unbalanced. The middleware increments on entry but crashes before decrementing on exit. The gauge climbs monotonically. The saturation panel is permanently red. Symptom: http_server_active_requests grows forever; the panel is unresponsive to traffic changes.

How to troubleshoot it

When a RED panel says something the operator does not believe, the diagnostic order is:

1. Confirm the metric is the right metric. Check the scrape target is up. Check the metric name is what the dashboard expects (the SDK name and the Prometheus-side name differ by the unit suffix).
2. Confirm the labels are bounded. Run count by (__name__)({__name__=~"http_server_.*"}) and confirm the per-metric series count is under 10,000 for the service. If it is above, the cardinality is leaking.
3. Confirm the histogram buckets are appropriate. Pull a single bucket series and confirm the bucket boundaries cover the latency range of the service. If the boundaries are wrong, the quantile is wrong.
4. Confirm the rate matches the application’s own counter. A typical FastAPI middleware emits an access log with a request count. The application-log count and the Prometheus rate should match within 1%.
5. Confirm the status code is recorded at the right layer. The 5xx counter should count errors the application raised, not errors the proxy wrapped. If the application uses a try/except that converts internal errors to a 200 response, the 5xx counter is silent.

Security implications

The RED metric set is small and the labels are bounded, but two risks appear in production:

• The http.route attribute leaks a user identifier. A route template like /users/{id}/profile is fine; a literal path /users/7291/profile is not. The Collector should drop the raw URL and the templated path should be reviewed for user identifiers before release.
• The status code can leak error details. A 500 response in the access log may contain a stack trace with file paths and configuration. The access log is not in RED; RED’s status code is a number. The number is safe; the log is not.

Restrict the Prometheus query of the RED metric set to operators with a recorded purpose. RED is not sensitive on its own, but the same label set may be enriched by custom metrics in the next lesson.

Performance implications

RED is cheap. The counter is a single in-memory increment per request. The histogram is a single record per request, with bucket selection in O(bucket_count) — typically under 20 nanoseconds. The OTLP exporter serialises the data every 10 seconds; the serialized payload is bounded by the number of unique label combinations.

The on-call cost is the query. A histogram_quantile() across the full bucket set for 100 services is a few hundred millicores of Prometheus query time. The recording rule resolves this; the dashboard should consume the recording rule, not the raw histogram.

If the cardinality is unbounded (http.url leaked onto the histogram), the cost is unbounded. The cardinality budget is the limit.

Production guidance

• One RED set per service, named in the SDK. The metric name is owned by the SDK; the Prometheus translation is derived from it. Drift between the two is the most common production bug.
• Use the stable HTTP semantic conventions. The http.request.method, http.route, and http.response.status_code attributes are the canonical attribute set. The deprecated http.method and http.status_code are still emitted by older instrumentations; both are equivalent, but dashboards should pin one set.
• Promote RED to recording rules. The PromQL for histogram_quantile is expensive. The recording rule evaluates every 30 seconds and the dashboard reads the pre-aggregated series.
• Validate the cardinality on every release. A pre-deploy check that runs count by (__name__)({__name__=~"http_server_.*"}) against the staging Prometheus and asserts the count is under budget. The check belongs in CI, not in the runbook.
• Pair RED with USE. RED is the user-visible layer; USE is the host layer. The two together give the on-call both the symptom and the cause.

Verification

You should now be able to answer:

• What is each of the three RED metrics, and what unit is it measured in?
• Which Prometheus metric name carries the rate, the errors, and the duration, and how do they relate to a single OpenTelemetry histogram?
• What is the cardinality trap in the http.route attribute, and how is it prevented?
• What does an unbalanced active_requests counter look like in a Grafana panel, and what causes it?
• Why is the recording rule preferred over the raw histogram in the dashboard?

Quiz