RED Methodology: Rate, Errors, Duration

RED — Rate, Errors, Duration — is a methodology invented by Tom Wilkie at Weaveworks for distributed services. Every service is described by three metrics: how many requests per second, what fraction fail, and how long the successful requests take.

RED is the methodology for services. USE is the methodology for hosts. The two together cover a distributed system.

What RED is

For every service endpoint:

1. Rate. Requests per second.
2. Errors. Fraction or count of requests that fail.
3. Duration. Time taken to complete each request (histogram).

A RED pass on a service produces:

Rate:
  rate = sum by (route, instance) (
    rate(http_requests_total[5m])
  )

Errors:
  rate = sum by (route, instance, status) (
    rate(http_requests_total{status=~"5xx"}[5m])
  )

Duration:
  histogram =
    http_request_duration_seconds_bucket{le="..."}
  p99 = histogram_quantile(0.99,
    sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
  )

RED is the canonical layer of instrumentation for a service. The instrumentation is typically one or two lines in the framework (HTTP server middleware, gRPC interceptor) and produces counters and histograms that the platform can ingest.

Why RED is the user-visible layer

Every dimension in RED is user-visible:

• Rate. A user-visible metric indirectly; users see spikes and dips, but rate itself is internal.
• Errors. A user-visible metric directly: errors are what the user sees.
• Duration. A user-visible metric directly: latency is what the user feels.

The combination of the three produces the user-visible state. USE alone does not produce the user-visible state. RED is what anchors the user-visible SLO pair (availability, latency).

How RED maps to SLOs

The canonical SLO pair for an HTTP API:

Availability SLO:
  SLI = sum(rate(http_requests_total{status!~"5xx"}[30d]))
      / sum(rate(http_requests_total[30d]))
  Target: SLI >= 0.999
  Window: 30 days

Latency SLO:
  SLI = sum(rate(http_request_duration_seconds_bucket{le="0.5"}[30d]))
      / sum(rate(http_request_duration_seconds_count[30d]))
  Target: SLI >= 0.99
  Window: 30 days

Both SLIs use the RED primitives (requests_total and request_duration_seconds_bucket). The SLO construction is mechanical once the primitives are instrumented.

USE vs RED

USE and RED are complementary, not competing. A complete platform layers:

• Service-level: RED on every HTTP API / RPC service / queue producer-consumer chain.
• Resource-level: USE on every host and infrastructure component.
• Network-level: NetUSE on every link.

The two methodologies operate at different layers. The on-call engineer consults the service-level RED first to find which service is failing; consults the resource-level USE second to find which resource is exhausted.

Choosing the right RED pair

For an HTTP API: rate + errors + duration (the full trio). For a job processor: rate of jobs + error rate + processing duration. For a queue producer: produced rate + publish error rate + publish latency. For a queue consumer: consumed rate + processing error rate + processing duration.

The shape varies by service type but the trio is the same.

RED instrumentation

Three instrumentation primitives the team must own:

1. Counter requests_total{status=...} — every completed request, labeled by status code.
2. Histogram request_duration_seconds{...} — every request’s duration.
3. Middleware/interceptor — the framework glue that records the metrics.

Most languages have a Prometheus client library that provides default middleware (Go: promhttp.InstrumentHandler*, Python: prometheus-client, Node: prom-client, Java: micrometer-registry-prometheus). The pattern is standard.

Production guidance

• Instrument RED on every service at the framework layer. The instrumentation should be automatic for every endpoint, not manual per endpoint.
• Use shared label sets across requests_total and request_duration_seconds so aggregation in PromQL works trivially.
• Pair every RED SLO with a corresponding RED alert: a burn-rate alert at 2× over a 1 h window is a typical page.
• Provide dashboards that put RED front-and-centre; USE is secondary.
• Audit the metrics regularly: an endpoint that does not appear in the RED dashboard is unmonitored.

Verification

You should be able to answer:

• What does RED stand for?
• Why is RED the user-visible layer that USE is not?
• Construct the SLO pair from RED primitives.
• Why are USE and RED complementary, not competing?

Quiz