Choosing the Right Methodology for Each System

Different methodologies fit different classes of system. The discipline is to choose the right methodology for each component, not to apply one everywhere.

This lesson is the bridge from “USE on hosts, RED on services” to “apply the right methodology to each system in the production topology.”

Methodology inventory

The course teaches RED and USE in production depth. The other methodologies are referenced at a glance.

Decision tree

For a new component, the question is:

1. Does the component expose a request/response to a user? Use RED on the component, USE on its host(s).
2. Is the component a host, a database, a broker — a resource- bounded system? Use USE.
3. Is the component a network link? Use NetUSE.
4. Is the component stateful with consistency requirements? Use USE on the host, plus a custom Consistency metric.

The answer is rarely “use one methodology on the whole system.” The disciplined choice is per-component.

Common patterns

API gateway

Public-facing HTTP endpoint (RED on gateway + a blackbox probe)
   |
Backend services (RED on each)
   |
Shared hosts (USE on each)

The gateway is a service (RED). The backends are services (RED). The hosts beneath them are resources (USE). The public HTTP path is also a target for blackbox probing.

Database

Application (RED)
   |
Database engine (USE on host + custom metrics for queries)
   |
Storage volume (USE)
   |
Replication (custom — replication lag, replica health)

A database has more dimensions than RED or USE alone capture. The engine has resource metrics (memory, disk, CPU), query metrics (duration, locks, replication), and replication metrics. The custom query metrics are part of USE on the engine internals.

Message queue

Producer (RED for the producer side: rate, errors, send latency)
   |
Queue broker (USE on broker, custom for queue depth)
   |
Consumer (RED for the consumer side: rate, errors, process latency)

A queue is service-shaped at both ends and resource-shaped in the middle. All three layers get instrumentation; each layer uses the methodology that fits.

CDN / DNS

Service-side (RED — origin returns data, latency, errors)
   |
Network-side (NetUSE — bandwidth to edge)
   |
External blackbox probe (HTTP probe of the public endpoint)

A CDN involves network edges, an origin server, and DNS. The correct set is RED on the service + NetUSE on links + a blackbox probe from outside the platform’s trust zone.

Pitfalls

1. Over-instrumentation. A team instruments every metric the exporter exposes, producing 50,000 time series and 100 panels. The on-call engineer cannot find the relevant panel during an incident. Disciplined instrumentation is less, well-chosen, and queried.
2. Under-instrumentation. A team instruments RED on every service but no USE on the hosts. A memory leak that drives the service to OOM is not flagged until it triggers an error. The discipline is to add USE on every host and every database.
3. Mismatched label sets. A team instruments requests_total and request_duration_seconds_bucket with different label sets (requests_total has status but request_duration does not). PromQL aggregation becomes painful. Shared labels are part of the design.
4. Methodology-as-dashboard. A team builds a single “USE dashboard” and a single “RED dashboard” and stops. Each service has its own dashboards; each host has its own; dashboards aggregate per service / per host, not per methodology.

Production guidance

• Choose the methodology when the service is designed, not when the dashboard is built. The choice should appear in the design document and the team’s “instrumentation checklist” reference.
• Promote RED instrumentation to the framework level. Most Prometheus client libraries provide default middleware; configure once per service.
• Promote USE instrumentation to the host baseline. node_exporter on every host; the methodology is automatic.
• Audit instrumentation quarterly. Every endpoint should be RED-instrumented; every host should be USE-instrumented; failures of either are remediation tickets.
• Document the methodology choice per service. A service page on the runbook site should say “RED: rate / errors / duration per endpoint; USE: integrated from host metrics.”

Verification

You should be able to answer:

• What is the decision rule for picking a methodology per component?
• How does a database differ from an API in instrumentation?
• What is the failure mode of over-instrumentation vs under- instrumentation?
• How does the discipline hold up across a complex topology?

Quiz