Monitor Everything Without Purpose

A platform team turns on the default instrumentation of every service. Within a quarter the Prometheus server holds twenty-six million active series. The Grafana dashboards load in seven seconds. Three quarters of the dashboards are never opened; the quarter that are opened are not the quarter anyone would build if they started again. The on-call engineer scrolls past two hundred panels to find the one that tells them what is wrong. The collection budget is gone and the investigation capability has not moved.

This is the monitor everything anti-pattern. It is not a mistake any one person makes. It is the natural outcome of “let us collect first and decide later,” repeated across services, deployments, and quarters, until the platform’s own telemetry becomes a performance problem.

What it is

The monitor everything anti-pattern is the practice of collecting telemetry without a question it answers. Every metric is emitted because the library emits it by default. Every log line is shipped because the application logs it by default. Every trace span is sampled at 100% because nobody turned the probability down. The platform grows. The questions do not.

Compare to the alternative: instrument for a question. A service-level indicator (SLI) is a numeric measure of a user- visible property of the system. Latency, error rate, throughput, and saturation are the canonical four. Each metric that lives in the platform should be answerable to “which SLI does this metric support?” If the answer is “none, but it might be useful,” it is a candidate for either a recording rule (compressed, cheaper to query) or for removal.

The trade-off is honest. Aggressive pruning costs you the unknown-unknown investigation: the failure mode you did not anticipate. Unbounded collection costs you the known-known investigation: the failure mode you anticipated but cannot find because the platform has buried it. Production observability practice sits between the two extremes. The lesson is about where to draw the line.

Why a sysadmin cares

The cost of monitoring everything is paid in three places, all of which hit the operator on call.

Storage cost. Prometheus stores every active series in memory and in the write-ahead log on disk. Twenty-six million series at a two-byte-per-sample scrape interval of fifteen seconds consumes roughly one gigabyte per scrape, written to a TSDB head block that must fit in RAM. The disk cost scales linearly with the series count. The RAM cost scales worse, because the head block is replayed on every Prometheus restart.

Query cost. PromQL evaluation over a high-cardinality label set blocks the query engine. A single dashboard with ten panels, each aggregating over a label set of millions, takes longer to load than the on-call engineer is willing to wait. The on-call engineer stops opening the dashboard. The dashboard becomes wallpaper.

Investigation cost. When the platform holds a million metrics and only ten thousand are useful, finding the useful ten thousand during an incident is the work. The on-call engineer does not have that time. The incident lasts longer.

How it works

The mental model is that every metric is a contract. The producer (the application) commits to emit it; the consumer (the dashboard, the alert, the runbook) commits to use it. A metric without a consumer is an orphan; it consumes resources and contributes nothing.

Application
     |
     v
+------------------------+
| Default instrumentation|
|  (library emits it)    |
+------------------------+
     |
     |  MetricRelabelConfig: keep | drop
     v
+------------------------+
| Prometheus / OTel      |
|  Collector / Alloy     |
+------------------------+
     |
     |  Sample probability, drop stages
     v
+------------------------+
| Storage                |
|  (TSDB / Loki / Tempo) |
+------------------------+
     |
     |  Query: which dashboards and alerts consume this metric?
     v
+------------------------+
| Consumers              |
|  (dashboard, alert,    |
|   recording rule)      |
+------------------------+
     |
     v
  Orphans are a write-only contract

The consumer question is the lever. If a metric has no consumer, the relabel rule drops it at the agent. If a consumer exists but the cardinality is too high, the relabel rule drops the offending label. If the consumer is a long-tail investigation that fires once a quarter, the recording rule compresses the metric into a pre-aggregated form that the dashboard reads instead.

How to configure it

The configuration is two parts. The Prometheus side keeps the metrics that justify their storage cost. The Alloy side keeps the logs that justify their index cost. The two rulesets together define the collection budget.

# /etc/prometheus/prometheus.yml (Prometheus 2.55.x)
scrape_configs:
  - job_name: app
    static_configs:
      - targets: ['app:9100']
    # Step 1: drop entire metrics that have no consumer.
    # The list is maintained from the dashboard / alert audit.
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: '(go_gc_.*|go_memstats_.*_bytes|process_.*_seconds_total)'
        action: drop
        # Drop the runtime metrics that no dashboard uses.
        # They are emitted by every Go application by default
        # and accumulate to millions of series at scale.

      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        action: drop
        # Buckets are useful for histograms but expensive. If
        # the consumer is a heatmap, keep buckets. If the
        # consumer is a quantile, keep buckets. Otherwise drop.

      # Step 2: cap cardinality on the labels we keep.
      - source_labels: [http_request_duration_seconds_bucket, le]
        regex: 'http_request_duration_seconds_bucket;.*'
        action: drop

The matching agent configuration drops the same shape at the log and trace boundary:

# /etc/alloy/config.alloy (Alloy current)
loki.relabel "enforce_budget" {
  forward_to = loki.process.parse.receiver

  # Drop labels whose value space is not bounded by the design.
  rule {
    action = "labeldrop"
    regex  = "(request_id|trace_id|user_id|session_id|customer_id)"
  }

  # Cap the length of any single label value at 256 chars.
  rule {
    action       = "labelmap"
    regex        = ".+"
    replacement  = ""
  }
}

otelcol.processor.tail_sampling "enforce_budget" {
  decision_wait = "10s"
  num_traces    = 100000
  policies {
    name        = "errors-only"
    type        = "status_code"
    status_code { match_codes = ["ERROR"] }
  }
  policies {
    name   = "low-volume-head"
    type   = "probabilistic"
    probabilistic { sampling_percentage = 1 }
  }
  forward_to = otelcol.exporter.otlp.default.input
}

The two configurations share a discipline: the consumer question drives the rule, the rule is documented, and the document is reviewed every release.

How to validate it

Three commands confirm the budget is in force.

# 1. Active series count by metric. The metric with the
# largest count is the first candidate for review.
# Severity: READ-ONLY
promtool query instant http://prometheus:9090 \
  'count by (__name__) ({__name__=~".+"})' \
  | sort -k2 -n -r | head -10

Expected output (illustrative):

http_requests_total{method="GET",status="200"} 184
http_requests_total{method="POST",status="500"} 91
app_info{version="1.4.2"} 73
...

A count in the millions for a single metric is the signal that a label is unbounded.

# 2. Top labels by cardinality per metric. Confirms that the
# label the budget caps is the one consuming the budget.
# Severity: READ-ONLY
promtool query instant http://prometheus:9090 \
  'topk(5, count by (method, status) (http_requests_total))' \
  | head -20

# 3. Dropped samples counter. Confirms the rule is firing.
# Severity: READ-ONLY
curl -s http://prometheus:9090/metrics \
  | grep '^prometheus_target_scrapes_exceeded_label_limits_total'

A non-zero counter confirms the rule is dropping. A zero counter after a release that changed the relabel config means the rule is not matching, which is itself a finding.

How it can fail

Five shapes recur when the budget is not enforced.

1. The default-emission drift. A Go service upgrades its prometheus/client_golang library; the new version emits go_gc_cycles_total_gc_cycles_per_ns by default. The metric fans out across every instance. The team does not notice for a quarter.
2. The route-as-label explosion. A frontend ships the request path as a label. The path space is unbounded; one customer’s UUID appears in the path; one UUID becomes one series; one million customers become one million series.
3. The histogram bucket stampede. Every team adopts the default histogram bucket set. The bucket count multiplied by the label count produces ten million series per service. The recording rule that was supposed to compress them was never written.
4. The orphan metric. A dashboard is deleted; the metric the dashboard consumed is not deleted. The metric continues to be scraped, indexed, and stored. The team has forgotten why it exists.
5. The tail-sample trap. Trace sampling is set at 100% on the premise that “we will turn it down when we have to.” The team never turns it down. Tempo stores every span.

How to troubleshoot it

1. Identify the largest metric by active series count
   (promtool query above)
        |
        v
2. Find the consumer (grep dashboards, alerts, recording rules)
        |
        v
3. If a consumer exists, ask: is the cardinality bounded?
        |
        +-- yes -> keep, document, alert on drop counter
        |
        +-- no  -> drop the unbounded label
        |
        v
4. If no consumer exists, drop the metric at the agent
        |
        v
5. Verify the drop counter is non-zero; verify storage growth
   rate has flattened
        |
        v
6. Document the rule and the consumer; review on the next
   release

Security implications

The collection budget is also a data budget. A metric that contains a user identifier is a compliance exposure even when the identifier is hashed. The label rule that drops user_id from the metric set is the same rule that protects the platform from a data-handling incident. The audit is the same audit: top labels by cardinality, top labels by sensitivity. The two should be run together.

Performance implications

The performance ceiling of a Prometheus server is set by the worst metric on the worst scrape target. One unbounded metric on one target raises the WAL replay time for every target on the server. The cost is shared even when the fault is local. The performance ceiling of a Loki cluster is set the same way: one bad label on one tenant raises query latency for every tenant. The collection budget is not a per-team cost; it is a shared- infrastructure cost.

Verification

You should now be able to answer:

• What is the consumer question, and why does it drive the collection budget?
• What are the three costs of the monitor everything pattern and which is paid first?
• Where in the pipeline should the cardinality cap be enforced, and where is the backstop?
• What does a healthy top labels by cardinality output look like?

Quiz