The cost of telemetry - cardinality, sampling and collector limits

Every previous lesson in this part has been about getting data into a pipeline. This one is about the fact that you cannot afford all of it, and about the specific way that failing to decide turns your observability stack into an outage.

There are two independent risks and they need separating.

• Cost. Metrics, logs and traces are billed by volume, whether by a vendor or by the storage and machines you run yourself. The bill grows with the fleet, so a configuration that is comfortable on ten hosts is not on five hundred.
• Stability. A cardinality explosion does not degrade gracefully. It exhausts memory in the backend, and it does so during whatever incident produced the new labels - so you lose your telemetry at the exact moment you need it.

How cardinality multiplies

A time series is one metric name combined with one unique set of attribute values. The count is the product of the distinct values of every attribute, not the sum.

A single metric http.server.duration with attributes http.route (40 routes), http.status_code (8 observed), http.method (4) and host.name (500) is

40 x 8 x 4 x 500 = 640,000 series

Add one attribute holding a customer id with 2,000 values and the same metric becomes 1.28 billion series. Nothing about the configuration change looks large.

The host metrics arithmetic

The hostmetrics receiver is where most Linux fleets meet this problem first, because the process scraper looks harmless and is not.

1. Core scrapers - cpu, memory, load, disk, filesystem, network - produce roughly 200 to 400 series per host depending on how many disks and interfaces it has
2. The process scraper produces a set of series per process, attributed by process name and pid
3. A host running 300 processes therefore contributes thousands of series on its own, and a fleet of 500 such hosts contributes millions
4. Process attributes such as the command line are themselves unbounded, because they contain arguments
5. Restarts change pids, so yesterday series stop and new ones start - the storage holds both

The fix is not to abandon process metrics but to bound them: include only the processes you would act on, and delete the attributes that carry the unbounded part.

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      load:
      filesystem:
      network:
      process:
        include:
          names: [nginx, postgres, myapp]
          match_type: strict

processors:
  attributes/trim_process:
    actions:
      - key: process.command_line
        action: delete
      - key: process.pid
        action: delete

Dropping data at the agent, not at the backend

Filter as early in the pipeline as possible. Data dropped at the gateway has already cost you the network egress from 500 hosts and the memory to hold it; data dropped at the backend has cost you ingestion as well, which is usually the expensive part of a bill.

processors:
  # Drop whole metrics you have decided you do not need.
  filter/metrics:
    error_mode: ignore
    metrics:
      metric:
        - 'name == "system.paging.faults"'

  # Drop health-check spans: high volume, no diagnostic value.
  filter/spans:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/health"'

  # Collapse an id-bearing path into a bounded route label.
  transform/routes:
    metric_statements:
      - context: datapoint
        statements:
          - replace_pattern(attributes["http.route"], "/orders/[0-9]+", "/orders/{id}")

Traces are not free

A span is a structured event with attributes, and a single request commonly produces tens of them. Instrumenting a service that handles 2,000 requests per second at 20 spans per request is 40,000 spans per second - and at a few hundred bytes each, that is terabytes per day before anyone has looked at one.

Sampling is therefore not an optimisation, it is part of the design. There are two kinds and the difference matters.

processors:
  # Head sampling: cheap, dumb, keeps 5% of everything.
  probabilistic_sampler:
    sampling_percentage: 5

  # Tail sampling: keeps all errors, all slow traces, 1% of the rest.
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

If you sample traces hard, derive your rate and latency metrics from the spans before sampling, using the spanmetrics connector. That keeps exact request rates and error rates while keeping only a fraction of the traces themselves.

connectors:
  spanmetrics:
    dimensions:
      - name: http.method
      - name: http.status_code

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [spanmetrics, otlp/tempo]
    metrics/from_spans:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [prometheus]

Note the trap in that snippet: every entry under dimensions becomes a metric attribute, so spanmetrics is itself a cardinality multiplier. Adding http.url there recreates the unbounded-label problem in the one place you were trying to avoid it.

Watch the Collector admit it is dropping data

The Collector exposes its own metrics on port 8888 by default. These are the ones worth scraping and alerting on.

$ curl -sS http://127.0.0.1:8888/metrics | grep -E 'otelcol_(receiver_refused|exporter_send_failed|exporter_queue|process_memory)' | head

otelcol_receiver_refused_spans{receiver="otlp"} 18422
otelcol_exporter_send_failed_spans{exporter="otlp/tempo"} 0
otelcol_exporter_queue_size{exporter="otlp/tempo"} 4812
otelcol_exporter_queue_capacity{exporter="otlp/tempo"} 5000
otelcol_process_memory_rss 1932735283

Three signals, three different conclusions:

• otelcol_receiver_refused_* rising: the Collector is over its memory limit and is pushing back. Either the input grew or the limit is too low.
• otelcol_exporter_send_failed_* rising: the backend is rejecting or unreachable. Check its rate limits before assuming a network fault - a backend that returns HTTP 429 because you exceeded an ingestion quota looks identical to one that is down.
• otelcol_exporter_queue_size near capacity: sustained overload rather than a spike. Raising the queue only buys minutes; the input rate has to come down or the backend has to grow.

Knowledge check