Telemetry Missing

The page does not fire. The on-call engineer opens the expression in Grafana, runs the query, and sees nothing. No series, no value, no error. The alert should have paged an hour ago. Something between the exporter and the rule evaluator has dropped every sample for the label the rule was written against. The rule is correct. The condition is irrelevant. The telemetry is gone.

This lesson is the layer below the rule: the exporter, the scrape, the relabel, and the TSDB. When the upstream telemetry is missing, no alert that depends on it can fire, and every downstream investigation starts with the same first question. Is the metric actually present?

What “missing telemetry” is

Missing telemetry in production terms is the absence of a sample series in the TSDB that the alerting rule expected to evaluate against. The condition may be true; the team cannot tell because the data is not in storage.

Four shapes produce missing telemetry, distinct from each other:

1. Exporter process is not running. The metric-generating process is dead, has crashed, or has never been deployed on the target host.
2. Scrape target is not in scope. Service discovery did not pick up the target, or the scrape_configs.job_name does not match the SD output, or a DNS resolution returned nothing.
3. Scrape returns an error. The endpoint returned 4xx, 5xx, or non-Prometheus text format. Prometheus records the scrape failure but produces no new samples for that scrape.
4. Relabel drops every series. A metric_relabel_configs block discards samples based on a label matcher before they are stored.

A fifth shape exists for teams that use a remote-write pipeline: the samples arrive at the local Prometheus but are filtered or misrouted before reaching the long-term store. This lesson focuses on the four shapes that affect a single Prometheus process.

Why a sysadmin cares

Missing telemetry is the silent predecessor to a non-firing alert. The team’s posture on alerts depends on the rule file and on the scrape stack. A perfect rule against an exporter that does not exist is a no-op. A complete scrape config against a relabel rule that drops everything is also a no-op.

The cost of missing telemetry is asymmetric with the cost of visible telemetry failure. A scrape that returns 500 produces a scrape_samples_scraped sample of zero and an up{job="..."} == 0 sample. Both are visible in dashboards and alert. A relabel that drops every series produces scrape_samples_scraped of exactly what was returned before the drop, plus no series in the rule’s selector. The drop is invisible unless the team has a separate “scrape samples returned vs stored” panel.

The diagnostic must therefore be specific to the layer. A generic “Prometheus is broken” claim is not actionable.

How it works

A Prometheus scrape is a sequence of stages between the SD output and the TSDB. Each stage produces an effect that is visible at a different metric endpoint.

+----------+    +----------+    +----------+    +----------+
| Service  |--->| Relabel  |--->| Scrape   |--->| Metric    |
| discovery|    | (target) |    | HTTP     |    | relabel   |
+----------+    +----------+    +----------+    +----------+
  SD query        drop / keep      GET /metrics    drop / keep
  yields          labels on        returns         labels on
  target list     each target      200 / 5xx       each sample
                                  parse text
                                       |
                                       v
                                  +----------+
                                  | TSDB     |
                                  | append    |
                                  +----------+

Each stage has its own success signal and failure signal in Prometheus’s own metrics:

• Service discovery. prometheus_sd_\{component\}_sd_configs_failed_total counts SD calls that returned an error. prometheus_sd_discovered_targets enumerates the targets currently in scope.
• Target relabeling. Targets that fail relabel are dropped silently. The visible signal is the count of targets after relabel against the count before, on a per-job basis, visible in /api/v1/targets.
• Scrape. up{job="..."} is 1 on success, 0 on failure. scrape_samples_scraped returns the count of samples before metric relabel. scrape_duration_seconds measures scrape duration.
• Metric relabel. The drop here is observable only as scrape_samples_post_metric_relabeling compared to scrape_samples_scraped. A delta of hundreds without a matching delta in /api/v1/series for the metric name indicates a drop.

The implicit fifth stage is remote-write, which has its own metric on the local process: prometheus_remote_write_\{success|failure\}_samples_total per remote-write target.

The most common cause

In a healthy platform, in roughly half of “missing telemetry” investigations the team runs, the cause is the scrape target being down. Most teams discover this through up{job="..."} == 0 first. The second most common cause is a relabel dropping everything for a label pattern the team forgot was in scope. The remaining quarter splits across SD errors, scrape errors, and remote-write misroutes.

The diagnostic order

The diagnostic front-to-back mirrors the pipeline. Run it in this order; do not skip ahead.

+----------------------------------------+
| 1. Is the target up?                   |
|    up{job="X"} == 1 ?                  |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 2. Is the target discovered?           |
|    /api/v1/targets?state=active       |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 3. Was the scrape attempted?           |
|    scrape_samples_scraped > 0 ?        |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 4. Did metric relabel pass?            |
|    scrape_samples_post_relabel?        |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 5. Is the metric in storage?           |
|    /api/v1/series?match[]=metric       |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| 6. Is remote-write delivering?         |
|    _remote_write path?                 |
+--------------------+-------------------+
                     |
                     v
+----------------------------------------+
| Find the first stage that fails.        |
+----------------------------------------+

Stage 1 is up{job="X"}. A 0 here means the scrape either did not return a 200 or returned malformed text. This is the most common production condition.

Stage 2 is /api/v1/targets. If the target is not in the active list at all, service discovery did not produce it. The fix is in the scrape_configs: block or upstream SD.

Stage 3 is scrape_samples_scraped. If the value is positive but the metric name the rule expects is not in the series returned, the exporter is the wrong one or has been upgraded without updating the rule.

Stage 4 is scrape_samples_post_metric_relabeling. A drop between stage 3 and stage 4 indicates a metric relabel filter has consumed the labels the rule cares about. Most often, this is a drop action on a label pattern that no longer matches.

Stage 5 is /api/v1/series?match[]=. If the metric name is in storage but the rule’s expression returns no vector, the rule’s label matchers do not match the live series. That is the Stage-2 cause of lesson 01; this lesson assumes the rule already matches and the underlying metric is missing.

Stage 6 is remote-write. For teams running Mimir, Thanos, or a long-term store receiver, the local Prometheus may have the data while the long-term store does not. The metric prometheus_remote_write_\{success|failure\}_samples_total on the local process is the indicator.

Under the hood

A scrape job in Prometheus is a single object that controls both the SD and the scrape. The same scrape_configs block holds relabel_configs for the target and metric_relabel_configs for the samples. The two chains run at different points in the pipeline, and a rule that breaks without an obvious cause often breaks because of one of them.

The flow on a scrape:

1. Service discovery runs. For each SD, a list of target groups is produced.
2. Each target is relabelled through relabel_configs. Targets whose relabel chain ends in action: drop are excluded from the scrape. Targets whose labels are rewritten or filtered have only the post-relabel labels.
3. Prometheus scrapes /metrics for each surviving target.
4. The response is parsed into samples.
5. Each sample goes through metric_relabel_configs. A sample whose labels do not match the keep pattern is dropped.
6. Surviving samples are appended to the head block.

Steps 2 and 5 are both silent. A team running only up{job} as their health signal sees neither of them as a problem, because the scrape technically completes.

How to configure it

A scrape_configs block that protects against the common missing-telemetry causes. Annotated example:

scrape_configs:
  - job_name: checkout-svc
    scheme: http
    metrics_path: /metrics
    scrape_interval: 30s
    scrape_timeout: 10s
    # Honor Kubernetes pod annotations for service discovery.
    kubernetes_sd_configs:
      - role: pod
    # Stage 1 relabel: pull the metadata the rule wants
    # out of the SD output before the scrape URL is built.
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: replace
        target_label: app
      - source_labels: [__meta_kubernetes_pod_phase]
        regex: Running
        action: keep
      # If `app` ends up empty, drop the target entirely.
      - source_labels: [app]
        regex: ^$
        action: drop
    # Stage 2 relabel: drop synthetic probe traffic before
    # storage; keep production only.
    metric_relabel_configs:
      - source_labels: [synthetic]
        regex: "true"
        action: drop
      - source_labels: [environment]
        regex: production
        action: keep

The two relabel chains are visible. The first chain keeps only running pods with a non-empty app label. The second chain drops synthetic traffic and keeps only production. A team investigating a “missing telemetry” alert should look at both chains and the exact regex patterns that govern survival.

For an OpenTelemetry Collector pipeline that mirrors the same telemetry into the long-term store, the equivalent configuration in Grafana Alloy looks like:

prometheus.scrape "checkout" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.relabel.checkout.receiver]
  job_name   = "checkout-svc"
  scrape_interval = "30s"
  metrics_path = "/metrics"
}

prometheus.relabel "checkout" {
  rule {
    action       = "keep"
    source_labels = ["__meta_kubernetes_pod_phase"]
    regex         = "Running"
  }
  forward_to = [prometheus.remote_write.default.receiver]
}

The prometheus.relabel component runs both the target relabel and the metric relabel through a single rule chain. The trade-off is that Alloy’s keep/drop regex applies to both chains unless the component is set to rule \{ target_label = "__metric__" ... \}.

How to validate it

Six checks, one per stage. Run them in order before changing configuration.

Stage 1: target up.

curl -s 'http://prometheus:9090/api/v1/query?query=up{job="checkout-svc"}' \
  | jq '.data.result[] | {instance: .metric.instance, value: .value[1]}'

Expected for a healthy job:

{"instance":"10.4.2.18:8080","value":"1"}
{"instance":"10.4.2.19:8080","value":"1"}

A 0 indicates a failed scrape and the target is the next thing to investigate.

Stage 2: target discovered.

curl -s 'http://prometheus:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[] | select(.labels.job=="checkout-svc") | {scrapeUrl, lastError}'

If the target list is empty, SD did not produce it. Check the scrape_configs block for an SD that has the right selector.

Stage 3: scrape returned samples.

curl -s 'http://prometheus:9090/api/v1/query?query=scrape_samples_scraped{job="checkout-svc"}' \
  | jq '.data.result[] | {instance: .metric.instance, samples: .value[1]}'

A value of zero means the exporter returned an empty payload or the scrape failed at the HTTP step.

Stage 4: metric relabel pass.

curl -s 'http://prometheus:9090/api/v1/query?query=scrape_samples_post_metric_relabeling{job="checkout-svc"}' \
  | jq '.data.result[] | {instance: .metric.instance, samples: .value[1]}'

A large delta between stages 3 and 4 indicates the metric relabel chain dropped most of the samples.

Stage 5: metric in storage.

curl -s 'http://prometheus:9090/api/v1/series?match[]=http_requests_total' \
  | jq '.data[] | {job: .job, instance: .instance}'

If the rule expects a job="checkout-svc" label and the series list shows job="checkout-app", a relabel rewrite has renamed the label.

Stage 6: remote-write success.

curl -s 'http://prometheus:9090/api/v1/query?query=rate(prometheus_remote_write_samples_total[5m])' \
  | jq '.data.result[] | {remote: .metric.remote_name, rate: .value[1]}'

A persistently low or zero rate against the long-term store’s remote-write target indicates the long-term path is broken even though the local TSDB has data.

How it can fail

Six failure shapes, each tied to a stage:

1. Exporter process not running. The metric-generating binary is dead, hung, or not yet deployed on the host. Symptom: up{job="X"} is 0; the SD target list is empty for the job.
2. Network ACL blocks the scrape. A firewall rule denies Prometheus’s source IP from reaching the target’s port. Symptom: scrape duration is the timeout (10 s) and up\{job="X"\} == 0; the Prometheus log records context deadline exceeded for the scrape.
3. relabel_configs drop regex matches too much. A new action: drop introduced in a config edit drops every target whose label does not match the keep pattern; the scrape_configs block is valid but yields no surviving targets. Symptom: /api/v1/targets?state=active is empty for that job; the configuration passes promtool check config.
4. metric_relabel_configs discards every sample. A source_labels reference is wrong (e.g., environment=production instead of environment="production" in YAML quoting). Symptom: scrape_samples_post_metric_relabeling is zero or near-zero while scrape_samples_scraped is in the hundreds.
5. DNS resolution returns the wrong address. SD picks up a Kubernetes service by name; the service’s cluster IP is pointing at a now-removed endpoint. Symptom: a series of “no such host” errors in the Prometheus log for the scrape.
6. Remote-write credentials are stale. The long-term store rotates its auth tokens, but the local Prometheus remote_write config still uses the old bearer token. Symptom: prometheus_remote_write_samples_total shows a non-zero rate of failures; the local TSDB has data; the query against the long-term store returns nothing.

How to troubleshoot it

Follow the six steps. The order is the order of the pipeline.

1. Step 1, target up. If up{job="X"} == 0, the scrape failed; next, look at the Prometheus log for the target’s last scrape error message.
2. Step 2, target discovered. If the target list is empty, run promtool check config against prometheus.yml. A typo in kubernetes_sd_configs.role produces an SD error but the scrape config still parses.
3. Step 3, scrape returned samples. If scrape_samples_scraped is zero, the response was empty; curl against /metrics directly to confirm.
4. Step 4, metric relabel pass. If scrape_samples_post_metric_relabeling is much smaller than scrape_samples_scraped, the relabel chain dropped samples. Print the chain by listing metric_relabel_configs and applying each rule manually.
5. Step 5, metric in storage. If the metric is in storage but the rule’s expression returns no vector, the label matchers in the rule file are wrong against the live labels. This is the Stage-2 cause of lesson 01; fix the rule expression, not the scrape.
6. Step 6, remote-write delivering. For long-term store issues, the local Prometheus is the data source to validate. Check prometheus_remote_write_* and the auth: block against the long-term store’s API.

Security implications

A scrape URL can carry credentials in the form of http://user:pass@host:port/metrics. Prometheus 2.55 documents this as a known support tier; basic auth tokens appear in the URL in process environment dumps and logs.

The bigger security implication is that a target whose scrape URL points at a service you do not control is an exfiltration channel. A malicious exporter can post arbitrary labels and values into the TSDB. Review scrape_configs for non-corporate URLs as a routine security check.

Relabel configurations have the privilege to rewrite labels, including labels added by service discovery. A misconfigured relabel can shadow the production job label with a user-controlled value, which then becomes a routing input to AM. Treat relabel as privileged code.

Performance implications

Every scrape burns CPU proportional to the number of samples returned. A relabel chain that drops samples after the parse has already paid the parse cost. Long-relabel chains are an unforced expense.

The head block has a fixed-size budget per series and block size. When the block fills, Prometheus compacts it and starts a new one. A scrape that returns ten times the expected samples forces compaction to run on a faster cadence. The write-throughput of the local process is bounded by this cycle.

Remote-write multiplies the cost. A long-term store connection that is slow or busy doubles the work the local process does for every scrape.

Production guidance

• Alert on up{job="X"} == 0. This is the cheapest health signal Prometheus produces and it catches the most common cause of missing telemetry at the loudest moment.
• Alert on a delta between scrape_samples_scraped and scrape_samples_post_metric_relabeling. A sustained delta is a relabel-chain defect that would otherwise be invisible.
• Pin scrape intervals to the alert evaluation interval. Scrape intervals shorter than the evaluation interval produce samples that the rule cannot use; longer scrape intervals produce rules that miss half the breaches.
• Restrict the cardinality of relabel chains. A source_labels reference that resolves to a per-request label produces a target explosion and a storage cost.

Verification

You should now be able to answer:

• What are the six stages from service discovery to remote-write at which telemetry can go missing?
• Which stage accounts for the majority of missing-telemetry cases, and how do you recognise it?
• How do you tell the difference between a target that is not discovered, a target whose scrape failed, and a relabel that dropped every sample?
• What is the metric that exposes a metric-relabel drop?

Quiz