Scrape Failure Detection

A Grafana dashboard shows the production HTTP error rate holding steady at 0.4 percent. For three weeks the number has been identical to four decimal places. The application team admires the stability and moves on. The actual error rate is twelve percent - the metrics stopped arriving at 14:00 three weeks ago when a single misconfigured relabel rule stripped the only label the dashboard grouped by. up is still 1. The target is “up”. The data is not.

Scrape failure detection is the discipline of distinguishing “the scrape succeeded” from “the scrape produced useful data”. The two are not the same.

What it is

A scrape is the HTTP request Prometheus sends to a target’s /metrics endpoint on every scrape_interval. The scrape either returns a parseable Prometheus exposition or it does not. The result is encoded into four families of metrics:

• up{job, instance} — one sample per target. Value 1 means the last scrape succeeded. Value 0 means it failed for any reason (connection refused, timeout, 4xx, 5xx, parse error).
• scrape_duration_seconds{job, instance} — histogram of scrape wall-clock time. Includes connection, TLS handshake, HTTP round-trip, and body read.
• scrape_samples_scraped{job, instance} — number of samples the target returned.
• scrape_series_added{job, instance} — number of new series this scrape introduced. A persistent non-zero rate is a cardinality leak.

The right approach is to alert on up == 0 for sustained failures (not flapping), watch scrape_duration_seconds for slow-but-successful scrapes, and watch scrape_series_added for cardinality growth.

Why a sysadmin cares

Scrape failures are the most common silent failure in a production Prometheus stack. The target is up from the operating system’s point of view. The application is up. The dashboard is green. The data is stale, missing, or wrong. The first time anyone notices is when an investigation opens and the relevant panel shows a flat line at the value from three weeks ago.

A second reason: a target that is “up” but slow is worse than a target that is down. A slow target blocks scrape pool slots, pushes the next scrape interval, and eventually causes overlap. The operator does not get a page; the operator gets delayed alerts and stale dashboards.

How it works

   scrape_interval (15s default)
          |
          v
+----------------------+
| scrape pool          |   group of targets sharing
| (per job)            |   scrape_interval, timeout,
+----------------------+   relabel, honor_labels, etc.
   |   |   |   |
   v   v   v   v
+---+ +---+ +---+ +---+
| T | | T | | T | | T |     T = target instance
+---+ +---+ +---+ +---+
   |       |       |
   |       |       +-- scrape_duration_seconds{instance=T3}
   |       +---------- scrape_samples_scraped{instance=T2}
   +------------------ up{instance=T1}  (1 or 0)

Each scrape job is one pool. Targets within a pool are scraped sequentially (or concurrently with a small worker count) on the pool’s scrape_interval. The pool’s scrape_timeout (default 10s) bounds each individual scrape. A target that takes longer than the timeout is treated as a failure and up is set to 0.

The four metrics above are emitted per scrape and labelled with the target’s job and instance. The labels are inherited from the scrape_configs block; that is why a relabel rule that drops or rewrites labels changes what shows up in up{...}.

Under the hood

How to configure it

The right alert (not the wrong one)

The common mistake is to alert on up == 0 with for: 0m. This pages on every transient blip. The right shape is to alert after a short for window AND to alert on the rate of failures over a longer window. Both together catch persistent outages without paging on a single dropped packet.

# /etc/observer/rules/scrape-failures.yml
groups:
  - name: scrape-failures
    rules:
      # 1. Persistent failure. Target has been down for 2 minutes.
      #    Pages if a healthy target stops responding entirely.
      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Target {{ $labels.job }}/{{ $labels.instance }} down"
          description: |
            Target has been failing scrapes for 2 minutes.
            Inspect the Prometheus log for the scrape error
            and the target's own logs for HTTP / TCP errors.

      # 2. Slow scrape. The scrape is succeeding but takes more
      #    than 80 percent of scrape_timeout. Catches the
      #    "almost broken" shape before it breaks.
      - alert: TargetScrapeSlow
        expr: |
          scrape_duration_seconds
            > (0.8 * 10)
            # scrape_timeout default is 10s; adjust if overridden
        for: 10m
        labels:
          severity: warning
          team: platform

      # 3. Cardinality leak from a single target. A persistent
      #    non-zero rate of new series added is the canonical
      #    "an exporter just shipped a bug" signal.
      - alert: TargetCardinalityLeak
        expr: rate(scrape_series_added[1h]) > 100
        for: 30m
        labels:
          severity: warning
          team: platform

      # 4. Zero samples returned. Target is "up" but the body
      #    contains no metrics. Almost always a relabel / path /
      #    auth misconfiguration.
      - alert: TargetReturningNoSamples
        expr: |
          up == 1
          AND on(job, instance) scrape_samples_scraped == 0
        for: 5m
        labels:
          severity: warning
          team: platform

      # 5. Scrape flapping. Target oscillates between up and
      #    down rapidly. Network or auth credential is rotating.
      - alert: TargetFlapping
        expr: |
          changes(up[10m]) > 6
        for: 10m
        labels:
          severity: warning
          team: platform

Honour the scrape timeout

Make the scrape timeout shorter than the scrape interval. The default of 10s on a 15s interval leaves 5s of slack, which is enough for one slow target to delay the next scrape and start a cascade. For high-fanout jobs, reduce the timeout to 5s.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: node
    scrape_interval: 15s
    scrape_timeout: 5s       # leaves 10s of slack
    static_configs:
      - targets: ['node-1:9100', 'node-2:9100']

How to validate it

Confirm up is being emitted for every target:

# READ-ONLY
curl -s 'http://prom-primary.internal:9090/api/v1/query?query=up' \
  | jq '.data.result[] | {job: .metric.job, instance: .metric.instance, value: .value[1]}'

A healthy output shows 1 for every target across the fleet. Any 0 is a failure that needs investigation.

Confirm scrape duration is within budget:

# READ-ONLY - p95 scrape duration per job, last 5 minutes.
curl -s 'http://prom-primary.internal:9090/api/v1/query' \
  --data-urlencode 'query=histogram_quantile(0.95, sum by (job, le) (rate(scrape_duration_seconds_bucket[5m])))' \
  | jq '.data.result[] | {job: .metric.job, p95_seconds: .value[1]}'

A healthy output shows p95 scrape duration well under scrape_timeout for every job. A p95 above 80 percent of the timeout means the job is one slow target away from cascading failures.

Confirm the alerts would fire if needed. Inject a failure:

# CONFIGURATION - block a single target's port in iptables.
sudo iptables -I OUTPUT -p tcp --dport 9100 -d node-3.internal -j DROP

# Wait 2 minutes, then check the alert state.
curl -s 'http://observer.internal:9090/api/v1/alerts' \
  | jq '.data.alerts[] | select(.labels.alertname=="TargetDown")'

# Undo.
sudo iptables -D OUTPUT -p tcp --dport 9100 -d node-3.internal -j DROP

A working setup shows a firing TargetDown alert with the right job and instance labels within two minutes. The cleanup step restores normal operation.

How it can fail

1. Connection refused

The target process is not listening on the configured port. Symptom: up == 0 for that instance, Prometheus log shows “connection refused”. Action: confirm the target process is running (systemctl status ... or equivalent), confirm the port is bound (ss -tlnp), confirm the firewall allows the scrape source.

2. Connection timeout

The TCP connect does not return. Symptom: up == 0, Prometheus log shows “context deadline exceeded” or “i/o timeout”. Action: this is the failure shape of a dropped packet (firewall silently dropping) rather than a rejected one. Test with curl --connect-timeout 5 from the Prometheus host to the target.

3. TLS handshake failure

The target is HTTPS with mTLS. Symptom: up == 0, Prometheus log shows “x509: certificate … ” or “tls: handshake failure”. Action: confirm the CA bundle is current, confirm the client cert is not expired (openssl x509 -in client.pem -noout -dates), confirm the SNI matches.

4. Auth failure (401, 403)

The target requires basic auth, bearer token, or mTLS. The credential is missing or wrong. Symptom: up == 0, Prometheus log shows HTTP 401 or 403 from the target. Action: rotate the credential, confirm the basic_auth, authorization, or tls_config block in the scrape job matches the target’s expectation.

5. Wrong metrics path

The scrape job is configured with metrics_path: /metrics but the target exposes metrics on a different path. Symptom: up == 0, Prometheus log shows HTTP 404. Action: confirm the path by curling the target manually; the path is often /actuator/prometheus, /q/metrics, or /api/v1/metrics depending on the framework.

6. Relabel drops the body

The target is responding correctly but every sample is dropped by metric_relabel_configs. The dashboard shows a flat line at zero or the previous value. Symptom: up == 1 (scrape succeeded), scrape_samples_scraped is non-zero, but the target’s metrics are absent from queries. Action: inspect the relabel chain in the Prometheus log (--log.level=debug shows the drops); confirm the action (drop, labeldrop, labelmap) is intended.

How to troubleshoot it

Security implications

Scrape credentials (basic auth, bearer token, mTLS client cert) are stored in the Prometheus configuration. They grant read access to the target’s metrics. The leak path is the config repo: any operator with read access to the configuration can extract a credential that is valid for the target. Treat the scrape config the same way you treat database credentials: a secret manager for production deployments, and a separate secret per environment.

up == 0 can be weaponised as a denial-of-monitoring attack. An attacker who can drop traffic between Prometheus and a target can silence alerts without touching the target. The fix is mTLS between Prometheus and the targets, plus a network policy that explicitly allows only the Prometheus source range.

Performance implications

Each scrape has a fixed cost: connection setup, TLS handshake, HTTP round-trip, and body parse. The cost is paid per target per scrape_interval. A 1000-target fleet on a 15-second interval is 67 scrapes per second. On a single Prometheus host with a 10-second timeout, each scrape occupies a worker goroutine for up to 10 seconds. With a default worker pool of 20 (-scrape.workers-per-scrape-pool), a single slow target blocks one worker but does not block the whole pool.

The right sizing pattern is: scrape timeout less than scrape interval, scrape interval greater than expected scrape duration p99 by at least 2x, and total scrape workers sufficient to handle the worst-case concurrent slow scrapes (usually 5-10 percent of the fleet).

Production guidance

• Set scrape_timeout to 5s on a 15s scrape_interval for most jobs. Reduce to 2s for very high-fanout jobs.
• Alert on up == 0 with for: 2m, not for: 0m. The transient blip is the noise floor; the persistent outage is the signal.
• Pair up == 0 with scrape_duration_seconds alerts. A slow scrape that crosses the timeout becomes up == 0; catching it before the cross is the early warning.
• Inject a failure quarterly. Block a target port with iptables and confirm the alert fires, the runbook is current, and the cleanup step restores normal operation.
• Keep scrape credentials in a secret manager. Rotate on the same schedule as application credentials.

Verification

You should now be able to answer:

• What does up == 0 actually mean, and what are the six buckets of failure it can represent?
• What does scrape_duration_seconds measure, and how does it complement up?
• Why is a slow target that is still “up” worse than a target that is down?
• What is the difference between alerting on up == 0 with for: 0m and for: 2m, and which is correct?
• How does a relabel rule that drops every metric produce a silent failure, and how do you detect it?

Quiz