Scrape Overload

At 03:00 the on-call receives a Slack message that the dashboards look six minutes behind. The user-facing services are healthy. up{job="..."} returns 1 for every target. The target endpoints answer /metrics in under 50 ms when probed by hand. Yet every panel in Grafana shows the same stale data. The cause is not the targets. The cause is Prometheus itself — scrape overload has set in and the scrapes have begun to back up.

This lesson is about the cost of scraping. A Prometheus instance does not absorb scrape requests the way a web server absorbs HTTP requests. Each target is fetched on a schedule and every fetch has a hard deadline. When the schedule slips the consequences are visible long before any alert fires.

What it is

Scrape overload is the condition where the time Prometheus needs to fetch one or more targets exceeds the configured scrape_interval. The result is that subsequent scrapes of those targets are skipped, samples arrive later than the interval suggests, and dashboards display stale data while every up metric reports healthy.

It is not a single failure. It is a class of failures that share one symptom: the time series exist, the targets respond, yet the platform appears to be lagging behind reality.

Why a sysadmin cares

Three production pains concentrate in scrape overload:

1. Stale dashboards without an obvious cause. Grafana shows data that is several minutes old, but the targets are up, the network is healthy and the dashboards themselves do not report an error. The on-call engineer wastes time validating the dashboard layer.
2. Quiet data loss. A scrape that times out is not retried in the same interval. The samples for that minute simply are not written. The data point does not appear in queries later. Recovery requires accepting the gap.
3. Head-of-line blocking. Prometheus runs one scrape goroutine per target. A handful of slow targets do not slow every other target — that part is good — but they do cause those specific targets to fall behind permanently and they consume WAL and head-block memory at a rate proportional to the original scrape interval.

The lesson is that scrape cost is not a configuration knob you set once. It is an operating point that you tune against the actual behaviour of the targets you choose to monitor.

How it works

A scrape is a small, well-defined pipeline that runs per target on every interval. The cost concentrates in three stages:

   scrape_interval
        |
        v
   +-----------------+      +-----------------+
   |  DNS resolve    | ---> |  TCP / TLS      |
   +-----------------+      +-----------------+
                                   |
                                   v
                          +-----------------+
                          |  HTTP fetch     |   timeout here is
                          |  GET /metrics   |   scrape_timeout
                          +-----------------+
                                   |
                                   v
                          +-----------------+
                          |  Parse          |   sample_limit is
                          |  text to        |   enforced here
                          |  samples        |
                          +-----------------+
                                   |
                                   v
                          +-----------------+
                          |  Ingest         |   head block + WAL
                          +-----------------+

Each stage has a cost. DNS is cheap on a hot cache and expensive on a cold cache. The TCP and TLS handshake is the first non-trivial cost. The HTTP fetch is the dominant cost for most targets. Parsing is proportional to samples and is bounded by sample_limit. Ingest is proportional to series and is normally much faster than the network round-trip.

The single most important metric is scrape_duration_seconds. When this metric, taken at the quantile that matters to you (typically p95 or p99), exceeds roughly 50 per cent of scrape_interval, the platform is on the edge of overload. When it exceeds scrape_interval, the overload is present.

How to configure it

Scrape overload is mitigated at the target level. Four configuration fields carry the bulk of the load:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  external_labels:
    cluster: prod-eu-west-1

scrape_configs:
  - job_name: 'node-exporter'
    scrape_interval: 15s       # matches global; explicit for readability
    scrape_timeout: 10s        # must be less than scrape_interval
    sample_limit: 2000         # drop the rest; counted in scrape_samples_dropped_total
    body_size_limit: 10MB      # reject before parse; cap on /metrics size
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'application-metrics'
    scrape_interval: 30s       # less critical exporter can run less often
    scrape_timeout: 10s
    sample_limit: 5000
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_gc_.*'
        action: drop

Four rules govern these knobs:

1. scrape_timeout must be strictly less than scrape_interval. Prometheus does not enforce this; an invalid combination silently produces overlap.
2. sample_limit defaults to zero, which means unlimited. An unbounded exporter that suddenly emits ten times as many samples will start dropping them silently until you set a cap.
3. body_size_limit defaults to zero, which historically meant no limit. From Prometheus 2.55 it defaults to 50 MB for safety. Set this explicitly so that a runaway exporter cannot make Prometheus read gigabytes per scrape.
4. metric_relabel_configs runs last, after parsing, and is the right place to drop unwanted metrics before they hit the head block.

How to validate it

Validation is a sequence of read-only checks. Start with the configuration, then move to the live metrics.

# Severity: READ-ONLY
promtool check config /etc/prometheus/prometheus.yml

The next step is to inspect the active targets. The /api/v1/targets endpoint returns each target with its last scrape duration, last error and last scrape time:

# Severity: READ-ONLY
curl -sG http://prometheus:9090/api/v1/targets \
  --data-urlencode 'state=active' \
  | jq '.data.activeTargets[] | {job:.labels.job, url:.scrapeUrl,
                                  duration:.lastScrapeDuration,
                                  health:.health}'

The expected result, in a healthy system, is duration values comfortably under the scrape interval — for a 15 s interval, p99 under 7 s.

The third check is the platform view. Prometheus emits a metric for every scrape it has ever performed:

# Severity: READ-ONLY
# p99 scrape duration per job, in seconds
quantile_over_time(0.99,
  sum by (job) (rate(scrape_duration_seconds[5m]))
) > 7

Any job where the p99 duration exceeds roughly half of the interval is a candidate for tuning.

Finally, confirm the sample-limit and body-size-limit are honoured:

# Severity: READ-ONLY
sum by (job) (rate(scrape_samples_dropped_total[5m]))
sum by (job) (rate(scrape_body_size_bytes_count[5m]))

A non-zero scrape_samples_dropped_total is the canonical sign that a target is exceeding its sample budget.

How it can fail

Six failure shapes account for nearly every scrape-overload incident:

1. A slow exporter. The target endpoint takes longer than scrape_timeout to respond. Prometheus aborts the fetch and writes no samples for that scrape. Symptom: up{job="x"}=1 but the data is several minutes stale.
2. A target that exceeds sample_limit. The body parses successfully but more samples arrive than the limit permits. The surplus is dropped and counted. Symptom: gaps in time-series graphs aligned to the scrape interval.
3. A target that exceeds body_size_limit. The HTTP body is rejected before parsing. Symptom: every scrape logs a body size limit exceeded warning.
4. DNS resolution that hangs. The target hostname is resolved once per scrape; a slow resolver stalls the whole pipeline. Symptom: scrape duration spikes by the DNS round-trip time, and the resolver logs show retries.
5. TLS handshake that hangs. The target requires a client certificate and the certificate file is missing or wrong. Symptom: scrape duration spikes by exactly the TLS handshake timeout.
6. A label explosion on one exporter. A single exporter begins emitting a high-cardinality label such as a UUID or request ID. The samples still arrive on time, but the parse stage allocates far more memory than expected. Symptom: scrape duration is fine, but Prometheus RSS grows until the OOM killer fires.

How to troubleshoot it

The diagnostic order is consistent across all six failure shapes. Form a hypothesis from the symptom, then test it against the evidence:

1. Confirm the symptom is scrape overload. Query time() - timestamp(prometheus_last_modified_time). If this is more than roughly two scrape intervals, Prometheus itself is behind.
2. Identify the slow jobs. Use the scrape_duration_seconds query above. Sort by p99 duration, descending.
3. Inspect each slow target. Query the targets endpoint to find targets whose last successful scrape is older than the interval.
4. Inspect the target endpoint. curl --max-time 10 http://target:9100/metrics | wc -l from the Prometheus host. Compare this with what Prometheus sees.
5. Inspect the Prometheus logs. Grep for context deadline exceeded, sample limit exceeded or body size limit exceeded.
6. Inspect the exporter logs. Most exporters log the per-scrape cost. Node Exporter logs a request-completed line; the application exporter logs the request handler duration.

Security implications

Scrape endpoints expose application internals. Three disciplines matter in production:

1. Bind metrics endpoints to the management network only. Do not expose /metrics on the public interface.
2. Use mTLS or a bearer token when the data is sensitive. Prometheus supports bearer_token_file and tls_config on every scrape job.
3. Treat the scrape credentials like any other secret. The token file lives in /etc/prometheus/secrets with mode 0400 and an owner distinct from the Prometheus user.

A scrape endpoint that is reachable from the internet is an information disclosure. Versions, build hashes, internal hostnames and feature flags all leak through /metrics.

Performance implications

Scrape cost is a function of three variables: number of targets, scrape interval, and the per-target cost in samples and bytes. The arithmetic is straightforward:

ingestion_samples_per_second
    = targets * samples_per_scrape / scrape_interval

A platform that scrapes 1000 targets at 15 s with 2000 samples each is ingesting roughly 130 000 samples per second. Doubling any one of the three inputs doubles the cost. Halving the scrape interval is the most common cause of an unexplained ingestion surge.

Verification

You should now be able to answer:

• What does scrape_duration_seconds tell you, and what threshold indicates scrape overload?
• Why must scrape_timeout be smaller than scrape_interval?
• What happens to the samples for a target whose scrape times out?
• Which two counters tell you that sample_limit or body_size_limit are being enforced?
• How would you find the slowest job on a live Prometheus?

Quiz