The Scrape Lifecycle

A fleet dashboard loses one node. The node is fine. The exporter is fine. The network is fine. A relabel rule merged an hour ago quietly dropped the entire job from discovery, and the only reason anyone noticed is that a graph went empty. Every incident of that shape lives inside the scrape lifecycle: the fixed pipeline each sample travels from “something that might exist” to “bytes in the TSDB”. This lesson walks the pipeline stage by stage.

The pipeline

service discovery   (static, file_sd, consul_sd, kubernetes_sd, ...)
        |  target groups carrying __meta_* labels
        v
target relabeling   (relabel_configs: keep / drop / replace)
        |  final target labels: __address__, __scheme__, job, instance
        v
scrape loop         (one per target; offset spreads load across the interval)
        |  HTTP GET /metrics with per-job TLS and auth
        v
parse               (Prometheus text / OpenMetrics exposition format)
        |  timestamp = scrape time (targets rarely send their own)
        v
metric relabeling   (metric_relabel_configs, applied per sample)
        v
limits              (sample_limit, label_limit, body_size_limit, target_limit)
        v
append              (head block + write-ahead log)
        |  up{job,instance} = 0 or 1 written here, after every scrape
        v
staleness           (absent series get stale markers; 5m lookback at query time)

Discovery and target relabeling

Each job’s service-discovery mechanism emits target groups: endpoints decorated with metadata labels such as __meta_consul_service or __meta_kubernetes_pod_name. Discovery’s output is the universe of possible targets.

relabel_configs then runs against those labels. keep and drop actions select which candidates become targets; replace actions build the final label set — classically, setting instance from __meta_consul_node instead of an anonymous IP:port. Labels beginning with __ are internal: they drive relabeling and are not stored with the samples.

The scrape itself

Per job there is one scrape pool; per target, one scrape loop. Each loop is offset by a hash of the target’s labels, so thousands of targets spread across the interval rather than stampeding at second zero. The HTTP GET uses the job’s scheme, TLS and credentials; the Accept header negotiates OpenMetrics versus the classic text format. The response body is parsed line by line — a malformed line fails the whole scrape.

After parsing, metric_relabel_configs runs per sample (drop that expensive histogram nobody queries), then the scrape limits apply: sample_limit (samples per scrape), label_limit, label_name_length_limit, label_value_length_limit, body_size_limit, and target_limit (how many discovered targets the pool will scrape at all). Defaults are 0, meaning unlimited — a rogue or compromised exporter can hurt you unless you set them. Exceeding a limit fails the scrape, which means up goes to 0. The safety valve and the outage look identical from the outside.

Finally, samples are appended to the head block and the WAL, and Prometheus writes up{job, instance} for the target: 1 on success, 0 on failure. That write happens after every scrape, which is why up is the ground truth of the pipeline.

Staleness

If a series was present in the previous scrape of a target but is absent now, Prometheus appends a stale marker. The same happens to all of a target’s series when the target is removed from the pool. At query time, a series with no sample inside the lookback window (--query.lookback-delta, default 5m) is simply not returned. Practically: graphs go blank rather than flatlining at the last value, and range-vector alerts see “no data”.

What a reload actually does

On SIGHUP — or POST /-/reload when --web.enable-lifecycle is set — Prometheus re-parses the configuration:

• If parsing fails, the old configuration keeps running, prometheus_config_last_reload_successful flips to 0, and the error is in the logs. Nothing pages you unless you alert on that metric.
• Scrape pools are rebuilt. Targets that still exist keep their running scrape loops and offsets — a reload does not cause a fleet-wide re-scrape. New targets start; removed targets receive stale markers and stop.
• Rule files are re-read (lesson 04).

How to configure it

A job that exercises the stages above:

scrape_configs:
  - job_name: node
    scrape_interval: 30s
    scrape_timeout: 10s        # must not exceed scrape_interval
    consul_sd_configs:
      - server: 'consul.internal:8500'
    relabel_configs:
      # only scrape nodes carrying the prod tag
      - source_labels: [__meta_consul_tags]
        regex: '.*,prod,.*'
        action: keep
      # instance = the consul node name, not an anonymous ip:port
      - source_labels: [__meta_consul_node]
        target_label: instance
    metric_relabel_configs:
      # drop a metric we never query before it costs storage
      - source_labels: [__name__]
        regex: 'node_textfile_mtime_seconds'
        action: drop
    sample_limit: 10000        # fail the scrape beyond this; protects the TSDB
    body_size_limit: 20MB      # protects against rogue or huge exporters

Note what is not here: honor_timestamps defaults to true, and exporters almost never send timestamps, so samples get the scrape time. Leave it alone unless you know why a target sends its own.

How to validate it

# Config sanity
promtool check config /etc/prometheus/prometheus.yml

# Trigger a reload and confirm it took effect
curl -s -X POST localhost:9090/-/reload
curl -s 'localhost:9090/api/v1/query?query=prometheus_config_last_reload_successful'

# Per-target truth: discovered vs final labels, health, lastError
curl -s 'localhost:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[] | {job: .labels.job,
        health: .health, lastError: .lastError,
        discovered: .discoveredLabels.__address__}'

The scrape-health metrics, queryable in PromQL:

# how long scrapes take, per job — watch it against scrape_timeout
scrape_duration_seconds

# samples per scrape — sudden jumps mean an exporter upgrade or a leak
scrape_samples_scraped

# what survived metric relabeling; the gap vs the line above is what you dropped
scrape_samples_post_metric_relabeling

# series churn — new series per scrape; sustained high values burn memory
scrape_series_added

How it can fail

1. A keep regex drops everything. Symptom: the job exists but has zero active targets, and up for the job is absent, not 0. The most silent failure in this lesson; absent() guards exist for it.
2. scrape_timeout tighter than a slow exporter. Symptom: intermittent up of 0 with “context deadline exceeded”, and scrape_duration_seconds pinned at the timeout value.
3. sample_limit exceeded after an exporter upgrade adds metrics. Symptom: the whole scrape fails — up is 0 and scrape_samples_scraped hovers at the limit. The protection looks like an outage.
4. A reload that never took effect. SIGHUP sent to the wrong PID in a container, or POST /-/reload returning 403 because --web.enable-lifecycle is not set. Symptom: the file on disk and /api/v1/status/config disagree.
5. honor_timestamps: true with a clock-skewed target that sends its own timestamps. Symptom: samples rejected as out of window; the target looks scraped but the series are empty.
6. target_limit set on a growing pool. Symptom: a stable subset of instances is scraped and the rest silently never appear; the target list shows the pool truncated.

How to troubleshoot it

1. Down or absent? up of 0 points at the scrape itself; a missing up series points at discovery or relabeling. Decide this first — it halves the search space.
2. Read /targets. Compare discoveredLabels with labels, and read lastError verbatim. It names the stage: connection, timeout, parse error, limit exceeded.
3. Classify with the scrape metrics. Duration? Sample count? Series dropped by relabeling? The pattern names the cause.
4. Reproduce the scrape by hand with curl from the Prometheus host, same scheme, port and path.
5. Verify the running config. /api/v1/status/config against the file, plus prometheus_config_last_reload_successful.
6. Logs. journalctl -u prometheus for reload failures and per-scrape errors.

Security implications

• Relabeling can leak __meta_* metadata into stored labels. Kubernetes annotations and Consul tags sometimes contain secrets; a careless labelmap rule writes them into the TSDB forever.
• Scrape credentials live in the configuration; per-job bearer_token_file / password_file with 0640 permissions.
• The lifecycle endpoint is unauthenticated: with --web.enable-lifecycle, anyone who can reach port 9090 can trigger reloads. Bind or proxy accordingly.
• Exporter endpoints disclose host internals to anyone who can reach them (lesson 01).

Performance implications

Scrape cost is targets × series ÷ interval, paid by both sides. The per-target offsets keep the server’s own load smooth across the interval. metric_relabel_configs cost CPU on every scrape of every target; sample_limit and body_size_limit are the seatbelts that keep one bad exporter from becoming a TSDB incident. Watch scrape_series_added: sustained churn (short-lived series) is the memory profile that kills Prometheus servers, not raw scrape volume.

Verification

You should now be able to answer:

• In what order do discovery, relabeling, scraping, metric relabeling and the append happen — and where does up get written?
• What is a stale marker, and why does a removed target’s series vanish from queries within about five minutes?
• What does a reload do to targets that existed before and after? What happens when the new config is invalid?
• Which scrape metric would you graph first to catch an exporter that suddenly started emitting ten times as many series?
• Why does exceeding sample_limit produce up of 0 rather than a truncated scrape?

Quiz