Scrape Configs

A new node_exporter rolls out fleet-wide on Monday. By Wednesday the Prometheus host is swapping. The cause is not a bug: the exporter enabled two new collectors, one job had no sample_limit, and every target tripled its series count overnight. Nobody noticed at the job level because nobody had ever read the job level — the scrape config was copied from a blog post two years ago and left alone.

scrape_configs is where ingestion is actually decided. The global block sets defaults; this list decides, per job, what is scraped, from where, how often, over which transport, and how much of it is allowed in. This lesson reads the anatomy field by field.

What a scrape config is

Each entry in scrape_configs is a job: a named unit of scraping that shares one configuration. A job resolves its targets (statically or via service discovery), scrapes each target’s HTTP endpoint on the effective interval, runs the results through relabelling and limits, and appends what survives to the head block of the TSDB.

 job: node-core
   |
   v
 target list (static_configs / service discovery)
   |
   v
 per-target scrape loop:  GET http://10.12.0.14:9100/metrics
   |                        every scrape_interval, killed at scrape_timeout
   v
 relabel_configs -> parse -> metric_relabel_configs -> limits
   |
   v
 append to TSDB head  +  synthesise up / scrape_* metrics

One exporter class can justify several jobs. node-core at 15s for infrastructure you page on, node-edge at 2m with a 20s timeout for a fleet behind high-latency links. Same exporter, different SLAs, different cost. Splitting by job is how you pay for resolution only where you need it.

The anatomy, field by field

scrape_configs:
  - job_name: node-core            # required; becomes the job label
    scrape_interval: 15s           # overrides global for this job
    scrape_timeout: 10s            # must not exceed the effective interval
    metrics_path: /metrics         # default; some exporters differ
    scheme: http                   # http | https
    static_configs:
      - targets:
          - 10.12.0.14:9100        # becomes the instance label
          - 10.12.0.15:9100
        labels:
          site: lon-2              # extra labels on every target here

    # Safety nets: 0 means unlimited, which is how Monday happens.
    sample_limit: 5000             # whole scrape FAILS if exceeded
    label_limit: 60                # per-sample label count cap; scrape fails

    honor_labels: false            # default; see below
    honor_timestamps: true         # default; see below

The fields that change stored data rather than scrape behaviour:

• honor_labels (default false): the exporter’s own label values normally lose to Prometheus’s identity labels. If an exporter exposes instance="foo", it is stored as exported_instance="foo" and the real instance stays the target address. Set true for federation and push-style exporters whose labels are the point — and accept that you now trust the source not to collide identities across hosts.
• honor_timestamps (default true): Prometheus trusts timestamps supplied by the target. Set false for exporters with skewed clocks or broken timestamp handling; Prometheus then stamps scrape time instead. Trusting bad timestamps produces out-of-order and too-old append errors that look like TSDB corruption but are not.
• params: URL query parameters appended to every request. The canonical use is exporter front-ends that multiplex — the blackbox exporter’s /probe?module=... and the SNMP exporter’s /snmp?module=...&target=....

A blackbox job shows params and metrics_path in their natural habitat (the relabelling dance itself is covered in the blackbox lesson; included here so the job is real):

  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com/health
          - https://grafana.example.com/api/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox.example.com:9115

Authentication and TLS

Exporters worth scraping are worth protecting. The job-level blocks:

    # Prefer files over inline secrets: the config file is far more
    # widely readable than a 0600 credential file.
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/secrets/node.token

    # Bearer tokens use the sibling block:
    # authorization:
    #   credentials_file: /etc/prometheus/secrets/api.bearer

    tls_config:
      ca_file: /etc/prometheus/tls/ca.crt
      cert_file: /etc/prometheus/tls/client.crt   # for mTLS exporters
      key_file: /etc/prometheus/tls/client.key
      server_name: node.example.com               # SNI/verify override
      # insecure_skip_verify: false  -- never true outside a lab

Credential files are read at config load. Rotate a token and the scrape keeps failing with 401 until you reload — the first thing to check when an otherwise healthy target goes dark right after a secret rotation.

Validating a job

# 1. Static validation (exit 0 = valid; also catches the
#    timeout-greater-than-interval invariant per job)
promtool check config /etc/prometheus/prometheus.yml

# 2. See the targets a job resolves to, with labels before and after
#    relabelling (trimmed, illustrative)
promtool check service-discovery /etc/prometheus/prometheus.yml node-core
#   Labels:
#     __address__="10.12.0.14:9100"
#     __metrics_path__="/metrics"
#     __scheme__="http"
#     job="node-core"
#     site="lon-2"
#   Scrape URL: http://10.12.0.14:9100/metrics

# 3. Live health from the running server
curl -s 'http://localhost:9090/api/v1/targets?state=active' | \
  jq -r '.data.activeTargets[] |
    [.labels.job, .labels.instance, .health, .lastError] | @tsv'
# node-core   10.12.0.14:9100   up
# node-core   10.12.0.15:9100   down   Get "http://10.12.0.15:9100/metrics": context deadline exceeded

# 4. The series view, in PromQL
#    up == 0
#    topk(10, scrape_samples_scraped)   -- find the expensive jobs

scrape_samples_scraped per job is the single most useful capacity number you own. Graph it; alert on its sudden growth; size sample_limit from roughly three times its steady-state value.

How scrape configs fail

1. Wrong metrics_path. The exporter serves /metrics; the job says /prometheus/metrics. Symptom: up is 0, lastError shows server returned HTTP status 404, scrape_samples_scraped is 0.
2. Scheme mismatch. scheme: https against an HTTP-only exporter. Symptom: up is 0 with server gave HTTP response to HTTPS client. The reverse — HTTP against HTTPS — returns a 400-class error from the TLS side.
3. Credential failure after rotation. Symptom: 401 Unauthorized in lastError, starting exactly at a secret change, persisting until Prometheus reloads the credential file.
4. TLS verification failure. Self-signed CA not in ca_file, or server_name unset for a certificate whose SANs do not match the target IP. Symptom: x509: certificate signed by unknown authority or x509: certificate is valid for ... in lastError.
5. Limit breach after an exporter upgrade. The upgrade adds collectors; samples cross sample_limit; the entire scrape is treated as failed. Symptom: up flips to 0 with exceeded sample limit — one burst takes down monitoring of the whole target, which is exactly when you wanted it.
6. honor_labels: true with identity collisions. An exporter stamps instance="prod" on every host. Symptom: series from many hosts interleave under one label set; graphs flap between hosts; duplicate sample or out-of-order errors appear at ingestion.

Troubleshooting, in order

1. Is the target even resolved? promtool check service-discovery offline, or the targets API live. A target absent from both is a static-config or SD problem, not a scrape problem.
2. What does the scrape itself say? lastError per target. It is the single highest-signal field in the whole subsystem.
3. Can you reproduce by hand? curl -sv the exact scrape URL from the Prometheus host — same path, scheme, headers, and CA. If curl fails, Prometheus is the messenger, not the problem.
4. Is it a limits failure? scrape_samples_scraped near sample_limit, or up flipping to 0 right after an exporter change, points at the cap.
5. Is the data wrong rather than missing? Identity collisions and timestamp problems show up in the TSDB, not in up. Check for exported_instance labels and for append errors in the logs.

Security implications

Scrape configs hold credentials. Inline password strings end up in backups, config-management diffs, and anyone’s read of the status API is redacted — but the file on disk is not. Prefer password_file and credentials_file at 0600 prometheus:prometheus, and keep prometheus.yml itself at 0640. insecure_skip_verify: true turns off certificate verification; outside a lab it is a decision to accept person-in-the-middle exposure for everything the scraper reads. Finally, /metrics endpoints leak operational detail — process arguments, versions, internal topology — so scrape over network segments you control and let mTLS say who may ask.

Performance implications

Cost per job is targets times series per target, divided by interval. The levers, in order of usual payoff: longer intervals for jobs that do not drive paging alerts; sample_limit and relabelling to drop series you never query; splitting fast and slow target classes into separate jobs; body_size_limit where exporters produce pathological responses. honor_timestamps: false costs nothing and occasionally saves you from a target whose clock makes every append a retry. Scrape CPU on the Prometheus host scales with parse volume; the scrape_duration_seconds histogram tells you which jobs dominate it.

Production guidance

• One job per (exporter class, SLA class). Never one giant job with per-target annotations deciding behaviour.
• Set sample_limit on every job from measured scrape_samples_scraped plus headroom, and revisit it at exporter upgrade time — that is when it fires.
• Credentials via files, TLS verification on, server_name explicit when targets are IPs.
• Keep honor_labels and honor_timestamps at defaults unless you are federating or taming a specific broken exporter — and write the reason in a comment.
• Alert on up == 0 per job, and on scrape_samples_scraped growth, not just on target count.

Verification

You should now be able to answer:

• Which fields turn a target list into a working job, and what do job_name and the target address become as labels?
• What exactly happens to stored series when honor_labels is false and an exporter exposes instance?
• Why does exceeding sample_limit take the whole target’s up to 0 rather than trimming the excess?
• How do you list a job’s resolved targets and their last scrape errors without restarting anything?
• Why do credential rotations need a Prometheus reload to take effect?

Quiz