Scrape Config Error

A platform engineer adds a new scrape job for a service that just shipped a custom /actuator/prometheus endpoint. They edit prometheus.yml, kill -HUP Prometheus, and watch the new target list. The new job has zero targets. The /api/v1/targets endpoint shows the old jobs as healthy and the new job as absent. The engineer opens the config, looks for a typo, finds none, and reloads again. Same result. After fifteen minutes they promtool check config the file and see a YAML error two lines above the new job: an unquoted colon in a label value that the YAML parser tolerated at the start of the file but rejected when the new job added a value with a colon in it. The fix is one line; the diagnostic was the missing step.

A scrape config error is the third link in the chain. It is the link that the operator owns most directly, and the link where validation is cheapest. This lesson is the discipline of running the validation.

What it is

A scrape config error is any condition that prevents the scrape configuration from doing what the operator intended. There are three distinct classes, with different signatures and different fixes:

1. YAML syntax error. The file does not parse. Promtool reports a line and column; Prometheus refuses to load the file at all; kill -HUP is a no-op because the new file fails to load and the old file remains in memory. Symptom: promtool check config exits non-zero; Prometheus logs error loading config; the live configuration shown by /api/v1/status/config is the previous one.
2. Schema error. The YAML parses but the structure does not match the Prometheus schema. A required field is missing; a value is the wrong type; an enum value is invalid. Symptom: promtool check config exits non-zero; the error message names the field.
3. Semantic error. The YAML parses, the schema is valid, the file loads, and the reload succeeds. The targets are resolved, but they point at the wrong host, the wrong port, the wrong path, or the wrong scheme. Symptom: the target appears in /api/v1/targets with up == 0 and a meaningful lastError; promtool check config exits zero because the schema is fine.

The third class is the most expensive. The first two are caught by promtool check config; the third is caught by promtool check service-discovery and the live /targets API.

Why a sysadmin cares

The reload of a Prometheus configuration is the highest-blast- radius reload in the observability stack. A bad reload can silently disable monitoring for a fleet, can corrupt the TSDB (if a write_relabel_configs change is wrong), or can page the on-call for a synthetic problem. The cost of running promtool check config before kill -HUP is one second. The cost of skipping it is whatever the next incident costs.

Three production pains follow:

1. Reload that drops jobs. A schema error causes Prometheus to refuse the entire file. The previous configuration stays in memory, but the operator does not know this and pages the on-call. The discipline is to read the reload response, not just send the signal.
2. Reload that loads but does nothing. A semantic error causes the reload to succeed but the targets to be wrong. The up series stays at 1 for the old targets and at 0 for the new ones, with lastError that names the problem. The discipline is to inspect /api/v1/targets after every reload.
3. Reload that breaks retention. A change to storage.tsdb.retention.time or storage.tsdb.retention .size that is wrong truncates blocks the operator did not intend to truncate. The discipline is to validate retention changes against the current blocks before reload.

How it works

The Prometheus configuration load is two-phase: parse, then validate. The parse phase reads the YAML; the validate phase applies the schema and resolves references. Either phase can fail; the failure mode and the recovery differ.

  prometheus.yml
        |
        v
  +-----------+
  | parse     |  YAML -> AST; any syntax error fails here
  +-----------+
        |
        v
  +-----------+
  | validate  |  AST -> config; schema errors fail here
  +-----------+  semantic errors pass through
        |
        v
  +-----------+
  | reload    |  atomic swap of the in-memory config
  +-----------+
        |
        v
  scrape pools reshuffle to match the new config
        |
        v
  /api/v1/status/config reflects the new file

A failure in the parse or validate phase is loud: Prometheus logs the error, refuses to load, and keeps the previous configuration in memory. A failure in the semantic phase is quiet: the new file is loaded, the targets resolve to the wrong thing, and the operator sees up == 0 in the targets list.

The reload is atomic. There is no window during which Prometheus is using part of the old configuration and part of the new. This is a feature: a bad reload does not corrupt the running state. It is also a trap: a bad reload looks like a successful reload because no error is visible.

Under the hood

How to configure it

A scrape config that surfaces every class of error and is easy to validate:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 30s
  scrape_timeout: 10s
  external_labels:
    cluster: prod-eu-1
    region: eu-west-1

scrape_configs:
  - job_name: node
    metrics_path: /metrics
    scheme: http
    scrape_interval: 15s        # overrides global for this job
    scrape_timeout: 5s          # must be less than scrape_interval
    static_configs:
      - targets:
          - 'node-1.internal:9100'
          - 'node-2.internal:9100'
        labels:
          site: lon-2
          env: production
    relabel_configs:
      - source_labels: [env]
        regex: production
        action: keep

  - job_name: app
    metrics_path: /actuator/prometheus
    scheme: https
    scrape_interval: 30s
    scrape_timeout: 10s
    tls_config:
      ca_file: /etc/prometheus/tls/ca.crt
      server_name: app.internal
    static_configs:
      - targets:
          - 'app-1.internal:8443'
          - 'app-2.internal:8443'

The invariants the configuration has to satisfy:

• scrape_timeout less than scrape_interval. Violations are silently accepted by the schema but make every scrape race the next interval.
• metrics_path starts with /. The schema accepts bare paths but the scrape URL is malformed.
• scheme is http or https. Anything else is a schema error.
• Each job_name is unique within the file. Duplicates are accepted by the schema but produce unpredictable behaviour.
• TLS ca_file, cert_file, key_file paths exist and are readable by the Prometheus user. The schema does not check.

How to validate it

The validation ladder for every configuration change.

# Step 1: parse and schema check (catches class 1 and class 2)
promtool check config /etc/prometheus/prometheus.yml
# expected: SUCCESS; the configuration is valid

# Step 2: service-discovery check for the affected job
promtool check service-discovery /etc/prometheus/prometheus.yml node
# expected: a JSON array of targets with discoveredLabels and
# labels (final). Compare against the previous output.

# Step 3: reload Prometheus (atomic)
kill -HUP "$(pidof prometheus)"

# Step 4: confirm Prometheus loaded the new file
curl -s http://prom:9090/api/v1/status/config \
  | jq '.data.yaml' | head -30
# expected: the new file content; compare against the file on disk

# Step 5: confirm the affected targets are healthy
curl -s 'http://prom:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[]
        | select(.labels.job=="node")
        | {instance: .labels.instance, health: .health,
           lastError: .lastError}'
# expected: health == "up" for every target in the job

# Step 6: confirm the change did not break unrelated jobs
curl -s 'http://prom:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[] | .labels.job' | sort -u
# expected: the same set of jobs as before the reload

A reload that fails step 4 did not load. A reload that fails step 5 has a semantic error in the affected job. A reload that fails step 6 has broken an unrelated job and the operator must investigate the merge.

How it can fail

Six failure shapes appear repeatedly. The first three are schema errors; the second three are semantic errors.

1. YAML indentation or quoting error. Tabs instead of spaces; an unquoted colon in a value; a value that should be quoted. Symptom: promtool check config exits non-zero; the error names the line and column. The fix is to correct the YAML.
2. Missing required field. A scrape_configs entry without job_name; a tls_config with ca_file but without server_name when the certificate uses SANs that do not include the target IP. Symptom: promtool check config exits non-zero with the field name. The fix is to add the field.
3. Wrong type. scrape_interval: 15 instead of scrape_interval: 15s; targets: as a string instead of a list; labels: as a list instead of a map. Symptom: promtool check config exits non-zero with the expected type. The fix is to match the type.
4. scrape_timeout greater than scrape_interval. The schema accepts it; the scrape races the next interval. Symptom: scrapes time out intermittently; scrape_duration _seconds rises toward the interval; the next scrape starts before the previous one finishes. The fix is to lower the timeout or raise the interval.
5. Wrong metrics_path. The exporter serves /metrics but the job says /prometheus/metrics. Symptom: lastError: server returned HTTP status 404; up == 0; promtool check service-discovery shows the wrong URL under __metrics_path__. The fix is to match the path.
6. scheme mismatch. scheme: https against an HTTP-only exporter. Symptom: lastError: server gave HTTP response to HTTPS client; up == 0. The fix is to match the scheme.

How to troubleshoot it

Security implications

The configuration file holds credentials, certificate paths, and target lists. The leak path is the same as for any configuration file: backups, config-management diffs, version control. The discipline:

• Inline credentials are read by anyone with shell on the Prometheus host. Use password_file and credentials_file at 0600 prometheus:prometheus.
• TLS key material is read at config load. Rotate a key and the scrape keeps failing with handshake errors until the reload reads the new file.
• insecure_skip_verify: true is never correct outside a lab. The setting turns off certificate verification for the affected job and exposes the metrics endpoint to a person-in-the-middle.
• Treat config write access as production credential-level access. Whoever can edit prometheus.yml can redirect every scrape.

Performance implications

The configuration is not the dominant cost; the scrape is. But three configuration mistakes produce real performance damage:

• sample_limit not set. The default is 0 (unlimited). An exporter that adds a high-cardinality collector can inflate the series count overnight. Lesson 06 covers the defence.
• scrape_interval too aggressive for the fleet. A 5-second interval quadruples the cost of a 15-second fleet. Match the interval to the SLA class.
• No scrape_timeout and tight interval. A slow scrape blocks the next one, which pushes the entire job off-schedule. Lesson on scrape configs covers the discipline.

Production guidance

• Run promtool check config in CI on every change to prometheus.yml. A pre-commit hook that runs the command catches the error before the operator does.
• Run promtool check service-discovery against a representative target set in CI as well. The semantic errors are not caught by check config.
• Always reload with kill -HUP and confirm with /api/v1/status/config that the file actually loaded. A silent no-op is worse than a loud failure.
• Inspect /api/v1/targets after every reload. The targets list is the live evidence that the reload did what was intended.
• Keep a copy of the previous prometheus.yml on the Prometheus host. A cp rollback is faster than a git revert when the page is open.
• Document the reload procedure in the runbook. The on-call engineer at 02:14 should be able to roll back in under a minute from the runbook alone.

Verification

You should now be able to answer:

• What are the three classes of scrape config error, and how does each one manifest in promtool and the Prometheus logs?
• Which two promtool subcommands catch which classes of error?
• What is the invariant between scrape_interval and scrape_timeout, and what is the failure shape when it is violated?
• How do you confirm a reload actually loaded the new configuration without restarting Prometheus?
• Why is insecure_skip_verify: true almost always wrong, and what is the right alternative?

Quiz