Runbook: Add a Scrape Target

Adding a scrape target is four lines of YAML. It is also an edit to the single file that decides what an entire organisation can see, applied by a reload that either takes the whole file or none of it. That asymmetry is the reason this has a runbook: the change is trivial and the way it fails is not.

When this runbook applies

• A new exporter, appliance or application endpoint needs to be scraped.
• An existing job needs another instance added to its target list.
• A target was removed during an incident and is being put back.

When it does not

• The target already exists and is down. That is a scrape failure, not an addition. up of 0 means Prometheus knows about the target and cannot scrape it; this runbook is for the case where Prometheus has never heard of it.
• The job exists but has no targets at all. A keep regex that matches nothing produces a job whose up series is absent rather than 0. Adding another target to that job changes nothing, because relabelling will drop it too.
• The exporter does not exist yet. Deploying the exporter and telling Prometheus about it are two changes. Doing them together means that when it does not work you have two suspects instead of one.

What this change actually costs

Filed as low risk, and that is honest — but only because of the pre-checks. Four things are being spent:

The reload is the interesting one. On a parse failure Prometheus keeps the previous configuration running: the server does not fall over, which is exactly why the failure is easy to miss. What it does do is hold back every other change sitting in that file — including, on a bad day, somebody’s alert-threshold fix. Your five-character typo is now their outage.

Step 1 — Prove the endpoint before you tell Prometheus about it

Run this from the Prometheus host. Not from your laptop, not from the target itself. The scrape is made by the server, so the server’s view of the network is the only one that counts.

# Substitute your own values before running:
TARGET=192.0.2.41:9100

curl -sf "http://$TARGET/metrics" -o /tmp/new.prom
head -12 /tmp/new.prom
promtool check metrics /tmp/new.prom

curl -sf fails with a non-zero exit rather than printing an HTTP error page, so a 404 from a wrong metrics_path is caught here rather than surfacing later as a scrape error nobody reads.

promtool check metrics parses the body against the exposition contract. This matters more than it looks: a scrape is parsed line by line and a single malformed line fails the whole scrape. There is no partial ingestion. An exporter that is 99% correct produces up of 0 and no data at all, which is indistinguishable from a dead host.

Step 2 — Count what you are about to ingest

grep -cv '^#' /tmp/new.prom

That is series per target. Multiply by the number of targets the job will have. Then compare against what the server is already holding:

# active series in the head block right now
prometheus_tsdb_head_series

# series created per hour - churn, which costs more than level
rate(prometheus_tsdb_head_series_created_total[1h])

A single node exporter is a rounding error. A per-container or per-request-path exporter across a large fleet is not, and the moment to discover that is now, on a workstation, rather than at the next memory ceiling. If the addition is a material fraction of the head budget, this stops being a configuration change and becomes a capacity conversation with an owner.

Step 3 — Decide which job it belongs to

The instinct is one job per exporter type. The useful rule is one job per (exporter class, SLA class), because scrape_interval, scrape_timeout and sample_limit are per job and nowhere else. The same node exporter belongs in a 15s job on infrastructure you page on, and in a 2m job with a longer timeout on a fleet behind slow links. Put them in one job and you have paid the high-resolution price for everything, or accepted the low-resolution answer for the things that page you.

If the new target fits an existing job’s interval, timeout and limits, add it to that job’s target list. If it does not, it needs its own job, and the reason belongs in a comment next to it.

Step 4 — Write the job, with its limits

scrape_configs:
  - job_name: node-core           # owner: platform; 15s because these page
    scrape_interval: 15s
    scrape_timeout: 10s           # must stay below the interval
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets:
          - 192.0.2.41:9100
          - 192.0.2.42:9100
        labels:
          site: lon-2
    sample_limit: 5000            # ~3x measured scrape_samples_scraped
    body_size_limit: 20MB

The limits are the part people skip. Their defaults are 0, which means unlimited: a bet that every exporter you will ever run stays well-behaved forever. Exporters add collectors in minor releases. Applications add label values under incident load. Set sample_limit from the number you measured in step 2, with headroom, and know what it does when it fires: the entire scrape is rejected and up goes to 0. The safety net and the outage look identical from the outside, which is why the limit needs a comment saying where its value came from.

Leave honor_labels and honor_timestamps at their defaults unless you can name the reason. With honor_labels: false (the default), an exporter that stamps its own instance has that value stored as exported_instance and Prometheus keeps the real one. Set it to true and two hosts claiming the same instance will append to the same series, with no error logged anywhere, because nothing failed.

Step 5 — Validate offline

promtool check config /etc/prometheus/prometheus.yml
promtool check service-discovery /etc/prometheus/prometheus.yml node-core

promtool check config links against the same loader the daemon uses at reload time, so what it accepts, the daemon accepts. It catches unknown keys, wrong types, missing required keys and duplicate blocks in about two hundred milliseconds. Expect:

SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config

What it does not do is connect to anything. A typo in a hostname is valid schema, and an expression that always returns empty is valid syntax. That is what step 1 was for.

promtool check service-discovery is the one people do not know about. It prints the final label set and the scrape URL for every target the named job resolves to — which is the difference between “the config parses” and “the config will scrape the thing I meant”.

Step 6 — Reload, and prove the reload happened

PROM=http://localhost:9090

curl -sf -X POST "$PROM/-/reload"
curl -sfG "$PROM/api/v1/query" --data-urlencode 'query=prometheus_config_last_reload_successful'

POST /-/reload requires the server to have been started with --web.enable-lifecycle. Without it the endpoint returns 403 and the reload is a SIGHUP instead — usually systemctl reload prometheus, if the unit is wired for it.

Step 7 — Verify the target, not the configuration

PROM=http://localhost:9090
JOB=node-core

curl -sf "$PROM/api/v1/targets?state=active" \
| jq -r --arg job "$JOB" '.data.activeTargets[]
    | select(.labels.job == $job)
    | [.labels.instance, .health, .lastError] | @tsv'

Read lastError verbatim. It is the highest-signal field in the whole subsystem and it names the stage that failed:

Then confirm the data, not just the plumbing:

# is the new instance up
up{job="node-core", instance="192.0.2.41:9100"}

# did the exporter try to name itself
{exported_instance!=""}

# does the sample count match what you measured by hand
scrape_samples_scraped{job="node-core"}

A target that is up with the wrong instance label is worse than a target that is down, because it looks finished.

Step 8 — Come back in an hour

Adding a target has a second-order effect that does not show up in the first minute. Check that the head grew by roughly what you predicted and that churn did not change shape:

prometheus_tsdb_head_series
rate(prometheus_tsdb_head_series_created_total[1h])
scrape_series_added{job="node-core"}

Sustained scrape_series_added — new series on every scrape — is the memory profile that kills Prometheus servers. It means the exporter is generating short-lived label values, and raw scrape volume is not the problem. If you see it, the fix is metric_relabel_configs on the label that is churning, or a conversation with whoever instrumented it.

Holding is a valid outcome

If step 2 says the addition is a significant fraction of the head budget, and the platform owner is not available, do not add it and then watch it. Record the measured series count, name the owner who has to approve the capacity, and set a time to come back. A target that was added at 17:00 on a Friday because it seemed small is the same target that is being hunted at 02:00 on Saturday when the server starts swapping.

Rollback

git revert --no-edit HEAD
promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST "http://localhost:9090/-/reload"

Reverting in the repository rather than editing the file in place is the whole point of configuration-as-code: the state you roll back to is the state CI validated, and it will still be there after the next deploy overwrites the server.

Two things rollback does not do. It does not free the memory the series already committed — those live in the head until they age out of it, so a server already under pressure may still need a restart. And it does not make the dashboards look untouched: a removed target’s series get stale markers and disappear from queries within the lookback window rather than flatlining at their last value. The gap is expected. It is not a second fault.

Common failures

That last row deserves the standing alert. A job with zero targets produces no up series to be 0, so nothing fires; the dashboard simply shows a gap, and humans rationalise gaps as dashboard problems. An absent() guard per job is the cheapest insurance in the stack.

Escalation

Escalate when the answer is not yours to give:

• curl from the Prometheus host cannot reach the endpoint but the owner says it is up. Network path, not Prometheus.
• The measured series count is a material fraction of the head budget. Capacity decision, with a bill.
• The reload is rejected and the offending block is not obvious within a few minutes. Other people’s changes are queued behind yours.
• The only configuration that works needs insecure_skip_verify: true. That is a decision to accept interception on the traffic, and it belongs to whoever owns the trust boundary, in writing.