Scrape Intervals and Resolution

A batch job pins a database host’s CPU for twenty seconds at 03:00 and the checkout latency SLO burns. In the post-incident review someone opens the node dashboard: the CPU graph is a flat, calm line. The host is scraped every 60 seconds. The spike happened between two samples and was gone before the next one. The monitoring was working exactly as configured; the configuration was simply blind to anything shorter than a minute.

The scrape interval is the sampling rate of your entire metrics pipeline. It sets the shortest event you can see, the accuracy of every rate(), the floor of your detection latency, and — linearly — your ingestion, storage and remote-write bill. Choosing it is a budgeting decision, not a default to accept.

What it is

Three intervals govern a Prometheus server, all configured in prometheus.yml:

• scrape_interval — how often Prometheus pulls /metrics from a target. Default 1m, set in global, overridable per scrape_config. This is the resolution of the stored data.
• scrape_timeout — how long a single scrape may take before Prometheus abandons it. Default 10s. Must be less than or equal to the scrape interval; a configuration that violates this fails to load.
• evaluation_interval — how often recording rules and alerts are evaluated. Default 1m, global only. This, plus the alert’s for: duration, sets alerting latency.

Resolution is the time distance between stored samples — at a 15s interval, each series holds one sample every 15 seconds. Everything downstream (PromQL windows, Grafana steps, alert evaluation) reasons on top of that grid and can never see finer than it.

Why a sysadmin cares

The interval is the answer to four operational questions:

1. Detection latency. An event is invisible until the next scrape. With 60s scrapes, mean time to detection has a floor of one minute before evaluation and for: even enter the picture.
2. Rate accuracy. rate() needs at least two samples inside its window. A window that is too small for the interval returns nothing; a window barely large enough returns a jagged, step-locked line.
3. Alert timeliness. MTTD floor = scrape_interval + evaluation_interval + for. Halve the interval and the whole stack gets faster; double it and no alert tuning can compensate.
4. Cost. Samples per second, storage growth, and remote-write bandwidth all scale inversely with the interval. 15s costs four times what 60s costs, for the same targets.

How it works

scrape_interval: 60s
samples:  |---------|---------|---------|---------|---------|
spike:            [==== 20s CPU burst ====]
result:   invisible — no sample lands inside the event

scrape_interval: 15s
samples:  |---|---|---|---|---|---|---|---|---|---|---|---|
spike:            [==== 20s CPU burst ====]
result:   one or two samples catch it; rate() over 1m shows a bump

Sizing rate() windows against the interval is the most common everyday decision. rate(http_requests_total[5m]) over 15s scrapes averages twenty samples — smooth and accurate. The same expression over 60s scrapes averages five. Drop the window to [30s] at 60s scrapes and most evaluations find one sample or none, producing gaps. The working rule: make every rate window at least two times the scrape interval, and prefer four times for anything you alert on. Grafana’s $__rate_interval variable exists to automate exactly this.

Staleness is the other interval-driven behaviour. When a target disappears, Prometheus marks its series stale; instant queries keep returning the last sample for up to the query lookback delta (--query.lookback-delta, default 5m) and then nothing. A dashboard can therefore show a plausible “last value” for several minutes after a target dies.

How to configure it

Intervals live in prometheus.yml, with per-job overrides for the targets that need them:

global:
  scrape_interval: 30s        # default for every job
  evaluation_interval: 30s    # rule and alert evaluation cadence

scrape_configs:
  - job_name: node
    scrape_interval: 15s      # fast-moving host metrics
    scrape_timeout: 10s       # must be less than scrape_interval
    static_configs:
      - targets: ['10.0.0.11:9100', '10.0.0.12:9100']

  - job_name: blackbox-http
    scrape_interval: 60s      # probes are expensive; once a minute
    scrape_timeout: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://shop.example.com/checkout']

  - job_name: slow-snmp
    scrape_interval: 120s     # the device takes 40s to answer
    scrape_timeout: 45s
    static_configs:
      - targets: ['10.0.0.1:9116']

The cost arithmetic, before and after a change, is simple and worth doing on paper:

100 node_exporter targets x 1,500 series each:

  60s interval: 100 x 1500 / 60 =   2,500 samples/s
  15s interval: 100 x 1500 / 15 =  10,000 samples/s
   1s interval: 100 x 1500 /  1 = 150,000 samples/s

At ~1.3 bytes per sample:
  15s -> roughly 1.1 GB/day     1s -> roughly 16.8 GB/day

How to validate it

# 1. Syntax and interval/timeout sanity before reload.
promtool check config /etc/prometheus/prometheus.yml
#   SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

# 2. Confirm the running server actually loaded the intervals.
curl -s http://localhost:9090/api/v1/status/config | jq -r .data.yaml \
  | grep -E 'scrape_interval|scrape_timeout|evaluation_interval'

# 3. Measure what scrapes really cost per target.
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=scrape_duration_seconds' | jq '.data.result[] | .metric.job, .value[1]'

# 4. Confirm actual ingestion rate matches the arithmetic.
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum(rate(prometheus_tsdb_head_samples_appended_total[5m]))' \
  | jq '.data.result[0].value[1]'

# 5. Check for scrapes failing against their budget.
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_target_scrapes_exceeded_sample_limit_total' \
  | jq '.data.result | length'

scrape_duration_seconds approaching scrape_timeout is the early warning that an interval is too ambitious for the exporter behind it.

How it can fail

1. Rate window smaller than two intervals. rate(x[30s]) at 60s scrapes. Symptom: panels show gaps or a jagged staircase; alerts flap between firing and no-data.
2. Timeout misconfigured against the interval. A timeout greater than the interval fails config validation outright; a timeout nearly equal to the interval on a slow exporter produces flapping up with “context deadline exceeded” on the targets page.
3. The 1s scrape. Someone sets scrape_interval: 1s to catch a transient. Symptom: exporter and Prometheus CPU climb, scrape_duration_seconds nears the timeout, samples per second jump 30x against 30s, and sample_limit breaches start failing whole scrapes.
4. Grafana min step below the interval. A panel steps at 10s over 60s data. Symptom: a smooth interpolated line that hides the sampling grid; gaps between scrapes become invisible and spikes that were never sampled look impossible.
5. Long interval, short SLA. 60s scrapes with for: 30s on the alert. Symptom: the alert fires minutes after users notice; the interval plus evaluation dominates the for budget.
6. Missed scrapes under load. Prometheus is CPU- or IO-bound and scrapes arrive late or not at all. Symptom: holes in recent data, stale “last values” on dashboards, and gaps in up itself.

How to troubleshoot it

1. Is the target scraped at all? /api/v1/targets shows health, lastScrape and lastError per target. A target whose lastError mentions timeouts is an interval/timeout problem, not a dead exporter.
2. How long do scrapes take? Query scrape_duration_seconds and compare it against the configured timeout and interval. Duration near the timeout means the exporter, the network or the interval is wrong.
3. Is ingestion keeping up? Compare sum(rate(prometheus_tsdb_head_samples_appended_total[5m])) against the arithmetic from your target and series counts. A shortfall means missed scrapes.
4. Are rules keeping up? prometheus_rule_group_iterations_missed_total greater than zero means evaluation_interval is too short for the rule load.
5. Do panel gaps correlate with the grid? If data holes appear at multiples of the scrape interval, suspect missed scrapes; if a rate panel gaps while the raw counter panel is continuous, the rate window is too small for the interval.

Security implications

An unauthenticated /metrics endpoint can be scraped by anyone, at any rate, as often as they like — your interval discipline does not constrain an attacker. A hard-pulling client is a cheap denial-of-service against a heavyweight exporter. Restrict exporter ports with network policy and put authentication on the scrape path where the threat model calls for it. Note the cost interaction: TLS or basic-auth scrapes (tls_config, basic_auth in the scrape config) add handshake and verification work to every scrape, so at very short intervals the security posture itself becomes a CPU line item.

Performance implications

Everything scales with samples per second: ingestion CPU, head memory churn, WAL writes, compaction work, disk growth and remote-write bandwidth. evaluation_interval has its own cost — halving it doubles rule evaluation work; heavy recording rules at a 5s evaluation interval can dominate a server’s CPU. The lever is never global: keep the fleet default moderate and buy resolution only for the jobs whose detection budget justifies it.

Production guidance

• Start with 30s global, 15s for hosts and critical services, 60s or longer for blackbox probes and slow exporters (SNMP devices, cloud APIs). Avoid sub-5s intervals outside a lab.
• Keep evaluation_interval equal to the scrape interval unless you have a measured reason to differ.
• Size every rate window at two to four times the interval of the job it queries; use $__rate_interval in Grafana so panels follow the configured interval automatically.
• Never raise scrape_timeout without checking it stays below the interval; a timeout that needs raising is telling you the exporter is slow.
• Alert on the platform’s own budget metrics: scrape_duration_seconds versus timeout, exceeded sample limits, and missed rule iterations.

Verification

You should now be able to answer:

• What is the difference between scrape_interval, scrape_timeout and evaluation_interval, and where is each configured?
• Why must a rate() window be at least twice the scrape interval?
• How do you compute the ingestion cost of moving a job from 60s to 15s scrapes?
• What symptoms distinguish a too-short rate window from missed scrapes under load?

Quiz