Stale Series

A host is in the dashboard at 14:00 and the same panel shows no line for the same host at 15:00. The host is still running. The exporter is still answering /metrics from curl. Prometheus is recording nothing because the scrape pool removed the target five minutes earlier and marked every series from that target as stale. Grafana, following the staleness marker, stops drawing the line. The operator looking at the dashboard sees a panel with one fewer series and no error.

The lesson is that “no line in Grafana” has three possible causes, only one of which is “the host is fine.” The other two are “Prometheus is not scraping” and “Prometheus is scraping but the target’s scrape failed and the series is now stale.” Both have to be visible to the operator; otherwise the missing line is misread as a successful state.

What it is

A stale series is a time series that Prometheus has stopped appending samples to but has not yet garbage-collected from the TSDB head. It carries an internal stale marker in the most recent sample: the value 1 on the synthetic label __name__ for the meta-metric scrape_stale_marker_present. PromQL hides stale samples from queries by default; the series stops contributing to instant vectors and rate functions.

Three pieces of metadata together describe a target’s lifecycle:

target down (host dead) -- scrape pool removes target
                             |
                             v
                         series becomes stale  <-- 5m after last
                             |                     successful scrape
                             v
                         garbage collected    <-- after
                             |                  stale:max_age
                             v
                         head does not see series any more

The window between “stale” and “garbage-collected” is the operational gap. The series is invisible to Grafana but still sits in the TSDB head block and still counts towards prometheus_tsdb_head_series.

Why a sysadmin cares

Stale series are invisible during normal operation. They become visible the first time an alert or a dashboard depends on the absence of a series rather than its value.

• absent(up{job="foo"}) is the canonical “is the scrape actually working” alert. It fires when no series with the job label foo exists in the head. If the series is stale, it still exists in the head, and absent() returns an empty vector. The alert does not fire. The operator is told the job is fine when it is not.
• up == 0 and up == 1 are complementary. A scrape that is failing produces up == 0. A scrape that has stopped because the target is gone produces a stale series whose last sample is the last successful one — up is whatever it was the last time the scrape succeeded. The two failure shapes are indistinguishable from the metric alone; the stale marker is what distinguishes them.
• Recording rules with for: 5m use the staleness of the input series. A stale input produces a stale output. The alert evaluation sees an empty vector and either fires on threshold (> 0 against empty) or quietly returns no result, depending on the rule shape.

The lesson returns to each of these in production depth.

How it works

The scrape pool maintains a per-target state machine:

         successful scrape
   HEALTHY  --------------->  next scrape sample appended
       ^
       | 5m after last successful sample
       |
     STALE                   <-- last successful sample is marked
       |                        stale; future queries do not see
       v                        it but the head series count holds
   REMOVED                 <-- target removed from scrape pool;
                                series will be GC'd after
                                stale:max_age (default 1h)

The 5-minute staleness threshold is the production default. It is set globally with --scrape.timeout and --scrape.staleness on the Prometheus command line:

prometheus \
  --storage.tsdb.retention.time=15d \
  --web.console.libraries=/usr/share/prometheus/console_libraries \
  --web.enable-lifecycle \
  --scrape.staleness=5m

The scrape pool carries an additional per-target override scrape_interval, scrape_timeout, and sample_limit in scrape_configs. The 5-minute stale marker is independent of the scrape interval: a target scraped every 15 seconds still becomes stale 5 minutes after its last successful scrape.

How to configure it

Two configuration surfaces: the scrape job and the HA flag.

Per-job staleness override

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  # Default staleness for every scrape job. Five minutes is the
  # production default and is correct for a 15-second scrape
  # interval. Reduce only when the underlying target emits more
  # often and operator alerting must react faster than 5m.
  external_labels:
    cluster: prod-eu-1

scrape_configs:
  - job_name: node
    # A 10-second scrape interval needs at least 1m windows in
    # the rate functions that consume this data. The staleness
    # here is independent of the window.
    scrape_interval: 15s
    static_configs:
      - targets: ['host-a:9100', 'host-b:9100']

  - job_name: batch-exporters
    # Some batch jobs are intentionally silent between runs.
    # A 30-minute staleness here lets the series persist between
    # runs and stops the absents() alerts from firing between
    # runs. The trade-off is the operator does not know between
    # runs whether the batch is running; that is asserted with a
    # separate "last successful completion" metric, not with up.
    scrape_staleness: 30m
    static_configs:
      - targets: ['batch-a:9100']

HA deduplication with —query.replica-label

Two Prometheus replicas scraping the same target produce two parallel time series for every exported metric. query.replica-label on the receiving side deduplicates them at query time by dropping all but the sample from the replica whose value of the named label is lexicographically highest.

prometheus-a  --enable-feature=agent-mode
prometheus-b  --enable-feature=agent-mode
                \                    \-- both push to remote_write
                 \                       configured identically
                  +----------+----------+
                             |
                       remote storage
                   (Thanos Receive, Mimir, etc.)
                             |
                             v
                        Grafana
                             |
                       --query.replica-label=prometheus_replica

The flag must match the label the replicas stamp on every sample. The convention is to set it in the receiving side:

# On the query side (Grafana datasource, sidecar)
prometheus \
  --query.replica-label=prometheus_replica
# On each scraping replica
prometheus \
  --enable-feature=agent-mode \
  --agent.remote-write.replica-label=prometheus_replica

A second flag, external_labels, must include the same label. Without both, the deduplication step is a no-op.

# Each replica stamps its own external label so samples can be
# distinguished at the receiving side
global:
  external_labels:
    region: eu-west-1
    prometheus_replica: 'a'   # each replica has a unique value

The interaction with the scrape lifecycle is that --query.replica-label does not affect whether the series is stale. It deduplicates at query time; the underlying scrape state machine is unchanged.

How to validate it

Three commands confirm the staleness pipeline is correct.

# 1. Inspect what the scrape pool currently sees. The label
#    health="up" or "down" is added by Prometheus itself and
#    never disappears; both up and down targets appear here,
#    with the up=1 or up=0 metric distinguishing them.
curl -sf http://prometheus:9090/api/v1/targets \
  | jq '.data.activeTargets[] | {job:.labels.job, instance:.labels.instance, health:.health, lastError:.lastError}'

# Example output (illustrative)
# {
#   "job": "node",
#   "instance": "host-a:9100",
#   "health": "up",
#   "lastError": ""
# }
# {
#   "job": "node",
#   "instance": "host-b:9100",
#   "health": "down",
#   "lastError": "context deadline exceeded"
# }

# 2. Find stale series in the head. The synthetic metric is
#    generated on the fly from the stale marker; a non-empty
#    result is the catalogue of series that are no longer
#    contributing but are still in the head.
curl -sf http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count_over_time(scrape_stale_marker_present[5m])'
# {"status":"success","data":{"resultType":"vector","result":[
#   {"metric":{"job":"node","instance":"host-b:9100"},
#    "value":[1724000000,"1"]}
# ]}}

# 3. Confirm HA deduplication is in effect. The replica label
#    should appear on every sample.
curl -sf http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count by (prometheus_replica) (up{job="node"})'
# {"status":"success","data":{"resultType":"vector","result":[
#   {"metric":{"prometheus_replica":"a"},"value":[1724000000,"6"]},
#   {"metric":{"prometheus_replica":"b"},"value":[1724000000,"6"]}
# ]}}

Three follow-on checks:

# Confirm the head series count is what the operator expects.
curl -sf http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=prometheus_tsdb_head_series'
# A number that grows steadily is a sign that stale series are
# not being garbage collected; the GC step is failing or the
# staleness window is set too long.

# Confirm the targets API is reporting the target correctly.
# 'health="down"' here corresponds to up=0 in the metrics; the
# scrape is failing but the target is still in the pool.
# 'lastScrape' older than 5 minutes plus 'scrapeStaleness' in
# the output would mean the series is stale.

# Compare the up metric to the targets API result count.
# `count(up == 1)` and 'health="up"' count should match. A
# mismatch means the scrape pool has stopped scraping some
# targets that are still in the configuration.

How it can fail

1. The scrape pool is failing silently. Target is reachable for curl but Prometheus cannot reach it. The target is removed from the pool; series are marked stale; Grafana stops drawing. up == 1 does not fire because the target is not in the pool. Symptom: nothing in dashboards, no alert, no error log. Detected by count(up{job="node"}) < expected_instances.
2. The scrape interval exceeds the staleness window. An exporter that is configured with scrape_interval: 10m and the global staleness: 5m will mark its own series stale on every cycle. Every rate function over a window longer than 5m returns no data. Symptom: panels go to “no data” between scrapes; alerts misfire at the boundary.
3. absent() does not see stale series. The series is in the head, marked stale, and excluded from query results. absent(up{job="foo"}) returns an empty vector because the stale sample still exists. The alert that should have fired on “no scrape” stays silent. Symptom: a target that has been unreachable for an hour still does not page.
4. HA replicas scrape the same target and cannot reconcile. Two replicas, neither configured with --query.replica-label, push the same series into the remote store. Cardinality doubles. Symptom: doubling of series counts in the remote store without a doubling of the underlying workload.
5. Long scrape_timeout causes the staleness window to fire during a slow scrape. A scrape that takes 50 seconds on a 10-second interval generates fewer samples than the operator expects. The target gets marked stale on the scrape pool’s accounting even though the underlying exporter is fine. Symptom: intermittent up == 0 and stale markers on a healthy target.
6. Recording rules consume stale series. A rule with for: 5m and absent_over_time(up[10m]) against an exporter that has been stale for 30 minutes interprets “stale” as “always present.” Symptom: alerts that should fire do not; alerts that should not fire do.

How to troubleshoot it

Ordered diagnostics:

1. Open the targets API. /-/targets. Look for health: "down" rows. The label lastError carries the actual failure reason from the last scrape attempt. The label scrapeStaleness would be set if the global default had been overridden for this job.
2. Compare count(up == 1) to the inventory. If the count is lower than the number of expected hosts, the scrape pool is dropping targets. The drop is silent.
3. Inspect scrape_stale_marker_present. The synthetic metric enumerates stale series. A non-empty result is always actionable.
4. Inspect prometheus_tsdb_head_series. A climbing number against an expected steady-state is the GC step failing or the staleness window set too long. The fix is prometheus_tsdb_head_series versus the same number from yesterday.
5. For HA setups, compare samples by replica label. If the receiving side sees the same series for both prometheus_replica="a" and prometheus_replica="b" without --query.replica-label, the dedup is not in effect.
6. Replay against a fixture. promtool test rules will consume stale samples the same way the live engine does; a fixture that asserts the alert under stale conditions is the test for the symptoms above.

Security implications

Stale series do not change the attack surface directly, but they interact with the rate-limits on the scrape pool. A malicious exporter that emits a unique label per scrape (a user_id or a request_id) and then stops emitting will not be flagged stale because the series count is bounded by its idle interval. The garbage-collection step is the only line of defence; verify the retention window is short enough that the cost is bounded.

Performance implications

Stale series live in the head block and inflate prometheus_tsdb_head_series even though they do not appear in dashboards. Two implications:

• A scrape pool that fails for hours can leave a non-trivial fraction of the head consumed by stale series. Query planning that touches those series will pay the cost of filtering them out.
• The compaction step carries stale markers into the level-1 blocks. They cost disk and CPU on every query that crosses a block boundary.

The production discipline is to keep the staleness window short enough that the upper bound on stale series is small and the GC step has time to reclaim them before the disk fills.

Production guidance

• Use the 5-minute default unless the alert SLA is shorter than 5m or the scrape interval is intentionally long. For batch exporters with multi-hour cycles, set scrape_staleness in the job definition to match the cycle.
• Alert on absent_over_time(up{job="..."}[10m]), not on absent(up{job="..."}). The absent-over-time version survives the stale-marker boundary.
• Alert on count(up{job="..."}) < expected_count for fleet-completeness. Even when the scrape pool has removed a target, the count metric can be asserted against a service-owned inventory.
• For HA setups, set --query.replica-label on the query side and external_labels.prometheus_replica on the scraping side to the same label name. Verify with count by (prometheus_replica) (up{job="node"}) in the receiving side.
• Audit prometheus_tsdb_head_series against a known steady-state. A steadily climbing number is the GC step failing.

Verification

You should now be able to answer:

• What is the difference between a missing series and a stale series?
• Why does absent() return an empty vector for a target that has been unreachable for an hour?
• Where in prometheus.yml is the per-job staleness configured, and what is the production default?
• What does --query.replica-label do, and why is its label name also needed in external_labels on the scraping replicas?
• How do you enumerate stale series in a running Prometheus?

Quiz