Recovery

The mitigation rule ships at 06:12. The on-call engineer runs promtool check config, the syntax passes, and the engineer sends SIGHUP. Twenty seconds later the head series count is falling; sixty seconds later it is below the budget. RSS is descending. The OOM alert will clear within two minutes. The recovery is the boring part of the incident, and that is what makes it worth doing carefully.

Recovery in a cardinality incident means getting the head block back under budget and confirming the platform is healthy enough to resume evaluating alerts and serving queries. The discipline is to verify, not to assume.

What recovery means here

Recovery is the set of checks that confirm the platform is back. It is not the restart. The restart, when needed, is a step inside recovery; the verification is the larger part.

Three things must be true before the incident is closed:

• The configuration reload succeeded.
• The head series count is back under budget.
• Process RSS is descending and has settled near its baseline.

If any of these is not true, recovery has not happened. The on-call rotation is still on the hook.

Why a sysadmin cares

Recovery is the difference between an incident and an outage. An incident that recovers cleanly is a one-hour event with a clear write-up. An incident that “looks fine” but has not been verified is a recurring incident that returns next quarter because nobody proved the underlying state was healthy.

How it works

The reload and the verification are two separate stages. Conflating them is the most common recovery mistake.

Apply the mitigation rule to prometheus.yml
    |
    v
promtool check config (syntax)
    |
    v
SIGHUP (in-place reload)
    |
    v
prometheus_config_last_reload_successful == 1
    |
    v
prometheus_tsdb_head_series descending
    |
    v
process_resident_memory_bytes descending
    |
    v
Recording rules evaluating
    |
    v
Alertmanager receiving alerts
    |
    v
Incident closed

Each step has a single confirmation. None of them is optional; each gates the next.

How to configure it

Recovery does not require configuration changes; it requires verifications. The shell session below is the canonical recovery procedure for a cardinality incident after the mitigation rule has shipped.

# CONFIGURATION. Validate the new configuration file.
promtool check config /etc/prometheus/prometheus.yml

# CONFIGURATION. Send SIGHUP to the running Prometheus.
PROM_PID="$(pidof prometheus)"
[ -n "$PROM_PID" ] && kill -HUP "$PROM_PID"

# READ-ONLY. Wait two scrape intervals for the reload to apply.
sleep 30

# READ-ONLY. Confirm the reload succeeded.
curl -s http://prometheus:9090/metrics \
  | grep '^prometheus_config_last_reload_successful '
# Expect: prometheus_config_last_reload_successful 1

# READ-ONLY. Confirm the head series count is descending.
curl -s http://prometheus:9090/metrics \
  | grep '^prometheus_tsdb_head_series '

# READ-ONLY. Confirm process RSS is descending.
curl -s http://prometheus:9090/metrics \
  | grep '^process_resident_memory_bytes '

# READ-ONLY. Confirm a recording rule is evaluating.
curl -s 'http://prometheus:9090/api/v1/query?query=up' \
  | jq '.data.result | length'

The kill -HUP is the in-place reload. A full restart (stop, start) is only required if the reload does not produce the expected descent within five minutes, or if the wrapper blocks SIGHUP.

How to validate it

Validation here means four concrete checks:

# READ-ONLY. Reload success.
curl -s http://prometheus:9090/api/v1/status/runtimeinfo \
  | jq '.status'

# READ-ONLY. Head series below budget.
curl -s http://prometheus:9090/api/v1/query?query=prometheus_tsdb_head_series \
  | jq '.data.result[0].value[1]'

# READ-ONLY. Process RSS below 80 percent of host memory.
ps -o pid,rss,cmd -p "$(pidof prometheus)"

# READ-ONLY. TSDB head compaction progressing.
curl -s http://prometheus:9090/metrics \
  | grep '^prometheus_tsdb_compactions_failed_total '
# Expect: 0

Illustrative output during recovery:

$ curl -s http://prometheus:9090/metrics \
    | grep '^prometheus_config_last_reload_successful '
prometheus_config_last_reload_successful 1

$ curl -s http://prometheus:9090/metrics \
    | grep '^prometheus_tsdb_head_series '
prometheus_tsdb_head_series 2.31e+06

$ ps -o pid,rss,cmd -p "$(pidof prometheus)"
    PID   RSS CMD
   1831 4827136 /usr/bin/prometheus

Reload succeeded. Head series count is 2.31 million (down from 18.4 million). RSS is 4.8 GiB (down from 11 GiB). The incident is over.

How it can fail

Six recovery failure shapes:

1. Reload reported as successful but rule did not apply. prometheus_config_last_reload_successful is 1 but the rule was syntactically valid yet semantically wrong (wrong regex, wrong action). The head series count does not fall.
2. Wrapper blocks SIGHUP. The process manager or container runtime intercepts SIGHUP and does not forward it. The reload never happened. The alert continues to fire.
3. Head compaction is stalled. The new rule is in place but the head block cannot compact to disk because of disk pressure or WAL corruption. RSS stays high. The next OOM follows within an hour.
4. Restart loop continues because the source is still pushing. The relabel rule applies on scrape; if the upstream scrape target still returns the bad label, the rule rewrites the series but the metric family is still large. The head count falls but does not normalise.
5. Memory descent is slow. The Go runtime releases memory back to the OS only when it can prove it will not need it again. A RSS descent from 11 GiB to 4 GiB may take ten to fifteen minutes even after the head count has normalised.
6. Recording rules do not re-evaluate. A recording rule that depended on the dropped label returns no rows. The alerting layer is silent. The recovery appears clean but the platform is partially blind.

How to troubleshoot it

The diagnostic order when recovery does not progress:

1. Confirm the reload. prometheus_config_last_reload_successful must be 1. If 0, the configuration file has a syntax error; promtool check config will report it.
2. Confirm the rule matched. Re-run the investigation query for the offending series. The result must be empty.
3. Confirm the head is compacting. prometheus_tsdb_head_series_removed_total must be climbing.
4. Confirm RSS is descending. If RSS is stable, the process has memory it is not releasing. Check go_goroutines for a runaway loop.
5. Confirm recording rules. Query the rule’s output directly via /api/v1/query. If empty, the rule broke; the rule file is the next place to look.

Security implications

Recovery touches the configuration file. The change-management audit trail is part of the recovery: the prometheus.yml diff, the change ticket, and the person who approved the mitigation rule are all part of the incident record. A recovery that omits the audit trail is incomplete from a post-incident-review standpoint, even if the platform is back.

The head block and WAL are sensitive surfaces during recovery: they contain the offending label values for the entire incident window. Backups taken during the incident carry the same label values. Retention policy and backup encryption apply as usual.

Performance implications

The SIGHUP reload is sub-second. The head compaction that removes the offending series runs on a background goroutine and is bounded by the size of the head block and the configured compaction interval. A 20-million-series head block takes longer to compact than a 2-million-series one; this is why the RSS descent is measured in minutes, not seconds.

A full restart is more expensive: the WAL must be replayed, the head block must be rebuilt, and the first scrape cycle after restart is delayed until replay completes. A restart should be reserved for cases where SIGHUP cannot apply.

Production guidance

• Always check prometheus_config_last_reload_successful after SIGHUP. The signal is in the metric, not in the exit code of kill.
• Wait for RSS to descend before closing the incident. A fast head count drop with a slow RSS drop is normal; a stable RSS is not.
• Stage the reload on a non-production Prometheus first. The same recording rules must be loaded; the rule’s blast radius is only visible at evaluation time.
• Document the mitigation in the incident record. The follow-up ticket for the source-side fix is part of the recovery deliverable.
• Keep a “cardinality incident” runbook entry. The diagnostic order (reload success, head count, RSS, rules) is the same every time.

Verification

• Which Prometheus 2.55.x metric confirms a SIGHUP reload was applied?
• What is the difference between a SIGHUP reload and a full restart, in recovery terms?
• What is the most common recovery shape?
• Why must the on-call engineer wait for RSS to descend before closing the incident?

Quiz