Prometheus Loss

A page fires at 04:11 for a payment backend that was green ten minutes ago. The Grafana dashboard for it is empty — no series, just a single panel that says No data. You open Prometheus directly and the targets page shows the targets are being scraped, but every series returns an empty result. The data directory has been wiped by a misconfigured systemd-tmpfiles timer that ran on the wrong path during a package upgrade.

This is the most common Prometheus loss shape in production: the local on-disk TSDB is gone, but the rest of the platform is intact. The recovery procedure is mechanical, but the order matters.

What it is

A Prometheus data-loss event is any condition in which the local TSDB no longer represents the truth the platform needs to query. Three shapes occur most often:

• On-disk corruption or deletion. Unclean shutdown, kernel panic, EBS reattach, or a hostile filesystem operation leaves the TSDB inconsistent or empty.
• Configuration loss without data loss. prometheus.yml, rules, or alerting rules are missing or corrupted. The TSDB is intact but the instance cannot scrape or alert.
• Remote_write receiver loss. The receiver is canonical for long-term storage; if it is gone, the local TSDB is the only copy, and it is bounded by --storage.tsdb.retention.time and --storage.tsdb.retention.size.

The recovery procedure depends on which shape you have. The first job is to identify it before doing anything destructive.

Why a sysadmin cares

Prometheus is the alert source. A page that fires because Prometheus is missing is worse than a page that fires because Prometheus is honest — the on-call engineer loses the most useful triage signal at the moment they need it most. The recovery procedure must restore the alert source before it restores the dashboards, because dashboards without alerts do not page anyone.

It is also the most-tested surface in the platform. Every other lesson assumes Prometheus is alive. A platform whose Prometheus recovery is slow is a platform whose incident-response time is bounded from below by the Prometheus recovery time.

How it works

Prometheus stores samples in per-time-range blocks on local disk. Each block is a self-contained directory of chunk files plus a metadata index. promtool tsdb snapshot produces a consistent copy that can be moved off-host and restored later by replacing the data directory.

+---------------------+        +----------------------+
| prometheus          |        | remote_write         |
| local TSDB          | -----> | receiver             |
| (--storage.tsdb.path)|        | (Mimir/Thanos/Cortex)|
+---------------------+        +----------------------+
         |                              |
         | snapshot                     | query
         v                              v
+---------------------+        +----------------------+
| backup / S3         |        | Grafana              |
+---------------------+        +----------------------+

Three restore paths:

1. Snapshot restore. Replace the data directory with the snapshot, restart, replay the WAL that Prometheus kept during the snapshot.
2. Empty restart with remote_write replay. Provision an empty Prometheus that remote_writes to the same receiver, and rely on the receiver to provide historical queries during the gap.
3. Receiver rebuild. If the receiver is the lost component, the recovery lives in the receiver runbook, not here.

The first path is right when the local TSDB is the only copy of the series. The second is right when the receiver is healthy.

Under the hood

How to configure it

A Prometheus instance that is built to be rebuilt has three properties: configuration in git, snapshot on a schedule, and a remote_write target that survives the local instance.

# /etc/default/prometheus -- the TSDB path and retention are
# flags. prometheus.yml has no key for either, and Prometheus
# refuses to start if one is invented.
ARGS="--storage.tsdb.path=/var/lib/prometheus/data \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=200GB"

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-eu-west-1
    replica: A

remote_write:
  - url: https://mimir.internal/api/v1/push
    basic_auth:
      username: prom-remote-write
      password_file: /etc/prometheus/mimir.password
    queue_config:
      capacity: 10000
      max_samples_per_send: 2000
      batch_send_deadline: 5s
      min_backoff: 30ms
      max_backoff: 5s
      retry_on_http_429: true

# /etc/prometheus/rules/service-level.yml
groups:
  - name: sli-recording
    interval: 30s
    rules:
      - record: sli:checkout_success:rate5m
        expr: |
          sum(rate(checkout_success_total[5m]))
          /
          sum(rate(checkout_attempts_total[5m]))

# /etc/cron.d/prom-snapshot
# Severity: SERVICE-IMPACT — produces a copy, does not stop the service.
0 * * * * prometheus /usr/bin/promtool tsdb snapshot /var/lib/prometheus/data \
  && find /var/lib/prometheus/data/snapshots -mtime +1 -delete

The remote_write target must exist independently of the local Prometheus. If both share a Kubernetes deployment, the receiver death is the same event as the local death.

How to validate it

Three checks, in order, before declaring recovery complete:

# 1. The process answers the ready endpoint.
curl -sf http://prom:9090/-/ready
# expected: 200 OK

# 2. Targets are being scraped and at least one is up.
curl -s http://prom:9090/api/v1/targets?state=active \
  | jq '.data.activeTargets | map({job: .labels.job, health: .health}) | .[0:5]'

{
  "health": "up",
  "job": "node"
},
{
  "health": "up",
  "job": "prometheus"
}

# 3. Rule evaluation is producing results.
curl -s http://prom:9090/api/v1/query?query=up \
  | jq '.data.result | length'
# expected: a number close to the target count

If any of those three returns an empty result or a 5xx, the recovery is not done.

How it can fail

• Snapshot taken from a corrupted TSDB. The snapshot is consistent but the data inside is bad. Restoring it restores the corruption.
• remote_write queue directory on the same volume as the TSDB. A volume loss takes both the on-disk history and the in-flight queue. The replay path is gone.
• Rules and configuration restored from different sources. Prometheus starts, scrapes, but evaluates no rules because the rules directory was missed.
• The receiver was a single instance with no replication. The recovery procedure rebuilds it from the same loss it just recovered from.
• The new instance has a different external_labels.replica or --enable-feature=exemplar-storage setting. Downstream queries that join on replica see two series for the same scrape.
• Backup transport tested only in staging. Production uses a different bucket or IAM role; the restore fails on a missing credential.

How to troubleshoot it

The diagnostic order:

1. Is the service running? (systemctl status prometheus, kubectl get pods -l app=prometheus)
2. Does the data directory exist and have content? (du -sh /var/lib/prometheus/data, promtool tsdb list /var/lib/prometheus/data)
3. Are the configuration and rule files present? (promtool check config /etc/prometheus/prometheus.yml)
4. Do targets exist? (/api/v1/targets)
5. Are samples flowing? (/api/v1/query?query=up)
6. Is remote_write succeeding? (/api/v1/status/config, check remote_write block for error field)

Security implications

The remote_write credential is the most sensitive artefact. A compromised credential lets an attacker write arbitrary series into the receiver, which can later mislead investigations. Treat it like any other write credential: rotate on schedule, scope to one receiver, audit usage.

Snapshots that leave the host — to S3, to a backup volume — must be encrypted at rest. The TSDB contains every metric the platform ever collected, which often includes high-cardinality labels with hostnames, paths, and request IDs.

Performance implications

A snapshot is a copy of every block. On a 30-day retention at 200 GB, a snapshot is up to 200 GB of IO. Run it off-peak, or use the --no-rename flag and ship the snapshot via a copy-on-write filesystem where the host supports it.

The remote_write queue is sized for steady state. A long receiver outage overruns the queue and drops samples. The queue depth is a signal that the receiver is unhealthy long before the receiver itself fails.

Production guidance

• Version the configuration and the rules in the same git repository. A restore that lands a 30-day-old rules file fires no alerts for things the on-call team already triaged.
• Take the first snapshot of the day before the dashboard refresh window. Snapshots are large and should not contend with peak scrape IO.
• Keep at least one historical snapshot per day for 7 days, plus one per week for 4 weeks. The RPO is the gap between the last good snapshot and the incident.

Verification

You should now be able to answer:

• What is the most common Prometheus data-loss shape, and how do you recognise it from the visible symptoms?
• When do you choose snapshot restore over remote_write replay, and what makes the choice obvious?
• What three endpoints confirm the recovered Prometheus is serving the truth rather than serving itself?
• Why does the remote_write credential deserve the same care as any other write credential?

Quiz