Lab: Recover from a Cardinality Incident

Objective

By the end of this lab you will have run a cardinality incident end to end, against a clock, on a Prometheus you deliberately made small enough to break. The point is not that you can recite the four phases; it is that you will know how long each one takes on your own hardware, which of them you can do while tired, and which of the three mitigation levers you would actually reach for at 03:00 — including the one where the correct action is to do nothing yet and say so out loud.

Architecture

Deliberately the same two-component shape as any real monitoring host: a Prometheus with a memory ceiling, and one target whose exposition you control. The ceiling is what makes the lab honest — a Prometheus with unlimited memory does not have this incident, it just gets slow and expensive somewhere you cannot see.

  host                                    compose network: obs-recovery
  ----                                    ----------------------------
  bin/explode.sh  ---writes--->  ./textfile/storefront.prom
                                              |
                                        node-exporter (textfile only)
                                              |
  127.0.0.1:9090 --> prometheus  <--scrape----+
                     mem_limit: 512m
                     --web.enable-lifecycle
                     --web.enable-admin-api

Two flags matter operationally. --web.enable-lifecycle is what makes the mitigation a curl instead of a restart, and --web.enable-admin-api is what makes delete_series available at all — a flag that is off by default for reasons Task 8 takes seriously.

Requirements

Linux or macOS with Docker Engine 28.x and the Compose v2 plugin. Ports 9090 and 9100 free on 127.0.0.1.
bash, awk, python3, and about 2 GB of RAM available to Docker.
A way to note the time. A terminal with timestamps, a paper notebook, or date in a second window — the timeline is a deliverable, and reconstructing it afterwards from memory is exactly the thing incident reviews cannot do.
Roughly 30 minutes of the 90 is waiting: for the head to settle, for a WAL replay, for an alert’s for: clause. That waiting is the lab.

Versions: Prometheus 2.55.x, node_exporter 1.8.x.

Scenario

You are on call for the platform. The storefront team deploys at 15:28. At 15:44 your phone goes off: HeadSeriesGrowthAnomaly, an alert you wrote three months ago and have never seen fire.

You have twenty minutes before the daily traffic peak, a Prometheus that also evaluates every other team’s alerts, and one decision to make in the right order.

Tasks

Task 1: Build the stack, and pre-stage the response

The pre-staging is not scene-setting. The difference between a twenty-minute incident and a three-hour one is almost entirely whether the alert, the drop rule and the reload path existed before 15:44 — none of which is discoverable under paging.

WORKDIR="$HOME/obs-recovery-lab"
mkdir -p "$WORKDIR"/{prometheus/rules,textfile,bin}
cd "$WORKDIR"
chmod 755 textfile

bin/explode.sh — the storefront exporter. The session_id label is the deploy at 15:28.

#!/usr/bin/env bash
# Emits the storefront service's metrics into the textfile collector.
# Usage: explode.sh SESSIONS    (0 = the healthy pre-deploy shape)
set -euo pipefail

SESSIONS="${1:-0}"
OUT_DIR="$(cd "$(dirname "$0")/.." && pwd)/textfile"
TMP="$OUT_DIR/storefront.prom.$$"

ROUTES="/ /cart /checkout /pay /account /search /help /terms"
CODES="200 302 404 500"

{
  echo "# HELP storefront_http_requests_total Storefront requests."
  echo "# TYPE storefront_http_requests_total counter"
  for r in $ROUTES; do
    for c in $CODES; do
      if [ "$SESSIONS" -eq 0 ]; then
        echo "storefront_http_requests_total{route=\"$r\",status=\"$c\"} 1"
      else
        s=1
        while [ "$s" -le "$SESSIONS" ]; do
          printf 'storefront_http_requests_total{route="%s",status="%s",session_id="s%07d"} 1\n' \
            "$r" "$c" "$s"
          s=$((s + 1))
        done
      fi
    done
  done
} > "$TMP"

mv "$TMP" "$OUT_DIR/storefront.prom"
chmod 644 "$OUT_DIR/storefront.prom"

compose.yaml. The memory limit is the whole experimental apparatus: pick a value your host can spare, and write down what you picked.

name: obs-recovery

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=1d'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    ports:
      - '127.0.0.1:9090:9090'
    mem_limit: 512m
    restart: unless-stopped

  node-exporter:
    image: quay.io/prometheus/node-exporter:v1.8.2
    command:
      - '--collector.disable-defaults'
      - '--collector.textfile'
      - '--collector.textfile.directory=/textfile_collector'
    volumes:
      - ./textfile:/textfile_collector:ro
    ports:
      - '127.0.0.1:9100:9100'
    restart: unless-stopped

volumes:
  prometheus_data:

prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: storefront
    scrape_interval: 15s
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          team: storefront

prometheus/rules/cardinality.yml — the alert that pages you at 15:44. Growth relative to an hour ago catches an explosion without needing a fixed threshold that goes stale as the platform grows.

groups:
  - name: cardinality-incident
    interval: 15s
    rules:
      - alert: HeadSeriesGrowthAnomaly
        expr: |
          (prometheus_tsdb_head_series
            - prometheus_tsdb_head_series offset 10m)
            / prometheus_tsdb_head_series offset 10m > 0.25
        for: 2m
        labels: {severity: critical, team: observability}
        annotations:
          summary: 'Head series grew more than 25 percent in 10 minutes'
          runbook: 'https://runbooks.example.com/cardinality'

      - alert: RuleEvaluationsMissed
        expr: increase(prometheus_rule_group_iterations_missed_total[5m]) > 0
        for: 1m
        labels: {severity: warning, team: observability}
        annotations:
          summary: 'Rule groups are missing evaluations; alerting is degraded'

drop-rules.snippet.yaml — the mitigation, written and reviewed today so that at 15:47 it is a paste rather than an act of authorship:

# Paste into the offending job's scrape_config. Reviewed 2026-08-19.
# labeldrop matches label NAMES and takes no source_labels.
    metric_relabel_configs:
      - action: labeldrop
        regex: '(session_id|user_id|request_id|trace_id|correlation_id)'
    sample_limit: 20000

Start in the healthy shape and let it settle:

cd "$HOME/obs-recovery-lab"
chmod +x bin/explode.sh
./bin/explode.sh 0
docker compose up -d
sleep 120
docker compose ps

Task 2: Measure the ceiling you are about to cross

Every number in this lab should be yours. Measure the per-series cost of this instance, then compute how many series 512 MiB can hold.

cd "$HOME/obs-recovery-lab"
cat > bin/q <<'SH'
#!/usr/bin/env bash
curl -sfG http://127.0.0.1:9090/api/v1/query --data-urlencode "query=$1" |
  python3 -c 'import json,sys; r=json.load(sys.stdin)["data"]["result"]; print(r[0]["value"][1] if r else "no data")'
SH
chmod +x bin/q

./bin/q 'prometheus_tsdb_head_series'
./bin/q 'process_resident_memory_bytes{job="prometheus"}'
docker stats --no-stream obs-recovery-prometheus-1

python3 - <<'PY'
import json, urllib.parse, urllib.request

def q(expr):
    url = "http://127.0.0.1:9090/api/v1/query?" + urllib.parse.urlencode({"query": expr})
    result = json.load(urllib.request.urlopen(url))["data"]["result"]
    return float(result[0]["value"][1]) if result else 0.0

rss = q('process_resident_memory_bytes{job="prometheus"}')
series = q('prometheus_tsdb_head_series')
limit = 512 * 1024 ** 2

print("baseline head series : %d" % series)
print("baseline resident    : %.1f MiB" % (rss / 1024 ** 2))
print("headroom to the limit: %.1f MiB" % ((limit - rss) / 1024 ** 2))
print()
print("Assume 4 KiB per additional series (lesson 01's planning figure).")
print("Series the headroom buys at that rate: %d" % ((limit - rss) / 4096))
PY

Write the last number in notes.md as your ceiling estimate. It is an estimate built on a planning figure rather than a measurement, because at a few hundred series the fixed overhead dominates and a measured per-series cost here would be meaningless. That is itself worth knowing: the honest per-series number can only be derived from an instance whose head is already large, which is why capacity planning for a new platform is genuinely hard and why the first real number you get is usually the one from an incident.

Task 3: Ship the deploy

Note the wall-clock time. From here to the end of Task 9 you are writing a timeline.

cd "$HOME/obs-recovery-lab"
date '+%H:%M:%S deploy'

# Roughly 1.5x the ceiling estimate from Task 2, divided by the 32 label
# combinations the generator already produces (8 routes x 4 status codes).
# Example: a 30000-series ceiling estimate -> 45000 / 32 -> about 1400.
./bin/explode.sh 1400

# A one-line status line, so watch does not need nested quoting.
cat > bin/head-status <<'SH'
#!/usr/bin/env bash
Q="$(dirname "$0")/q"
printf '%s  ' "$(date '+%H:%M:%S')"
printf 'series=%s  ' "$("$Q" 'prometheus_tsdb_head_series')"
printf 'created/s=%s  ' "$("$Q" 'rate(prometheus_tsdb_head_series_created_total[2m])')"
printf 'up=%s\n' "$("$Q" 'up{job="storefront"}')"
SH
chmod +x bin/head-status

watch -n 5 ./bin/head-status

Leave that running and watch the number climb. This is the only part of the lab where sitting and watching is the correct action, and it is worth doing once: the shape of the curve — a step, not a ramp — is what distinguishes an instrumentation explosion from organic growth on a graph you see later.

Task 4: Phase 1, recognise

Press Ctrl-C on the watch. Note the time. Now answer one question before touching anything: is this cardinality, or is it something else that also makes Prometheus unhappy?

cd "$HOME/obs-recovery-lab"
date '+%H:%M:%S recognise'

./bin/q 'prometheus_tsdb_head_series'
./bin/q 'process_resident_memory_bytes{job="prometheus"}'
./bin/q 'rate(prometheus_tsdb_head_series_created_total[5m])'
./bin/q 'increase(prometheus_rule_group_iterations_missed_total[10m])'
./bin/q 'scrape_duration_seconds{job="storefront"}'

# Did the container die? 137 is a SIGKILL, which on a limited container is
# usually the kernel OOM killer.
docker compose ps
docker inspect -f 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}' obs-recovery-prometheus-1

Whether or not your container was OOM-killed, record it. If it was, restart: unless-stopped will have brought it back, into a WAL replay, into a still-bad exposition — which is the loop lesson 05 calls restart-first, arrived at without anyone choosing it.

Task 5: Phase 2, identify

Three commands name the metric family, the job and the label. None of them changes anything.

cd "$HOME/obs-recovery-lab"
date '+%H:%M:%S identify'

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=topk(5, count by (__name__) ({__name__=~".+"}))' \
  | python3 -c '
import json, sys
for r in json.load(sys.stdin)["data"]["result"]:
    print("%10s  %s" % (r["value"][1], r["metric"].get("__name__", "?")))'

curl -sf http://127.0.0.1:9090/api/v1/status/tsdb | python3 -c '
import json, sys
d = json.load(sys.stdin)["data"]
print("--- distinct values by label name")
for e in d["labelValueCountByLabelName"][:8]:
    print("%10s  %s" % (e["value"], e["name"]))'

./bin/q 'count by (job) ({__name__=~"storefront_.+"})'

Then anchor it to a time. In production this is a deploy marker and process_start_time_seconds for the emitting target; here the exporter did not restart, so the evidence is the file the generator wrote:

cd "$HOME/obs-recovery-lab"
./bin/q 'node_textfile_mtime_seconds'
date -d "@$(./bin/q 'node_textfile_mtime_seconds' | cut -d. -f1)" 2>/dev/null \
  || date -r "$(./bin/q 'node_textfile_mtime_seconds' | cut -d. -f1)"
stat -c '%y %n' textfile/storefront.prom 2>/dev/null || stat -f '%Sm %N' textfile/storefront.prom

You now have the sentence the incident review needs: a change at HH:MM added the label session_id to storefront_http_requests_total on job storefront, and head series went from N to M. Put it in notes.md before you mitigate, because mitigation destroys some of the evidence.

Task 6: Phase 3, mitigate — stop the inflow first

Paste the pre-staged snippet into the storefront job. The whole job should now read:

  - job_name: storefront
    scrape_interval: 15s
    sample_limit: 20000
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          team: storefront
    metric_relabel_configs:
      - action: labeldrop
        regex: '(session_id|user_id|request_id|trace_id|correlation_id)'

Then do the two things people skip: validate, and confirm the running process actually loaded it.

cd "$HOME/obs-recovery-lab"
date '+%H:%M:%S mitigate'

docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && echo reloaded

# The file on disk proves nothing. Read what the process is running.
curl -sf http://127.0.0.1:9090/api/v1/status/config \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["yaml"])' \
  | grep -A4 metric_relabel_configs

cd "$HOME/obs-recovery-lab"
sleep 60

# Inflow: what the target served versus what survived relabeling.
./bin/q 'scrape_samples_scraped{job="storefront"}'
./bin/q 'scrape_samples_post_metric_relabeling{job="storefront"}'
./bin/q 'up{job="storefront"}'

# Level: unchanged. This is the point of the task.
./bin/q 'prometheus_tsdb_head_series'
./bin/q 'rate(prometheus_tsdb_head_series_created_total[5m])'

Also note what your labeldrop did to the scrape’s health. session_id was the only label distinguishing those samples from one another; with it removed they collapse onto identical label sets within a single scrape. Record up, lastError from /api/v1/targets, and prometheus_target_scrapes_sample_duplicate_timestamp_total. If the scrape is now failing outright, that is still a successful mitigation of the platform problem — the storefront has lost its own monitoring, which is the correct allocation of pain during an incident, and it is the argument for the sample_limit line that was in the same snippet.

Task 7: The decision — hold, restart, or neither

Three options. Two of them are actions and one of them is the answer more often than people expect.

Hold. The inflow is stopped and the head drains itself at the next truncation. Nothing else is required. Hold is correct when the instance is not thrashing: memory is below the limit, scrape durations are inside the interval, and rule groups are evaluating on schedule.

Hold is a decision, not an absence of one, and it is only safe if you make it explicitly:

DECISION 16:12 — HOLD.
  Inflow stopped at 16:09, confirmed by /api/v1/status/config.
  head_series 41k, flat. RSS 380 MiB of 512 MiB. Rule groups evaluating.
  Owner: your name here. Re-evaluate at 17:15, or now if RSS passes 460 MiB.
  If not drained by 18:15, restart during the low-traffic window.

An owner and an end time are what separate “hold” from “forgot about it”. Write that block into notes.md now, whichever option you go on to take.

Restart. Frees the head immediately at the cost of a WAL replay, during which nothing is scraped and no rule is evaluated. Correct when the instance is thrashing now and cannot wait for truncation. Time it, because the duration is the single number that decides whether restart is affordable next time:

Service impact possiblehost— Restarts Prometheus. Scrapes and rule evaluation stop for the whole WAL replay.

$ cd ~/obs-recovery-lab && date '+%H:%M:%S restart' && docker compose restart prometheus

cd "$HOME/obs-recovery-lab"

# 503 until the replay finishes. Poll until it is ready, and time it.
until curl -sf http://127.0.0.1:9090/-/ready >/dev/null 2>&1; do
  printf '.'
  sleep 2
done
date '+%H:%M:%S ready'

docker compose logs prometheus 2>&1 | grep -i wal | tail -5

Record the replay duration. Then look at what the restart did and did not achieve:

cd "$HOME/obs-recovery-lab"
./bin/q 'prometheus_tsdb_head_series'
./bin/q 'process_resident_memory_bytes{job="prometheus"}'

The replay rebuilt the head from the WAL, which still contains the poisoned series — the relabel rule applies at scrape time, not at replay time. What makes the restart effective is the combination: replayed series receive no new samples because the rule is live, so they go stale quickly rather than being refreshed forever. Restart without the relabel rule is the failure mode lesson 05 names first, and it costs one WAL replay per attempt.

Task 8: The lever you cannot undo

delete_series exists for series that are actively harmful to keep — a label value carrying a credential or personal data — not for series that are merely large. It is available only because this lab started Prometheus with --web.enable-admin-api, which is off by default precisely so that this call is a decision someone made.

Data-loss riskhost— Writes tombstones for the named series. Irreversible.

$ curl -sf -g -X POST 'http://127.0.0.1:9090/api/v1/admin/tsdb/delete_series?match[]=storefront_http_requests_total{session_id!=""}' -o /dev/null -w '%{http_code}\n'

cd "$HOME/obs-recovery-lab"

# Queries stop returning them straight away.
./bin/q 'count(storefront_http_requests_total)'

# Space comes back at compaction, not on the call. This asks for it now.
curl -sf -X POST http://127.0.0.1:9090/api/v1/admin/tsdb/clean_tombstones \
  -o /dev/null -w '%{http_code}\n'

A 204 from either endpoint is success. Note the ordering in your timeline: the query result changes immediately, the disk does not, and head_series behaves differently again — which is why “did it work?” needs three different checks rather than one.

Under the hood

Three different clocks run during this recovery, and confusing them is what makes the response feel like it is not working.

The append clock. metric_relabel_configs are applied per sample, per scrape, after parsing and before the appender. A reload changes behaviour on the very next scrape — seconds. This is why scrape_samples_post_metric_relabeling is the correct thing to check first: it moves immediately or your rule is not live.

The staleness and truncation clock. A series already in the head keeps its memory until it receives no samples for long enough to be considered stale and the head is truncated, which happens on the block boundary — hours. Nothing you type shortens this except a restart, and a restart replaces it with a replay.

The compaction clock. Tombstones from delete_series hide data from queries at once, but the bytes live in the block until compaction rewrites it. clean_tombstones asks for that work now rather than waiting.

Read against those three clocks, the standard sequence stops looking arbitrary. Relabel first, because it is the only lever with an immediate effect on inflow. Then decide whether the instance can survive the truncation clock; hold if it can, restart if it cannot. Reach for tombstones only when the content of the series is the problem, because that is the only clock that cannot be waited out.

Task 9: Phase 4, recover and prevent

Recovery is declared against evidence, not against a feeling that things look better.

cd "$HOME/obs-recovery-lab"
date '+%H:%M:%S recover'

./bin/q 'prometheus_tsdb_head_series'
./bin/q 'rate(prometheus_tsdb_head_series_created_total[15m])'
./bin/q 'increase(prometheus_rule_group_iterations_missed_total[15m])'
./bin/q 'up{job="storefront"}'
curl -sf http://127.0.0.1:9090/-/ready

Declare recovery when head series is back inside budget, churn is at baseline, rule groups are evaluating on schedule, and you can state the start and end of the gap in the graphs.

Then fix the source, which is the only change that removes the relabel rule’s reason to exist:

cd "$HOME/obs-recovery-lab"
./bin/explode.sh 0
sleep 60
./bin/q 'scrape_samples_scraped{job="storefront"}'
./bin/q 'scrape_samples_post_metric_relabeling{job="storefront"}'
./bin/q 'up{job="storefront"}'

With the source fixed, the two sample counts converge and the scrape is healthy again. The relabel rule can now be removed — and should be, with a comment in the commit naming the incident, so that the next person to read the config does not find an undocumented load-bearing line.

Finally, write the impact statement. Vague status updates erode trust in the platform you just saved, so be specific:

IMPACT — cardinality incident, storefront job
  Cause      : deploy at 15:28 added session_id to storefront_http_requests_total
  Detected   : 15:44 (HeadSeriesGrowthAnomaly)
  Mitigated  : 16:09 (labeldrop + sample_limit, reload confirmed via API)
  Recovered  : 16:31
  Metrics gap: 16:14-16:23 for ALL jobs (Prometheus restart, WAL replay 9m)
  Alerting   : rule evaluation degraded 15:52-16:23; N missed group iterations
  Logs/traces: unaffected
  Follow-up  : source fix shipped; sample_limit now permanent on this job;
               session_id added to the standing drop-rules snippet

Validation

The stack runs with mem_limit: 512m on Prometheus and both admin flags present in docker compose config.
notes.md contains a ceiling estimate derived from your own baseline numbers, and states which planning figure it used.
After explode.sh, prometheus_tsdb_head_series shows a step, and rate(prometheus_tsdb_head_series_created_total[5m]) is far above baseline.
You recorded whether the container was OOM-killed, with the OOMKilled and ExitCode values, not an impression.
Phase 2 named the family, the job and the label without changing anything, and the mtime evidence anchors it to a time.
/api/v1/status/config shows the metric_relabel_configs block — you did not trust the file on disk.
After mitigation, scrape_samples_post_metric_relabeling falls immediately while prometheus_tsdb_head_series does not.
notes.md contains a written HOLD decision with an owner and an re-evaluation time, whether or not you went on to restart.
If you restarted: the replay duration is recorded, and /-/ready returned 503 for that whole period.
If you ran delete_series: the selector named a metric and a label, both calls returned 204, and you noted that the query result changed before the disk did.
The timeline has real clock times for deploy, recognise, identify, mitigate and recover, and the impact statement names jobs and minutes.

Expected Outcome

obs-recovery-lab/
├── bin/{explode.sh,q}
├── compose.yaml
├── drop-rules.snippet.yaml
├── notes.md
├── prometheus/
│   ├── prometheus.yml
│   └── rules/cardinality.yml
└── textfile/storefront.prom

A Prometheus back at its baseline series count, a written timeline with your own timings for each phase, and an impact statement that an incident review can use without asking you a single follow-up question.

Troubleshooting

The alert never fires. HeadSeriesGrowthAnomaly compares against offset 10m, so the instance must have been running for at least ten minutes before the explosion, and for: 2m adds two more. Check the rule loaded at all with curl -s http://127.0.0.1:9090/api/v1/rules; an unmatched rule_files glob loads zero rules and reports no error anywhere.

The explosion is too small to matter. The generator produces sessions x 32 series. If head series barely moved, you asked for too few — go back to the ceiling estimate in Task 2 and aim for one and a half times it.

The explosion is so large the exporter times out. scrape_duration_seconds above the interval means the target cannot serialise the exposition in time, and you are now testing a slow exporter rather than a cardinality incident. Reduce the session count; the arithmetic is the lesson, not the size.

The reload returns 404. --web.enable-lifecycle is missing from the container’s command. docker compose config prints the effective command list, which is the fastest place to check.

delete_series returns 405. --web.enable-admin-api is missing. That is the default, and in production it is the correct default; a 405 here is Prometheus telling you that somebody has to decide to enable this, on purpose, with a network policy in front of it.

Prometheus will not come back after a restart. If replay is pathologically long, the emergency lever is moving the wal/ directory aside, which discards the samples it holds. In this lab the data is disposable, so it is a safe thing to practise; in production it is an explicit, recorded data-loss decision, and the far better answer is a second HA replica that stayed up.

Head series never fall, hours later. Check up{job="storefront"} — if the scrape recovered while the source was still emitting session_id and the relabel rule was reverted or never live, you are refilling the head as fast as it drains. Re-read the effective config from the API rather than the file.

Cleanup

Step 1. Keep the timeline. It is the deliverable, and the rehearsal is worth nothing if the timings are lost:

mkdir -p "$HOME/obs-lab-deliverables"
cp -a "$HOME/obs-recovery-lab/notes.md" \
      "$HOME/obs-recovery-lab/drop-rules.snippet.yaml" \
      "$HOME/obs-recovery-lab/prometheus" \
      "$HOME/obs-lab-deliverables/"

Step 2. Remove the project and its TSDB volume. The volume is the only thing here with any size to it:

Destructivehost— Removes the project's containers, network and the prometheus_data volume.

$ cd ~/obs-recovery-lab && docker compose down -v

docker compose -p obs-recovery ps -a
docker volume ls | grep obs-recovery || echo "no volumes remain"

rm -rf removes the whole lab tree, including anything you added

The block below deletes ~/obs-recovery-lab recursively and without confirmation — the generator, the snippet, the rules and the notes. Step 1 copies out the lab’s own deliverables and knows nothing about anything else you put in that directory.

Confirm what you are about to remove:

ls -la "$HOME/obs-recovery-lab"

rm -rf "$HOME/obs-recovery-lab"

What You Learned

The response is decided before the incident. The alert, the reviewed drop rule and a reload path that does not need a restart are the difference between twenty minutes and three hours, and none of them can be created while paging.
Two commands separate cardinality from everything else. RSS and head series moving together is cardinality; RSS alone is a query, a rule or the exemplar buffer, and restarting on that diagnosis buys nothing but a replay.
Identify before you mitigate, because mitigating destroys evidence. The family, the job, the label and the time — four facts, three read-only commands, and they are what the review will ask for.
A relabel drop stops inflow in one scrape and frees memory in hours. Both halves of that sentence matter: the first is why it goes first, the second is why it is not the end.
Hold is a decision with an owner and an end time. Written down it is a plan; unwritten it is the incident nobody closed.
A restart buys a fresh head at the price of a replay, and only works in combination with the rule — replay repopulates from the WAL, and it is the absence of new samples that makes the replayed series go stale.
delete_series is for series that are harmful to keep, not merely large. It is irreversible, it hides data before it frees disk, and the admin API is off by default for exactly that reason.
Recovery is declared against four numbers: level, churn, missed rule iterations and scrape health — plus a gap you can state in minutes.

Why this matters

A cardinality incident is the one class of outage that removes the instruments you would use to handle every other class. While the head is growing, alerts are not evaluating, dashboards are stale, and the estate is running without telemetry — usually during a deploy window, because a deploy is what caused it.

That is why this is a rehearsal rather than a reading. The four phases are not hard; they are hard at 15:44, with a peak coming, on an instance that also carries everyone else’s alerts. What survives that moment is what you have already done: a written alert, a reviewed snippet, a reload that is one curl, and the knowledge — from having timed it once, on your own hardware — of exactly how long a WAL replay costs you. The first live run should not be the first run.

Objectives

Prerequisites