Lab: Cardinality Optimisation

Objective

By the end of this lab you will have taken one exporter from roughly twelve thousand series to a few hundred, three different ways, and you will be able to say which of the three actually returned resources and which merely moved the cost somewhere less visible. You will also have hit the wall that the scrape layer runs into when it removes the label that made the series unique — the result that explains why “we will relabel it later” is not a plan.

Architecture

Two containers and a directory. The interesting component is the directory: node_exporter’s textfile collector reads every *.prom file it finds and exposes the contents on the next scrape, which gives you an exporter whose cardinality you control exactly, with a shell script and no application to deploy.

  host                                  compose network: obs-cardinality
  ----                                  --------------------------------
  gen-checkout-metrics.sh
        writes                          node-exporter  --collector.disable-defaults
   ./textfile/checkout.prom  --mount-->    --collector.textfile
                                                 |
  127.0.0.1:9100 -------------------------------- + exposes /metrics
                                                 ^
  127.0.0.1:9090 --> prometheus --scrape---------+

--collector.disable-defaults is not a lab convenience. It is lesson 03’s recommendation applied: start from nothing, enable what you query, and every series on the wire is one you asked for. It also means the baseline is a small, fixed set of Go runtime and handler metrics, so every number that moves in this lab moved because of something you did.

Requirements

Linux or macOS with Docker Engine 28.x and the Compose v2 plugin.
bash, awk, python3 on the host. All three ship with any distribution this course targets.
About 1 GB of disk and 1.5 GB of RAM available to Docker. The Prometheus container in this lab has no memory limit; the recovery lab is the one that puts a ceiling on it deliberately.
Ports 9090 and 9100 free on 127.0.0.1.

Versions: Prometheus 2.55.x, node_exporter 1.8.x. Both the TSDB status API and the scrape-limit behaviour used here have been stable across those lines.

Scenario

A team wants to deploy an exporter for their checkout service on Monday. The change record says “adds metrics for the checkout API”. The platform Prometheus has 2.1 million series and 8 GB of RAM allocated to it, and the last time an exporter was deployed without an audit the platform team spent a weekend on it.

You have been asked to review the endpoint. Nobody has run curl against it yet, which is the whole of the problem and also the whole of the fix.

Tasks

Task 1: Build the exporter you are about to audit

WORKDIR="$HOME/obs-cardinality-lab"
mkdir -p "$WORKDIR"/{prometheus/rules,textfile,bin}
cd "$WORKDIR"
chmod 755 textfile

bin/gen-checkout-metrics.sh — the instrumentation, written the way a well-meaning developer writes it the first time. Every label looked reasonable in the pull request.

#!/usr/bin/env bash
# Emits the checkout service's metrics into the textfile collector directory.
# Usage: gen-checkout-metrics.sh [users] [buckets]
set -euo pipefail

USERS="${1:-200}"
BUCKETS="${2:-11}"
OUT_DIR="$(cd "$(dirname "$0")/.." && pwd)/textfile"
TMP="$OUT_DIR/checkout.prom.$$"

ROUTES="/checkout /cart /pay /confirm /refund /address /coupon /gift /ship /track"
METHODS="GET POST"
CODES="200 404 500"
LE_11="0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 2.5 5 10"
LE_6="0.025 0.05 0.1 0.2 0.5 1"

if [ "$BUCKETS" = "11" ]; then LADDER="$LE_11"; else LADDER="$LE_6"; fi

{
  echo "# HELP checkout_http_requests_total Checkout requests."
  echo "# TYPE checkout_http_requests_total counter"
  for r in $ROUTES; do
    for m in $METHODS; do
      for c in $CODES; do
        if [ "$USERS" = "0" ]; then
          echo "checkout_http_requests_total{route=\"$r\",method=\"$m\",status=\"$c\"} 1"
        else
          u=1
          while [ "$u" -le "$USERS" ]; do
            printf 'checkout_http_requests_total{route="%s",method="%s",status="%s",user_id="u%04d"} 1\n' \
              "$r" "$m" "$c" "$u"
            u=$((u + 1))
          done
        fi
      done
    done
  done

  echo "# HELP checkout_request_duration_seconds Checkout latency."
  echo "# TYPE checkout_request_duration_seconds histogram"
  for r in $ROUTES; do
    for m in $METHODS; do
      n=0
      for le in $LADDER; do
        n=$((n + 1))
        echo "checkout_request_duration_seconds_bucket{route=\"$r\",method=\"$m\",le=\"$le\"} $n"
      done
      echo "checkout_request_duration_seconds_bucket{route=\"$r\",method=\"$m\",le=\"+Inf\"} $((n + 1))"
      echo "checkout_request_duration_seconds_sum{route=\"$r\",method=\"$m\"} 1.5"
      echo "checkout_request_duration_seconds_count{route=\"$r\",method=\"$m\"} $((n + 1))"
    done
  done
} > "$TMP"

# Atomic rename. The collector reads whatever is there at scrape time, so a
# partially written file is a parse error on a live endpoint.
mv "$TMP" "$OUT_DIR/checkout.prom"
chmod 644 "$OUT_DIR/checkout.prom"

compose.yaml:

name: obs-cardinality

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=1d'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    ports:
      - '127.0.0.1:9090:9090'
    restart: unless-stopped

  node-exporter:
    image: quay.io/prometheus/node-exporter:v1.8.2
    command:
      - '--collector.disable-defaults'
      - '--collector.textfile'
      - '--collector.textfile.directory=/textfile_collector'
    volumes:
      - ./textfile:/textfile_collector:ro
    ports:
      - '127.0.0.1:9100:9100'
    restart: unless-stopped

volumes:
  prometheus_data:

prometheus/prometheus.yml — deliberately without the checkout job yet. The audit happens before the scrape config exists, which is the point.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

cd "$HOME/obs-cardinality-lab"
chmod +x bin/gen-checkout-metrics.sh
printf 'groups: []\n' > prometheus/rules/cardinality.yml
./bin/gen-checkout-metrics.sh 200 11
docker compose up -d
sleep 5
docker compose ps

Task 2: Predict, then audit

Do the arithmetic before you run anything. Lesson 03’s multiplication table applies directly: the series count of a metric is the product of the sizes of its label domains, and a classic histogram is one series per bucket plus _sum and _count.

Fill this in before running the next command.

Metric family	Label domains	Predicted series
`checkout_http_requests_total`	route 10, method 2, status 3, user_id 200
`checkout_request_duration_seconds`	route 10, method 2, buckets 11 + `+Inf` + `_sum` + `_count`
Total from the checkout service

Now audit the endpoint the way the deployment checklist should have:

cd "$HOME/obs-cardinality-lab"
# 1. Total exposed series: sample lines only, not HELP/TYPE.
curl -sf http://127.0.0.1:9100/metrics | grep -vc '^#'

# 2. Series per metric family, worst first.
curl -sf http://127.0.0.1:9100/metrics | grep -v '^#' | \
  sed 's/{.*//; s/ .*//' | sort | uniq -c | sort -rn | head -10

# 3. Distinct values of each suspect label.
for l in user_id route method status le; do
  n=$(curl -sf http://127.0.0.1:9100/metrics | grep -v '^#' | \
      grep -o "$l=\"[^\"]*\"" | sort -u | wc -l)
  echo "$l: $n distinct values"
done

# 4. Lint the exposition itself.
curl -sf http://127.0.0.1:9100/metrics | \
  docker compose exec -T prometheus promtool check metrics

Read-only / Safehost— Step 2: which family is responsible.

$ curl -sf http://127.0.0.1:9100/metrics | grep -v '^#' | sed 's/{.*//; s/ .*//' | sort | uniq -c | sort -rn | head -5

  12000 checkout_http_requests_total
  240 checkout_request_duration_seconds_bucket
   20 checkout_request_duration_seconds_count
   20 checkout_request_duration_seconds_sum
    6 go_gc_duration_seconds

Illustrative output

Your predictions should match: 10 x 2 x 3 x 200 = 12,000 for the counter, and 10 x 2 x (11 + 1 + 2) = 280 for the histogram. Two label domains you cannot enumerate — user_id here — turn a 60-series metric into a 12,000-series one, and the diff that did it was one word long.

promtool check metrics will complain about naming and typing but says nothing at all about cardinality. That is not a gap in the tool; it is the reason the uniq -c count in step 2 belongs in your checklist next to it.

Task 3: Ingest it, and measure your own per-series cost

Add the job and let it run for five minutes, which is long enough for the head to settle.

  - job_name: checkout
    scrape_interval: 15s
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          team: checkout

cd "$HOME/obs-cardinality-lab"
docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && echo reloaded
sleep 300

Now the three measurements that matter, all read-only:

cd "$HOME/obs-cardinality-lab"

# One instant-query helper, used for the rest of the lab.
cat > bin/q <<'SH'
#!/usr/bin/env bash
curl -sfG http://127.0.0.1:9090/api/v1/query --data-urlencode "query=$1" |
  python3 -c 'import json,sys; r=json.load(sys.stdin)["data"]["result"]; print(r[0]["value"][1] if r else "no data")'
SH
chmod +x bin/q

./bin/q 'prometheus_tsdb_head_series'
./bin/q 'count by (job) ({job="checkout"})'
./bin/q 'process_resident_memory_bytes{job="prometheus"}'
./bin/q 'rate(prometheus_tsdb_head_series_created_total[5m])'

Derive the constant lesson 01 tells you never to take on faith. The script queries both numbers itself rather than asking you to copy them, so the division is reproducible:

python3 - <<'PY'
import json, urllib.parse, urllib.request

def q(expr):
    url = "http://127.0.0.1:9090/api/v1/query?" + urllib.parse.urlencode({"query": expr})
    result = json.load(urllib.request.urlopen(url))["data"]["result"]
    return float(result[0]["value"][1]) if result else 0.0

rss = q('process_resident_memory_bytes{job="prometheus"}')
series = q('prometheus_tsdb_head_series')
per_series = rss / series

print("head series:  %d" % series)
print("resident:     %.1f MiB" % (rss / 1024 ** 2))
print("per series:   %.2f KiB" % (per_series / 1024))
print("an 8 GiB head at that rate would hold %d series" % (8 * 1024 ** 3 / per_series))
PY

The per-label breakdown from the TSDB status API is the evidence you attach to the review ticket:

curl -sf http://127.0.0.1:9090/api/v1/status/tsdb | python3 -c '
import json, sys
d = json.load(sys.stdin)["data"]
print("--- series by metric name")
for e in d["seriesCountByMetricName"][:5]:
    print("%8s  %s" % (e["value"], e["name"]))
print("--- distinct values by label name")
for e in d["labelValueCountByLabelName"][:8]:
    print("%8s  %s" % (e["value"], e["name"]))'

user_id sits at 200 in this lab and would sit at daily-active-users in production. Any label whose distinct-value count approaches five figures belongs on the suspect list, and this API is how you find it without knowing which label to ask about first.

Task 4: Cut it at the source

The source fix is the only one that returns resources everywhere: the target stops allocating and formatting the series, the network stops carrying them, and Prometheus stops parsing them before dropping them.

Two changes, both of which the generator already supports: remove the unbounded label, and choose the bucket ladder deliberately around the service’s SLO rather than accepting eleven defaults.

cd "$HOME/obs-cardinality-lab"
./bin/gen-checkout-metrics.sh 0 6

curl -sf http://127.0.0.1:9100/metrics | grep -vc '^#'
curl -sf http://127.0.0.1:9100/metrics | grep -v '^#' | \
  sed 's/{.*//; s/ .*//' | sort | uniq -c | sort -rn | head -5

The counter is now 10 x 2 x 3 = 60 series and the histogram is 10 x 2 x (6 + 1 + 2) = 180. Two orders of magnitude, from removing one label and five buckets.

Confirm that the collector actually re-read the file rather than serving a cached copy, and that nothing in the new file failed to parse:

curl -sf http://127.0.0.1:9100/metrics | grep -E '^node_textfile_(mtime_seconds|scrape_error)'

node_textfile_mtime_seconds should carry the timestamp of the file you just wrote. node_textfile_scrape_error must be 0; a 1 there means the collector could not read or parse a file and is silently serving you less than you think.

Write down what you just lost, before you forget you had it

user_id is gone, and with it every answer of the form “what did this customer experience?”. The eleven-bucket ladder is gone, and with it any quantile estimate above one second and below twenty-five milliseconds.

Both losses are real and both are for the whole retention window — you cannot backfill a dimension you chose not to store. The honest position is not that the losses do not matter; it is that metrics are the wrong signal for per-entity questions, and the replacement is a trace or a structured log with the trace ID as the join key. If nobody funds that path, the question is genuinely unanswerable, and that belongs in the change record rather than in a surprise six weeks later.

Add both to notes.md under “questions this optimisation cost us”.

Task 5: Cut it at the scrape layer, and meet the wall

Put the bad instrumentation back — you are now playing the platform team who cannot change the source before Monday.

cd "$HOME/obs-cardinality-lab"
./bin/gen-checkout-metrics.sh 200 11

The containment tool is metric_relabel_configs, which runs after the target’s labels are set and before ingestion. Predict what happens, and write the prediction down before you reload.

  - job_name: checkout
    scrape_interval: 15s
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          team: checkout
    metric_relabel_configs:
      # labeldrop matches label NAMES and takes no source_labels.
      - action: labeldrop
        regex: '(user_id|session_id|request_id|trace_id)'

cd "$HOME/obs-cardinality-lab"
docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && sleep 60

# What the target served, versus what survived relabeling.
./bin/q 'scrape_samples_scraped{job="checkout"}'
./bin/q 'scrape_samples_post_metric_relabeling{job="checkout"}'
# Did the scrape stay healthy?
./bin/q 'up{job="checkout"}'
./bin/q 'prometheus_target_scrapes_sample_duplicate_timestamp_total'

Also look at the target’s health directly, because the health string names the error:

curl -sf 'http://127.0.0.1:9090/api/v1/targets?state=active' | python3 -c '
import json,sys
for t in json.load(sys.stdin)["data"]["activeTargets"]:
    if t["labels"].get("job") == "checkout":
        print(t["health"], "|", t.get("lastError", ""))'

Dropping the label that made the series unique

user_id was the only thing distinguishing 200 of those samples from each other. Remove it and all 200 collapse onto one identical label set, arriving in the same scrape with the same timestamp and different values. Prometheus has exactly one thing it can do with that, and it is not to pick a winner.

Record what up, lastError and prometheus_target_scrapes_sample_duplicate_timestamp_total actually said on your instance. Then reason about the general case: a labeldrop is safe when the remaining labels still identify each series uniquely — dropping pod_uid when pod is still there, dropping trace_id from a metric that also carries route — and it is not safe when the dropped label was carrying the identity.

This is why lesson 03 puts relabel rules in the “containment” category rather than the “fix” category. The instrumentation was emitting one series per user; no scrape-time rule can turn that into a correct aggregate, because aggregation is an operation on values and relabeling only edits labels.

The honest containment for this metric family is therefore to drop the family, not the label, until the source is fixed:

    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'checkout_http_requests_total'
        action: drop

cd "$HOME/obs-cardinality-lab"
docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && sleep 60
./bin/q 'scrape_samples_scraped{job="checkout"}'
./bin/q 'scrape_samples_post_metric_relabeling{job="checkout"}'

The gap between those two numbers is what lesson 03 calls the relabel waste: the series the target built, serialised and shipped, and Prometheus parsed, before throwing them away. It is the measurement that stops a temporary relabel rule from becoming permanent, because it puts a number on what the shortcut still costs every fifteen seconds.

Task 6: The admission layer, which fails loudly on purpose

Neither of the previous layers protects the platform from the next bad exporter. sample_limit does, and its failure shape is the one you want: noisy at the source, silent at the reservoir.

Remove the metric_relabel_configs block and add limits instead:

  - job_name: checkout
    scrape_interval: 15s
    sample_limit: 5000
    label_limit: 30
    label_name_length_limit: 120
    label_value_length_limit: 512
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          team: checkout

cd "$HOME/obs-cardinality-lab"
docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && sleep 60

./bin/q 'up{job="checkout"}'
./bin/q 'scrape_samples_scraped{job="checkout"}'
./bin/q 'increase(prometheus_target_scrapes_exceeded_sample_limit_total[5m])'

up is 0. Not “partially ingested”, not “truncated at 5,000” — the whole scrape is discarded and the counter increments. That is the correct allocation of pain: the job that misbehaves loses its own monitoring, and the shared head block is untouched.

Now fix the source and watch the same job come back without any change to the limit:

cd "$HOME/obs-cardinality-lab"
./bin/gen-checkout-metrics.sh 0 6
sleep 30
./bin/q 'up{job="checkout"}'
./bin/q 'scrape_samples_scraped{job="checkout"}'

Under the hood

The order of operations inside one scrape decides which lever can do what.

Prometheus fetches the exposition, parses it, applies metric_relabel_configs to each sample’s label set, and then appends the survivors to the head. Two consequences follow directly.

First, metric_relabel_configs cannot save the target or the network any work, because both have already happened — which is the whole content of scrape_samples_scraped minus scrape_samples_post_metric_relabeling. Only relabel_configs (target relabeling, lesson 05) can prevent a request being made at all, because it runs at discovery time and drops whole targets.

Second, the label set a sample carries after relabeling is its identity. Two samples that end up identical are two appends to the same series at the same timestamp, which the appender rejects rather than resolving. Relabeling edits identity; it cannot aggregate values. Every “just drop the label” proposal has to answer the question of whether the remaining labels still distinguish the series, and the answer is usually visible in one line of the exposition.

sample_limit is checked during the append phase against the post-relabeling count, and breaching it aborts the whole scrape rather than truncating it — because a truncated scrape is a silent, arbitrary subset, and no alert built on a subset means anything.

Task 7: Write the budget, and the alert that fires before the wall

A budget is a number, a measurement and a tripwire. You now have a measured number to put in it.

cardinality-budgets.yaml — this file lives in version control next to prometheus.yml, not in a wiki:

# Owned by the observability team. Reviewed quarterly.
# Unit: active series contributed to the shared Prometheus head.
platform:
  head_series_soft_cap: 2000000      # derived from the per-series cost
  warn_at_fraction: 0.8              # measured in Task 3 on THIS instance
teams:
  - team: checkout
    jobs: [checkout]
    max_active_series: 5000
    contacts: ['#checkout-oncall']

prometheus/rules/cardinality.yml:

groups:
  - name: cardinality-budget
    interval: 30s
    rules:
      # Recording the attribution makes it a cheap lookup rather than an
      # expensive ad-hoc query at the worst possible moment.
      - record: job:active_series:count
        expr: count by (job) ({__name__=~".+"})

      - alert: JobCardinalityOverBudget
        expr: job:active_series:count{job="checkout"} > 5000
        for: 2m
        labels: {severity: warning, team: checkout}
        annotations:
          summary: 'checkout uses {{ $value }} series against a 5000 budget'

      - alert: ScrapesHittingSampleLimit
        expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
        for: 1m
        labels: {severity: warning, team: observability}

cd "$HOME/obs-cardinality-lab"
docker compose exec prometheus promtool check rules /etc/prometheus/rules/cardinality.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && sleep 60

# The recording rule should now answer the attribution question instantly.
./bin/q 'job:active_series:count'

# Trip the budget on purpose and watch the alert arrive.
./bin/gen-checkout-metrics.sh 200 11
sleep 180
curl -sf http://127.0.0.1:9090/api/v1/alerts | python3 -c '
import json,sys
for a in json.load(sys.stdin)["data"]["alerts"]:
    print(a["labels"]["alertname"], a["state"], a["labels"].get("team",""))'

Note which alert fires and which does not: with sample_limit: 5000 still in place, the scrape fails before the series are ingested, so ScrapesHittingSampleLimit fires and JobCardinalityOverBudget does not. The admission control did its job and the budget alert never needed to. That ordering — the limit fires first, the budget alert is the backstop, the OOM never happens — is what defence in depth looks like when it is working.

Restore the good instrumentation before you finish:

cd "$HOME/obs-cardinality-lab"
./bin/gen-checkout-metrics.sh 0 6
sleep 30
./bin/q 'up{job="checkout"}'

Validation

The predicted counts in Task 2 match the audit: 12,000 for the counter, 280 for the histogram family.
promtool check metrics passes on an exposition with 12,000 series, proving it says nothing about cardinality.
node_textfile_scrape_error is 0 and node_textfile_mtime_seconds tracks each regeneration.
The per-series figure you derived in Task 3 is written in notes.md together with the head-series count it was derived at.
/api/v1/status/tsdb names user_id in labelValueCountByLabelName with 200 distinct values while the bad instrumentation is in place.
After the source fix, the endpoint serves 240 checkout series instead of 12,280 — a reduction you can quote as a ratio.
With labeldrop on user_id, you have recorded the target’s health, lastError and the duplicate-timestamp counter, and can explain why relabeling cannot aggregate.
With action: drop on the metric name, scrape_samples_scraped stays high while scrape_samples_post_metric_relabeling falls — and you can state what that gap costs per scrape.
With sample_limit: 5000 against a 12,280-series exposition, up is 0 and prometheus_target_scrapes_exceeded_sample_limit_total increases.
job:active_series:count returns a value, and ScrapesHittingSampleLimit reaches firing.

Expected Outcome

obs-cardinality-lab/
├── bin/gen-checkout-metrics.sh
├── cardinality-budgets.yaml
├── compose.yaml
├── notes.md
├── prometheus/
│   ├── prometheus.yml
│   └── rules/cardinality.yml
└── textfile/checkout.prom

An exporter serving a few hundred series instead of twelve thousand, a budget file whose number came from a measurement you took, an alert you have watched fire, and a written list of the questions the optimisation cost you.

Troubleshooting

The textfile metrics never appear. Ownership and permissions are the usual cause: the directory must be 755 and the .prom file 644, because the exporter runs as a non-root user. Check node_textfile_scrape_error — a 1 means the collector found the file and could not use it, which is a different problem from not finding it at all.

node_textfile_scrape_error is 1 after regenerating. The exposition is malformed. Common causes are a missing trailing newline, a duplicated # TYPE line for the same family, or a mv that raced a scrape. The generator writes to a temporary file and renames, which is what prevents the third.

up{job="checkout"} is 0 and no limit is configured. Look at lastError on /api/v1/targets. A duplicate-sample error, a parse error and a connection refusal all present as up 0 and mean entirely different things.

A relabel rule appears to do nothing. Compare scrape_samples_scraped with scrape_samples_post_metric_relabeling; equal numbers mean no sample was dropped. Then confirm Prometheus loaded the config you edited: curl -s http://127.0.0.1:9090/api/v1/status/config. Reading the file on disk proves nothing about what is running.

Series do not disappear after a successful drop rule. They will not, for a while. A relabel rule changes what is appended, not what is stored; existing series stay in the head until they go stale and the head is truncated, which is hours away. scrape_samples_post_metric_relabeling falling immediately while prometheus_tsdb_head_series falls later is the expected shape, and this lab is too short to show the second half of it.

The generator takes a long time. Twelve thousand lines from nested shell loops is not fast. If it is unbearable on your host, reduce the user count — the arithmetic is the lesson, not the absolute number.

Cleanup

Step 1. Keep the artefacts. The budget file and the notes are the deliverables:

mkdir -p "$HOME/obs-lab-deliverables"
cp -a "$HOME/obs-cardinality-lab/cardinality-budgets.yaml" \
      "$HOME/obs-cardinality-lab/notes.md" \
      "$HOME/obs-cardinality-lab/prometheus" \
      "$HOME/obs-lab-deliverables/"

Step 2. Remove the project, including the TSDB volume:

Destructivehost— Removes the project's containers, network and the prometheus_data volume.

$ cd ~/obs-cardinality-lab && docker compose down -v

docker compose -p obs-cardinality ps -a
docker volume ls | grep obs-cardinality || echo "no volumes remain"

rm -rf removes the whole lab tree, including anything you added

The block below deletes ~/obs-cardinality-lab recursively and without confirmation — the generator, the budget file, the notes, and anything else you put there. Step 1 copies out the lab’s own deliverables and knows nothing about yours.

Look before you delete:

ls -la "$HOME/obs-cardinality-lab"

rm -rf "$HOME/obs-cardinality-lab"

What You Learned

Cardinality is a product, and the product is invisible in review. One unbounded label turned a 60-series metric into 12,000, and the diff that did it was a single word. The audit that catches it is four commands and runs before the scrape config exists.
promtool check metrics and a cardinality audit are different checks. The first is about naming and types; the second is sed, sort, uniq -c, and it is the one that would have caught this.
Never quote a per-series cost you did not measure. You derived one, and you learned why a lab instance’s figure is not transferable — which is exactly why the production figure is not transferable either.
A classic histogram is thirteen series per label set, not one. Choosing six buckets around the SLO instead of eleven defaults is a real cut, and it usually produces a better latency picture than the defaults did.
Only the source fix returns resources. Relabeling still pays for allocation, serialisation, transfer and parsing every scrape; scrape_samples_scraped minus scrape_samples_post_metric_relabeling is that bill, and it is payable every fifteen seconds.
Relabeling edits identity and cannot aggregate. Dropping the label that distinguished the series produces duplicate samples, not a smaller metric. When the source emits one series per user, the containment is to drop the family until the source is fixed.
sample_limit fails the whole scrape on purpose. A truncated scrape would be an arbitrary subset, and no alert built on an arbitrary subset means anything.
Every cut costs an answer. Writing down which answers, at the moment you give them up, is the difference between a deliberate trade-off and a surprise during an incident six weeks later.

Why this matters

The uncomfortable question lesson 03 asks of any /metrics endpoint is which of these series have appeared in a query, dashboard or alert in the last ninety days, and the answer is routinely under twenty percent. The other eighty percent is not neutral. It is the RAM, the index, the WAL, the query scan time and the remote-write invoice that together decide how much headroom the platform has when a real incident arrives — and the platform’s headroom is exactly what determines whether that incident is visible while it happens.

This lab is the cheap version of that conversation. Four commands in staging, an arithmetic prediction, a measured constant and a written trade-off. The expensive version is the same conversation held at 16:04 with the head block growing vertically, which is the next lab.

Objectives

Prerequisites