Lab: Recover from a Cardinality Incident

Objective

By the end of this lab you will have built, configured, and validated a production element of the observability platform related to cardinality-recovery.

The lab exercises the canonical production pathway:

Configure the component.
Validate the configuration.
Run the component.
Confirm the platform reports the component correctly.

Requirements

Linux or macOS workstation with shell access.
Docker Engine 28.x and Docker Compose v2.
The RunBook Academy observability lab docker-compose skeleton (provided below).
Access to the canonical observability stack (Prometheus 2.55, Grafana 11, Loki 3, Tempo current).

Architecture

A single-host deployment using Docker Compose:

+-------+
| Host  |
+-------+
    |
    +-- docker-compose
            |
            +-- Prometheus
            +-- Alertmanager
            +-- Grafana
            +-- Loki
            +-- Tempo
            +-- Grafana Alloy
            +-- node_exporter
            +-- OpenTelemetry Collector

A minimal docker-compose.yaml is provided; expand it for the specific lab.

Scenario

You are tasked with cardinality-recovery. The lab environment provides one or more Docker Compose stacks. You exercise the pathway with disposable resources.

Tasks

Task 1: Setup

mkdir -p ~/rb-obs-lab && cd ~/rb-obs-lab

Task 2: Pull and start the stack

docker compose pull
docker compose up -d

Task 3: Validate the deployment

docker compose ps
curl -sf http://localhost:9090/-/ready

Task 4: Exercise the topic

For this lab topic (cardinality-recovery), execute the relevant commands:

# Topic-specific commands

Task 5: Verify validation outputs

Record the validation outputs in your lab notes.

Validation

Each lab has a specific validation. Common patterns:

# Prometheus: /-/ready and /-/healthy
curl -sf http://localhost:9090/-/ready && echo READY
curl -sf http://localhost:9090/-/healthy && echo HEALTHY

# Loki: ready endpoint
curl -sf http://localhost:3100/ready && echo READY

# Tempo: ready endpoint
curl -sf http://localhost:3200/ready && echo READY

# Grafana: /api/health
curl -sf http://localhost:3000/api/health && echo READY

Expected Outcome

The component is deployed, validated, and producing telemetry that the rest of the stack can consume.

Troubleshooting

Common issues:

Container exits. Check docker compose logs.
Port conflict. Verify nothing else uses 9090 / 3100 / 3200 / 3000.
Permission denied. The stack runs as 65534 (nobody); check filesystem permissions on volumes.

Cleanup

cd ~/rb-obs-lab
docker compose down -v

What You Learned

The deploy cycle for cardinality-recovery.
The validation commands that confirm operational status.
The cleanup pattern that returns the host to working state.

Objectives

Prerequisites