Lab: Blackbox Probes

Objective

By the end of this lab you will have a running blackbox probe suite that you built rather than inherited, and — more usefully — you will have made it lie to you four times on purpose. A probe that reports green through a real outage is the single most expensive failure this component has, and every one of the four ways it happens here is a configuration you will meet in somebody’s repository.

Architecture

One Compose project, four containers on one user-defined bridge network. The targets are ordinary nginx containers with no published ports: they exist to be probed from inside the network, which is exactly the position a probe occupies in production.

  host                                     compose network: obs-blackbox
  ----                                     ----------------------------
  127.0.0.1:9090 --> prometheus  ---scrape /probe?target=X&module=Y--->  blackbox
                         |                                                  |
                         |                                          probes from here
  127.0.0.1:9115 --------+------------------------------------------->     |
                                                                            v
                                                       shop:80        (plain HTTP nginx)
                                                       secure-shop:443 (TLS nginx, self-signed)

Prometheus never talks to shop. It talks to blackbox, and blackbox talks to shop. That indirection is the whole mechanism, and it is the reason the relabel chain in Task 3 matters more than anything else in the lab.

Requirements

A Linux or macOS host with Docker Engine 28.x and the Compose v2 plugin. docker compose version must print v2.x.
openssl 1.1.1 or newer, for -addext. openssl version on the host.
Roughly 700 MB of disk for images and about 400 MB of RAM for the stack.
Outbound HTTPS to a registry for the first docker compose pull. Nothing after that reaches the internet: every probe target is inside the project.
No root on the host beyond whatever your Docker installation already requires. Nothing here touches the host’s network configuration, firewall or SSH daemon, so there is no lockout risk and no out-of-band access requirement.

Versions this lab is written against: Prometheus 2.55.x, blackbox_exporter 0.26.x, nginx from the stable-alpine tag. The exporter’s metric names have been stable across the 0.2x line; the Prometheus relabel semantics have been stable far longer.

Scenario

You have inherited a probe suite. It has been green for eleven weeks. Yesterday a customer could not reach the shop for forty minutes, and the incident review has one question on it: why did the synthetic monitoring not fire?

Nobody has yet looked at the instance label on those green series.

Tasks

Task 1: Build the project

Confirm the ports are free before anything else — a port conflict here surfaces as a container that exits immediately, which is a confusing way to start.

WORKDIR="$HOME/obs-blackbox-lab"
mkdir -p "$WORKDIR"/{blackbox,prometheus/rules,shop/html,shop/conf.d,secure-shop/conf.d,secure-shop/certs}
cd "$WORKDIR"

# Nothing else may already answer on 9090 or 9115. Portable check.
curl -s --max-time 2 http://127.0.0.1:9090/ >/dev/null && echo "IN USE: 9090"
curl -s --max-time 2 http://127.0.0.1:9115/ >/dev/null && echo "IN USE: 9115"
echo "port check complete"

docker compose version

The two probe targets. The shop serves a healthy page and a health endpoint; the health endpoint is a separate location so you can break it in Task 6 without touching the page.

shop/html/index.html:

<!doctype html>
<title>Shop</title>
<h1>Shop is open</h1>
<p>Checkout available.</p>

shop/conf.d/default.conf:

server {
    listen 80;
    server_name shop;

    location /healthz {
        default_type application/json;
        return 200 '{"status":"ok"}';
    }

    location / {
        root /usr/share/nginx/html;
        index index.html;
    }
}

Now the TLS target. The certificate is deliberately given twenty days of validity so that the thirty-day expiry alert in Task 8 fires during the lab rather than being something you have to take on trust.

cd "$HOME/obs-blackbox-lab"

openssl req -x509 -newkey rsa:2048 -nodes -days 20 \
  -keyout secure-shop/certs/secure-shop.key \
  -out    secure-shop/certs/secure-shop.crt \
  -subj   "/CN=secure-shop" \
  -addext "subjectAltName=DNS:secure-shop"

# The exporter runs as uid 65534 and must be able to read the certificate
# it validates against. The key stays private to the nginx container.
cp secure-shop/certs/secure-shop.crt blackbox/lab-ca.pem
chmod 644 secure-shop/certs/secure-shop.crt blackbox/lab-ca.pem
chmod 600 secure-shop/certs/secure-shop.key

openssl x509 -in blackbox/lab-ca.pem -noout -dates -subject -ext subjectAltName

secure-shop/conf.d/default.conf:

server {
    listen 443 ssl;
    server_name secure-shop;

    ssl_certificate     /etc/nginx/certs/secure-shop.crt;
    ssl_certificate_key /etc/nginx/certs/secure-shop.key;

    location / {
        default_type text/plain;
        return 200 'secure shop ok';
    }
}

Task 2: The exporter config, and a probe run by hand

Name modules after the question, not the protocol — the discipline lesson 01 asks for. Four questions, four modules.

blackbox/blackbox.yml:

modules:

  # Q: does the shop's health endpoint return exactly 200 over plain HTTP?
  http_2xx_shop:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes: [200]
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      follow_redirects: false
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true

  # Q: does it return 200 AND not be serving the maintenance page?
  # Same request, stricter acceptance. This is the Task 5 module.
  http_2xx_shop_strict:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes: [200]
      preferred_ip_protocol: ip4
      fail_if_body_matches_regexp:
        - 'maintenance'
      fail_if_body_not_matches_regexp:
        - '"status":"ok"'

  # Q: is the listener accepting TCP connections? Nothing more than that.
  tcp_connect_shop:
    prober: tcp
    timeout: 3s
    tcp:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true

  # Q: does the TLS chain validate against the trust store we gave it?
  http_2xx_tls:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes: [200]
      preferred_ip_protocol: ip4
      tls_config:
        insecure_skip_verify: false

compose.yaml:

name: obs-blackbox

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=1d'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    ports:
      - '127.0.0.1:9090:9090'
    restart: unless-stopped

  blackbox:
    image: prom/blackbox-exporter:v0.26.0
    command:
      - '--config.file=/etc/blackbox/blackbox.yml'
    volumes:
      - ./blackbox:/etc/blackbox:ro
    ports:
      - '127.0.0.1:9115:9115'
    restart: unless-stopped

  shop:
    image: nginx:stable-alpine
    volumes:
      - ./shop/html:/usr/share/nginx/html:ro
      - ./shop/conf.d:/etc/nginx/conf.d:ro
    restart: unless-stopped

  secure-shop:
    image: nginx:stable-alpine
    volumes:
      - ./secure-shop/conf.d:/etc/nginx/conf.d:ro
      - ./secure-shop/certs:/etc/nginx/certs:ro
    restart: unless-stopped

volumes:
  prometheus_data:

Prometheus needs a config file to exist before it will start, so write a minimal one now and grow it in Task 3.

prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

cd "$HOME/obs-blackbox-lab"
printf 'groups: []\n' > prometheus/rules/blackbox.yml
docker compose config --quiet && echo compose-ok
docker compose up -d
docker compose ps

Now the part most people skip: probe by hand, before Prometheus is involved. The exporter has no scheduler — an HTTP request is the probe — so curl reproduces exactly what a scrape does, with none of the relabel machinery in between. If a probe is wrong here, no amount of Prometheus configuration will save it.

Read-only / Safehost— The probe, run by hand. This is what a scrape does.

$ curl -sfG http://127.0.0.1:9115/probe --data-urlencode 'module=http_2xx_shop' --data-urlencode 'target=http://shop/healthz' | grep -E '^probe_'

probe_dns_lookup_time_seconds 0.0004
probe_duration_seconds 0.0031
probe_failed_due_to_regex 0
probe_http_redirects 0
probe_http_status_code 200
probe_http_version 1.1
probe_ip_protocol 4
probe_success 1

Illustrative output

Read every line of that, because it is the whole vocabulary. probe_success is the headline. probe_http_status_code says what was actually returned, which matters when success was defined loosely. probe_failed_due_to_regex is the only place a body-match failure is visible. probe_dns_lookup_time_seconds being non-zero proves the name was resolved by the exporter, inside the network — not by you, outside it.

Run the same probe against the TCP module and against the TLS target:

for probe in \
  'module=tcp_connect_shop&target=shop:80' \
  'module=http_2xx_tls&target=https://secure-shop/' ; do
  echo "--- $probe"
  curl -sf "http://127.0.0.1:9115/probe?$probe" | grep -E '^probe_success|^probe_ssl'
done

The TCP probe returns probe_success 1 and no HTTP metrics — the protocol is too thin to expose them. The TLS probe returns probe_success 0, because nothing in the exporter’s trust store signed that certificate. Leave it failing; Task 7 is about the two ways to make it green and why only one of them is honest.

When a probe fails and you cannot see why

Append &debug=true to the /probe URL. The exporter returns the full probe log and the module configuration it used, in place of the metrics. It is the fastest way to tell a DNS failure from a connection refusal from a certificate rejection, and it needs no access to the container’s stderr.

curl -s 'http://127.0.0.1:9115/probe?module=http_2xx_tls&target=https://secure-shop/&debug=true' | head -40

Task 3: Wire Prometheus — and reproduce the eleven green weeks

Replace the scrape_configs block in prometheus/prometheus.yml with the following. Read the relabel rules before you apply them and predict the instance label you will see.

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: blackbox_http
    metrics_path: /probe
    params:
      module: [http_2xx_shop]
    scrape_interval: 15s
    static_configs:
      - targets: ['http://shop/healthz']
        labels:
          service: shop
    relabel_configs:
      # INHERITED, AND WRONG. Do not fix it yet.
      - target_label: __address__
        replacement: blackbox:9115
      - source_labels: [__address__]
        target_label: __param_target

cd "$HOME/obs-blackbox-lab"
docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && echo reloaded

promtool passes. The config is valid — validation checks shape, not meaning. Now stop the shop, a complete and unambiguous outage, and ask the probe about it.

Service impact possiblehost— Stops the probe target. Only this Compose project is affected.

$ docker compose stop shop

sleep 45
curl -sf 'http://127.0.0.1:9090/api/v1/query?query=probe_success' \
  | python3 -m json.tool

Look at the instance label, not the colour

The instance label reads blackbox:9115. That single fact is the whole finding, and it is true regardless of what probe_success says.

Rules run in order, each seeing the previous rule’s output. The first rule overwrote __address__ with the exporter’s own address; the second rule then copied that value into __param_target. For eleven weeks this job has been asking the exporter about the exporter. The shop has never appeared in it.

Record both values in notes.md before you fix anything, because either outcome is a lie of a different kind:

probe_success is 1. The exporter probed itself and its own landing page answered. This is failure mode one from lesson 01: a green panel through a real outage, with instance as the only visible difference from a correct suite. Nothing errors, promtool is satisfied, up is 1.
probe_success is 0. The exporter rejected the target it was handed. The panel is red, an alert may even have fired — and it still says nothing about the shop, so the on-call would have spent the outage debugging the wrong component.

A probe series whose instance is the prober carries no information about any target, in either state. That is the sentence the incident review needs.

Task 4: Fix the ordering, watch it go red

The correct order is the one lesson 05 states as a general rule: filter, redirect, identify, clean. Here that means capture the real target first, build instance from it, and only then point the scrape at the exporter.

    relabel_configs:
      # 1. Remember the real target before anything overwrites it.
      - source_labels: [__address__]
        target_label: __param_target
      # 2. Identity comes from what is being probed, not from the prober.
      - source_labels: [__param_target]
        target_label: instance
      # 3. Only now aim the scrape itself at the exporter.
      - target_label: __address__
        replacement: blackbox:9115
      # 4. Stamp the module so panels and alerts can group by it.
      - target_label: module
        replacement: http_2xx_shop

cd "$HOME/obs-blackbox-lab"
docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && echo reloaded
sleep 45

curl -sf 'http://127.0.0.1:9090/api/v1/query?query=probe_success' \
  | python3 -m json.tool

probe_success is now 0, and instance is http://shop/healthz. The series identity changed with the relabel rule, so this is a new series: the eleven weeks of green history belongs to the old label set and does not become retroactively false. That is worth knowing before you write the review — the graph will show a gap and a new line, not a line that changes colour.

Bring the shop back and confirm recovery:

cd "$HOME/obs-blackbox-lab"
docker compose start shop
sleep 45
curl -sf 'http://127.0.0.1:9090/api/v1/query?query=probe_success' | grep -o '"value":\[[^]]*\]'

Task 5: The 200 that is not working

The shop now returns 200 on /healthz. Put it into maintenance the way a real deployment does — the page still serves, the status is still 200, the content says the service is unavailable.

cd "$HOME/obs-blackbox-lab"
cat > shop/conf.d/default.conf <<'CONF'
server {
    listen 80;
    server_name shop;

    location /healthz {
        default_type text/html;
        return 200 '<h1>scheduled maintenance</h1>';
    }

    location / {
        root /usr/share/nginx/html;
        index index.html;
    }
}
CONF

docker compose exec shop nginx -s reload
curl -sf 'http://127.0.0.1:9115/probe?module=http_2xx_shop&target=http://shop/healthz' \
  | grep -E '^probe_success|^probe_http_status_code|^probe_failed_due_to_regex'

probe_success 1. probe_http_status_code 200. The service is down for every customer and the probe is content, because the probe was only ever asked about the status code.

Now the strict module, which was asked a better question:

curl -sf 'http://127.0.0.1:9115/probe?module=http_2xx_shop_strict&target=http://shop/healthz' \
  | grep -E '^probe_success|^probe_http_status_code|^probe_failed_due_to_regex'

probe_success 0, probe_http_status_code 200, probe_failed_due_to_regex 1. The combination is the diagnosis: the transport worked, the status was right, and the content was wrong. probe_failed_due_to_regex is the only metric that distinguishes this from a healthy probe, which is why it belongs on the panel next to probe_success rather than three panels away.

Add a second scrape job for the strict module, so both questions are asked of the same target continuously:

  - job_name: blackbox_http_strict
    metrics_path: /probe
    params:
      module: [http_2xx_shop_strict]
    scrape_interval: 15s
    static_configs:
      - targets: ['http://shop/healthz']
        labels:
          service: shop
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115
      - target_label: module
        replacement: http_2xx_shop_strict

Reload, wait one interval, and compare the two series:

curl -sf -X POST http://127.0.0.1:9090/-/reload && sleep 20
curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=probe_success' | python3 -m json.tool | grep -E 'module|"1"|"0"'

Two series, same instance, different module, different answers. Restore the healthy config before continuing:

cd "$HOME/obs-blackbox-lab"
cat > shop/conf.d/default.conf <<'CONF'
server {
    listen 80;
    server_name shop;

    location /healthz {
        default_type application/json;
        return 200 '{"status":"ok"}';
    }

    location / {
        root /usr/share/nginx/html;
        index index.html;
    }
}
CONF
docker compose exec shop nginx -s reload

Task 6: TCP green, HTTP red — naming the boundary

A tcp_connect probe is a connect() and a close. It sends no bytes and reads none. Break the application while leaving the listener perfectly healthy, and the pair of probes tells you which layer failed.

cd "$HOME/obs-blackbox-lab"
cat > shop/conf.d/default.conf <<'CONF'
server {
    listen 80;
    server_name shop;
    location / { return 503 'upstream unavailable'; }
}
CONF
docker compose exec shop nginx -s reload

echo "--- tcp_connect (transport)"
curl -sf 'http://127.0.0.1:9115/probe?module=tcp_connect_shop&target=shop:80' | grep '^probe_success'
echo "--- http_2xx (application)"
curl -sf 'http://127.0.0.1:9115/probe?module=http_2xx_shop&target=http://shop/healthz' \
  | grep -E '^probe_success|^probe_http_status_code'

TCP is green: the three-way handshake completed, because nginx is running and bound. HTTP is red with probe_http_status_code 503. Together those two facts say “the host is up, the port is open, the application is refusing” — which rules out the network, the firewall and DNS in one comparison, before anyone opens a packet capture.

The inverse combination is just as informative. Stop the container entirely and both go red, which says the failure is below the application. Record both combinations in notes.md as a two-by-two table; it is the fastest triage artefact this component produces.

cd "$HOME/obs-blackbox-lab"
cat > shop/conf.d/default.conf <<'CONF'
server {
    listen 80;
    server_name shop;
    location /healthz {
        default_type application/json;
        return 200 '{"status":"ok"}';
    }
    location / { root /usr/share/nginx/html; index index.html; }
}
CONF
docker compose exec shop nginx -s reload

Task 7: Two ways to make a TLS probe green

The http_2xx_tls probe has been failing since Task 2. Confirm why:

curl -s 'http://127.0.0.1:9115/probe?module=http_2xx_tls&target=https://secure-shop/&debug=true' \
  | grep -iE 'x509|certificate|error' | head

The chain terminates at a certificate no trust store knows. There are two edits that turn the probe green, and the difference between them is the whole security argument of this component.

The wrong one. Add insecure_skip_verify: true to the module, SIGHUP the exporter, probe again:

cd "$HOME/obs-blackbox-lab"
sed -i 's/insecure_skip_verify: false/insecure_skip_verify: true/' blackbox/blackbox.yml
docker compose kill -s HUP blackbox
sleep 2
curl -sf 'http://127.0.0.1:9115/probe?module=http_2xx_tls&target=https://secure-shop/' \
  | grep -E '^probe_success|^probe_ssl_earliest_cert_expiry'

Green. Now undo it and do it properly: keep verification on, and tell the probe which certificate authority to trust. server_name sets the SNI the probe sends, which is what the certificate’s subjectAltName must match.

cd "$HOME/obs-blackbox-lab"
sed -i 's/insecure_skip_verify: true/insecure_skip_verify: false/' blackbox/blackbox.yml
python3 - <<'PY'
from pathlib import Path
p = Path('blackbox/blackbox.yml')
s = p.read_text()
s = s.replace(
    "      tls_config:\n        insecure_skip_verify: false\n",
    "      tls_config:\n        insecure_skip_verify: false\n"
    "        ca_file: /etc/blackbox/lab-ca.pem\n"
    "        server_name: secure-shop\n")
p.write_text(s)
PY
docker compose kill -s HUP blackbox
sleep 2
curl -sf 'http://127.0.0.1:9115/probe?module=http_2xx_tls&target=https://secure-shop/' \
  | grep -E '^probe_success|^probe_ssl_earliest_cert_expiry'

Green again — but this time the probe verified the chain and the hostname, so the green means what a reader assumes it means. Convert the expiry metric into the number a human can act on, using the arithmetic from lesson 06:

EXPIRY=$(curl -sf 'http://127.0.0.1:9115/probe?module=http_2xx_tls&target=https://secure-shop/' \
  | awk '/^probe_ssl_earliest_cert_expiry/ {printf "%d", $2}')
NOW=$(date +%s)
echo "days remaining: $(( (EXPIRY - NOW) / 86400 ))"

# The ground truth the metric must agree with.
openssl x509 -in secure-shop/certs/secure-shop.crt -noout -enddate

Roughly twenty days, and the two must agree. If they disagree, the probe is reaching a different endpoint than openssl is.

Task 8: Alert on the four boundaries

prometheus/rules/blackbox.yml:

groups:
  - name: blackbox
    rules:
      # Two consecutive failed scrapes, not one. A single failed probe is
      # noise; the interval is 15s, so `for: 1m` needs four in a row here.
      - alert: ProbeFailing
        expr: probe_success == 0
        for: 1m
        labels: {severity: critical}
        annotations:
          summary: 'Probe {{ $labels.module }} failing for {{ $labels.instance }}'

      # The body-regex failure has a different cause and a different owner
      # from a transport failure, so it gets its own alert.
      - alert: ProbeContentUnexpected
        expr: probe_failed_due_to_regex == 1
        for: 2m
        labels: {severity: warning}
        annotations:
          summary: '{{ $labels.instance }} returns 200 with unexpected content'

      - alert: TLSCertExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 30 * 86400
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: 'Certificate for {{ $labels.instance }} expires within 30 days'

The TLSCertExpiringSoon rule needs a TLS probe running on a schedule, so add the fourth and final scrape job:

  - job_name: blackbox_tls
    metrics_path: /probe
    params:
      module: [http_2xx_tls]
    scrape_interval: 30s
    static_configs:
      - targets: ['https://secure-shop/']
        labels:
          service: secure-shop
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115
      - target_label: module
        replacement: http_2xx_tls

Then validate and reload:

cd "$HOME/obs-blackbox-lab"
docker compose exec prometheus promtool check rules /etc/prometheus/rules/blackbox.yml
docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -sf -X POST http://127.0.0.1:9090/-/reload && sleep 90

curl -sf http://127.0.0.1:9090/api/v1/alerts \
  | python3 -c 'import json,sys; [print(a["labels"]["alertname"], a["state"]) for a in json.load(sys.stdin)["data"]["alerts"]]'

TLSCertExpiringSoon should be firing, because you gave the certificate twenty days. That is the point of the twenty: an alert you have watched fire is an alert you believe.

Under the hood

The exporter holds no state between probes. /probe builds a fresh Prometheus registry per request, runs the prober for the named module against the named target, and formats that registry into the response — which is why two curl calls a second apart are two genuinely independent probes with no caching between them, and why scrape_interval alone controls probe frequency.

That statelessness is also why the relabel chain has to be exactly right. The exporter has no idea what it should be probing; it probes whatever arrives in the target parameter. In Task 3 it was told to probe blackbox:9115, so it did, faithfully, for eleven weeks. A component with no opinion about its own inputs pushes the entire correctness burden onto the configuration that feeds it, and that configuration is four lines of YAML that promtool cannot evaluate for meaning.

SIGHUP re-reads blackbox.yml in place. If the new file is invalid the exporter keeps the old configuration and logs the parse error rather than exiting, so a silent typo leaves you probing with the previous module definitions — check docker compose logs blackbox after every reload rather than assuming.

Validation

Each of these is a claim the lab makes. Check them rather than trusting the narrative.

curl against /probe with module=http_2xx_shop and a healthy shop returns probe_success 1 and probe_http_status_code 200.
With the inherited relabel order and shop stopped, instance reads blackbox:9115, and you have recorded the probe_success value that came with it — the series describes the exporter either way.
With the corrected order, instance reads http://shop/healthz and probe_success follows the shop’s real state.
Against the maintenance page, http_2xx_shop reports probe_success 1 and http_2xx_shop_strict reports probe_success 0 with probe_failed_due_to_regex 1 — same request, same status code, opposite verdicts.
With nginx returning 503, tcp_connect_shop is green and http_2xx_shop is red with probe_http_status_code 503.
http_2xx_tls is red with the default trust store, green with insecure_skip_verify: true, and green with ca_file plus server_name — and only the third of those verified anything.
probe_ssl_earliest_cert_expiry converted to days agrees with openssl x509 -noout -enddate on the same certificate.
TLSCertExpiringSoon appears in /api/v1/alerts in state firing.
promtool check config and promtool check rules both pass on the final files.

Expected Outcome

obs-blackbox-lab/
├── blackbox/
│   ├── blackbox.yml
│   └── lab-ca.pem
├── compose.yaml
├── notes.md
├── prometheus/
│   ├── prometheus.yml
│   └── rules/blackbox.yml
├── secure-shop/
│   ├── certs/{secure-shop.crt,secure-shop.key}
│   └── conf.d/default.conf
└── shop/
    ├── conf.d/default.conf
    └── html/index.html

Four probe modules, four scrape jobs whose relabel chains you can defend line by line, three alert rules, and a notes.md containing the two-by-two triage table and the evidence from the broken chain.

Troubleshooting

A container exits immediately after up -d. Almost always a port conflict on 9090 or 9115, or a YAML error in a mounted config. docker compose logs blackbox and docker compose logs prometheus print the parse error verbatim; the exporter refuses to start on an invalid blackbox.yml, which is itself the config check.

probe_success 0 for everything, including targets you can reach. Probes run from inside the network, so shop resolves for the exporter and not for you. Test from where the probe runs: docker compose exec blackbox wget -qO- http://shop/healthz if the image has wget, or re-run the /probe URL with &debug=true and read the resolution step.

The scrape shows up 0 for a blackbox job. That is a scrape-boundary problem, not a probe problem. Either __address__ was never rewritten to the exporter (Prometheus is trying to fetch /probe from the target itself) or the exporter is down. Check /targets in the Prometheus UI: the scrape URL is printed there, and it should point at blackbox:9115.

A relabel change appears to do nothing. Relabel regexes are anchored at both ends, so regex: shop matches the value shop and nothing else. Confirm what Prometheus actually loaded with curl -s http://127.0.0.1:9090/api/v1/status/config rather than reading the file on disk — an unreloaded config is the commonest cause of a fix that “did not work”.

The TLS probe stays red after adding ca_file. Two candidates. The exporter runs as uid 65534 and cannot read a mode-600 file: ls -l blackbox/lab-ca.pem should show 644. Or the SNI does not match the certificate’s subjectAltName — server_name must be secure-shop, exactly the name in the -addext from Task 1.

The expiry alert never fires. for: 5m plus a 15s evaluation interval means at least five minutes of continuous truth. Check the rule is loaded at all with curl -s http://127.0.0.1:9090/api/v1/rules; an unmatched rule_files glob loads zero rules and reports no error.

docker compose exec shop nginx -s reload fails. The config is mounted read-only and nginx validates before reloading, so a syntax error leaves the old config running — which is the behaviour you want. docker compose exec shop nginx -t prints the line number.

Cleanup

Step 1. Keep the deliverables. They are the reason the lab existed:

mkdir -p "$HOME/obs-lab-deliverables"
cp -a "$HOME/obs-blackbox-lab/notes.md" \
      "$HOME/obs-blackbox-lab/blackbox/blackbox.yml" \
      "$HOME/obs-blackbox-lab/prometheus" \
      "$HOME/obs-lab-deliverables/"

Step 2. Remove the containers, the network and the TSDB volume. Without -v the named volume survives and quietly holds a few hundred megabytes.

Destructivehost— Removes the project's containers, network and the prometheus_data volume.

$ cd ~/obs-blackbox-lab && docker compose down -v

Step 3. Confirm nothing of the project is left running, and that no other Docker workload was touched:

docker compose -p obs-blackbox ps -a
docker volume ls | grep obs-blackbox || echo "no volumes remain"

rm -rf removes the whole lab tree, including anything you added

The next block deletes ~/obs-blackbox-lab recursively and without confirmation, including the private key, both nginx configs and any notes you wrote that the copy step above did not know about. Step 1 saves the lab’s own deliverables and nothing else.

The path is written in full and anchored to $HOME deliberately: a relative rm -rf run from the wrong directory is how people delete work they meant to keep. Look at what you are about to remove first:

ls -la "$HOME/obs-blackbox-lab"

rm -rf "$HOME/obs-blackbox-lab"

The images remain in your local cache. docker image rm prom/prometheus:v2.55.1 prom/blackbox-exporter:v0.26.0 nginx:stable-alpine removes them if you want the disk back; leaving them costs nothing but space.

What You Learned

The exporter probes whatever it is told to probe. It has no opinion, no scheduler and no state, so a relabel chain that hands it the wrong target produces a green dashboard for as long as nobody looks at instance.
Order is the whole of the relabel chain. Capture the target, build identity from it, then redirect the scrape. Reversing the first and last rule is valid YAML, passes promtool, and inverts the meaning.
instance is attribution, not decoration. Left to default it becomes the prober, and every alert for every target collapses into one.
HTTP 200 is a statement about the status line. fail_if_body_matches_regexp and fail_if_body_not_matches_regexp are what turn a reachability probe into a correctness probe, and probe_failed_due_to_regex is the only metric that distinguishes the two failures.
A pair of probes names a boundary. TCP green plus HTTP red is “the listener is up and the application is refusing”; both red is “the failure is below the application”. One probe on its own says much less than two.
insecure_skip_verify: true is a way of making a probe green by deleting the check. ca_file plus server_name is the fix that leaves the assertion intact, and only the second one means anything to the customer.
probe_ssl_earliest_cert_expiry is a countdown, not a status. It must agree with openssl x509 -enddate, and the alert on it should be one you have watched fire.

Why this matters

The incident in the scenario is not hypothetical in shape: a probe suite that was green through an outage is one of the most common findings in a synthetic monitoring review, and the cause is nearly always one of the four in this lab. None of the four produces an error. None is caught by config validation. Each is a small, reasonable-looking piece of YAML that changes what question is being asked, while leaving the dashboard’s answer looking identical.

That is why the discipline this lab drills is reading the label, not reading the colour. instance tells you what was probed. module tells you what was asked. probe_http_status_code and probe_failed_due_to_regex together tell you which layer answered. A green panel with no attribution is not evidence of anything, and the forty minutes the customer spent unable to reach the shop is what that costs.

Objectives

Prerequisites