Blackbox Incident Runbook

A paging alert fires at 14:42. The body says: probe_success{service="customer-portal"} == 0 for 2m. The engineer opens the panel; the customer portal is red. The checkout service next to it is green. The internal HTTP panel says the customer portal service is healthy. Three signals, three viewpoints. The engineer needs a runbook that does not lead them into the wall of dashboards. This lesson is that runbook.

What it is

A blackbox incident is an event where the blackbox_exporter panel disagrees with the internal observability stack or with the user. The shapes vary; the disciplined response is the same. The runbook has four phases: shape recognition, boundary isolation, evidence collection, and rollback or verification. Each phase has a small set of commands that take seconds to run and tell the engineer which boundary is broken.

The runbook assumes three things about the platform:

• The exporter is deployed in at least two regions so the operator has a second perspective to compare against.
• The exporter’s probe job references a dedicated service label so alerts group by the user-facing service, not by exporter host.
• The scrape config and blackbox.yml are version controlled and recoverable.

Why a sysadmin cares

A failing blackbox probe is a paging-grade signal. The signal is right; the runbook’s job is to turn the page into an actionable diagnosis inside the SLO budget. Without the runbook the engineer opens panels, does not know which to trust, and either escalates too early or waits until the issue is user-visible.

The four-phase shape exists because blackbox failures classify into a small set of distinct boundaries: the exporter process, the target’s service, the route between them, and the exporter’s configuration. Each boundary has a fingerprint in the metric stream and a distinct first command to run.

How it works

   phase                  boundary            first signal
   ----------------------+-------------------+----------------------
   1. shape recognition  | the metric itself | promql:
                          |                   |   sum by (service,module) (probe_success == 0)
   2. exporter alive?    | the exporter host | curl http://exporter:9115/-/ready
   3. target alive?      | the target's path | curl http://exporter:9115/probe?module=...&target=...
   4. compare to internals| the platform view | up{job="..."} / synthetic from another region
   5. configuration?     | the change        | git log /etc/prometheus /etc/blackbox
   6. rollback / document| the runbook       | git revert + verify

Each step takes seconds. The order matters because the cost of chasing the wrong boundary is the cost of the SLO budget.

Phase 1: shape recognition

The first sign of a blackbox incident is the panel turning red. The shape of the failure tells the engineer which boundary is broken before any command runs.

# Cardinality of the failure: how many targets, which modules.
probe_success == 0
# Read the legend. Look for these signatures:
#   * many services, one module  -> probably a module regression
#   * many services, all modules -> exporter host or its egress
#   * one service, all modules   -> target or its route
#   * one service, one module    -> module config or target change
#   * one service, intermittent -> flapping exporter or rate-limited probe

The four shapes correspond to four different boundaries. The operator’s first decision is which shape they see.

Phase 2: exporter alive?

Confirm the exporter process is up and answering requests before debugging the targets.

# READ-ONLY — does not modify anything.
curl -sf http://blackbox:9115/-/ready
# ready

curl -sf http://blackbox:9115/metrics | grep -E '^blackbox_'
# blackbox_exporter_build_info{version="0.26.0"} 1
# blackbox_exporter_config_last_reload_successful 1
# blackbox_exporter_probe_total 18

# If ready returns 5xx or times out, the exporter is the failure.
# If config_last_reload_successful is 0, a recent reload was bad.

config_last_reload_successful == 0 is the most important signal here. A bad reload is the most common root cause for “every probe failed simultaneously” and points at the change log rather than the network.

Phase 3: target alive?

Run the same probe manually, against the same exporter, against the same target. This isolates the boundary between the exporter and the target.

curl -sf "http://blackbox:9115/probe?module=http_2xx_portal&target=https://portal.example.com/" \
  | grep -E '^probe_(success|duration|http_status)'
# probe_success 0
# probe_duration_seconds 4.892  (hit the 5s timeout)
# probe_http_status_code 0       (no response captured)

curl -sf "http://blackbox:9115/probe?module=tcp_connect_portal&target=portal.example.com:443" \
  | grep -E '^probe_(success|duration)'
# probe_success 1
# probe_duration_seconds 0.184

A red HTTP probe and a green TCP probe proves the route and TLS handshake succeeded; the failure is in the application layer of the target. A red TCP probe proves the failure is at the network or listener.

Phase 4: cross-check with the internal and synthetic views

The probe is one view of the world. The platform has two more views; the runbook compares all three.

# Internal Prometheus up{} for the service.
up{job="customer-portal-svc"}
# {instance="customer-portal-7c4...:8080", job="customer-portal-svc"} 1

# Synthetic from another region (using the same module).
sum by (probe_success) (probe_success{service="customer-portal", region="eu-west-1"})
# 0
sum by (probe_success) (probe_success{service="customer-portal", region="us-east-1"})
# 1

Four combinations and their meaning:

Phase 5: configuration change?

A bad reload is the root cause more often than the network. The audit step is two minutes.

# READ-ONLY — confirm the recent change set.
git -C /etc/prometheus log --oneline -10
git -C /etc/blackbox log --oneline -10
# 7a3f2c1 change: tighten http_2xx_portal regex to require text/plain
# 6f8a... change: add 2 second timeout on icmp_router

The audit is the input to phase 6.

Phase 6: rollback or verify

If the change set is recent and consistent with the incident, roll back. If the change set is missing or not consistent, the failure is upstream and the runbook turns into an incident post-mortem.

# CONFIGURATION — reverts the change to the previous known-good.
git -C /etc/prometheus revert --no-edit 7a3f2c1
git -C /etc/blackbox revert --no-edit 7a3f2c1
systemctl reload prometheus
systemctl reload blackbox_exporter

# VERIFY — back to phase 2.
curl -sf http://blackbox:9115/-/ready
curl -sf "http://blackbox:9115/probe?module=http_2xx_portal&target=https://portal.example.com/" \
  | grep -E '^probe_success'
# probe_success 1

How to configure it for the runbook to work

The runbook assumes a small set of pre-conditions. None of them are large, and each one pays for itself the first time an incident lands.

Distinct exporters in two regions

# The exporter is stateless. Running two copies is free.
# The difference between their views is the signal you want.
deployments:
  - name: blackbox-eu-west-1
    region: eu-west-1
  - name: blackbox-us-east-1
    region: us-east-1

Each exporter registers a region label on every series through relabeling. The PromQL queries in the runbook group by that label.

Service label and module label

# Module name becomes a label on every sample.
# Service is set in the scrape config.
- target_label: module
  replacement: http_2xx_portal
- target_label: service
  replacement: customer-portal

The alert probe_success{service="X"} == 0 for 2m groups by service; the operator’s first action is to look at the service and module combination.

Versioned configuration

# The configuration is in git. Every change is reviewed.
# Every reload is recorded in the exporter's
# config_last_reload_successful metric.
ls /etc/blackbox/
# blackbox.yml
ls /etc/prometheus/
# prometheus.yml
# alerts/blackbox.yml

The audit step in phase 5 is a git log. The rollback step in phase 6 is a git revert. Both are sub-second commands.

Alert that names the boundary

# /etc/prometheus/alerts/blackbox.yml
groups:
  - name: blackbox
    rules:
      - alert: ProbeFailure
        expr: probe_success == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: 'Blackbox probe failure: {{ $labels.service }} module {{ $labels.module }}'
          description: 'Probe to {{ $labels.instance }} has been failing for 2m. Open the blackbox runbook.'
          runbook: 'https://runbooks.example.com/blackbox/probe-failure'
          dashboard: 'https://grafana.example.com/d/blackbox'

The annotation names the boundary the operator should open. It is the difference between a useful page and a useless one.

How to validate it

The runbook itself has a validation. Every step is a command that either succeeds or prints an error; the operator reads the output, not the metric, on the way to diagnosis.

# 1. Confirm the runbook's commands are still right.
promtool check config /etc/prometheus/prometheus.yml
blackbox_exporter --config.check --config.file /etc/blackbox/blackbox.yml

# 2. Confirm the alert rule loads.
promtool check rules /etc/prometheus/alerts/blackbox.yml

# 3. Confirm the synthetic from another region is alive.
curl -sf http://blackbox-us-east-1:9115/-/ready
curl -sf http://blackbox-us-east-1:9115/metrics | grep probe_success | head -3

# 4. Confirm the audit command runs against the right directory.
git -C /etc/prometheus status
git -C /etc/blackbox status

A dry-run of the runbook against a known-good baseline is the right monthly discipline for the on-call rotation.

How it can fail

The runbook fails in one of four shapes. Each shape has a distinct symptom.

1. Phase 1 misread. The operator conflates a local exporter-host failure with a service failure. The subsequent phases execute against the wrong hypothesis. Symptom: phase 3 confirms the target is reachable when the original alert said it was not — the diagnostic contradicts itself.

2. Phase 2 blind spot. config_last_reload_successful == 0 was true before the alert fired. The operator confuses this with the new failure, rolls back the wrong change, and the alert persists. Symptom: phase 6 “fixes” the alert temporarily; phase 2 still says 0; the next alert fires.

3. Phase 4 mismatch. The internal Prometheus has a different up than the probe says. The operator trusts the probe and ignores the internals, or vice versa, without forming a hypothesis. Symptom: the runbook stops at phase 4 and the team argues about which view is authoritative. The runbook says: form a hypothesis, find evidence, test, validate.

4. Phase 5 audit missed. A change was made through a back-channel not stored in the git repo. The audit step returns nothing; the operator concludes no change was made. Symptom: phase 6 cannot revert; phase 7 — the post-mortem — must catch the audit gap as its own finding.

5. Phase 6 rollback did not reload. A git revert followed by systemctl reload prometheus is the expected sequence. A revert without a reload leaves the running config pointing at the bad change. Symptom: probe_success does not change after the rollback; the alert persists.

6. Loud alert, silent incident. A alert fires for a single intermittent probe failure the team has been ignoring because the timeseries was noisy. The runbook is followed; the operator burns thirty minutes on a noise page. Symptom: the alert culture reacts to every individual page; pages become wallpaper; pages stop being read. The longer-running cost is the page fatigue, not the page itself.

How to troubleshoot when the runbook itself fails

A runbook failure is a meta-incident — the runbook did not lead the operator to the answer. The response is preservation-first.

1. Stop the active runbook invocation.
2. Preserve the state the runbook already modified (the partial rollback, the partial reload, the screenshot of the panel).
3. Open a senior engineer; the meta-incident is rarely an on-call job.
4. Write the post-mortem before the next shift, not at end-of-quarter.
5. Treat the runbook as the artefact to be revised; do not patch the symptom.

Security implications

The runbook prescribes actions that have a security footprint. The discipline is to apply the smallest necessary change at each phase and to record what was changed.

• Phase 5 reads git logs but does not commit. Read-only.
• Phase 6 reverts a change. The revert must be reviewed before deploy if the change was load-bearing.
• If the runbook prescribes opening a firewall port “to test from the office laptop,” do not do that. The test from the exporter host is already the right test.
• The exporter’s /probe endpoint is unauthenticated. During the runbook, do not pass credentials in the query string.

Performance implications

The runbook commands are cheap. The cost of running the whole sequence end-to-end during a quiet shift is well under a minute of operator time and a few hundred kilobytes of curl output. The expensive steps — the git revert, the systemctl reload — are infrequent and only on confirmed root cause.

What costs money is not running the runbook and instead opening every dashboard in parallel while the incident ticks. The discipline is the value; the commands are nearly free.

Production guidance

• Version-control every blackbox configuration file and every scrape config that touches a blackbox target.
• Reload through systemd, not through a shell session. systemctl reload is auditable.
• Add a small dashboard whose top row is the legend of the panel. The legend is the first action of phase 1. Make it the first thing the on-call looks at.
• Treat probe_success == 0 as one of the alert sources, not as the only one. Alert on up{job="<internal>"} for the internal service, alert on the synthetic from another region, and inhibit the blackbox alert when the synthetic is green for the same service.
• Add a monthly dry-run of the runbook. One engineer follows the runbook against a known-good baseline. Thirty minutes; one finding; one fix.

Verification

You should now be able to answer:

• Why does the shape of a blackbox failure (which services, which modules) tell you which boundary is broken before any command runs?
• In which phase would you discover that the failure is local to the exporter’s egress and how do you confirm it?
• What is the difference between rolling back via the exporter’s --config.check and systemctl reload? Which is the safer discipline?
• Why does cross-checking the probe with the internal up{} and with a synthetic from another region reduce the cost of being wrong about the boundary?
• What is the audit-trail discipline the team follows after the runbook concludes?

Quiz