White-Box and Black-Box Monitoring · Observability

White-box monitoring reads telemetry from inside the system: counters, gauges, internal queues, internal state. Black-box monitoring probes the system from the outside: “does a network request to the service succeed?” Both are required. A platform that has only one of the two has an observability gap that the production failure will find.

What white-box monitoring is

A white-box signal is emitted by the system itself. The source is the system’s own code:

A Go service exposes /metrics over HTTP. Prometheus scrapes it. The metrics carry the service’s view of itself: requests served, internal queue depth, goroutines, memory.
node_exporter reads /proc/stat, /proc/meminfo, /sys/class/net, and the kernel filesystem. The metrics are the host’s view of itself.
A database exporter reads its own engine statistics: connection pool, query latency, lock waits, replication lag.

White-box signals are:

Rich. They expose internal state that no external probe can see.
Cheap. The instrumentation is already in the service or exporter.
Conditional. They are not affected by load balancer or DNS configuration; they describe the source.

They are not, however, a substitute for black-box:

A service can be alive and responsive to its own scraper while blacking out external traffic because of a network ACL or DNS failure.
A service can be working perfectly internally while presenting a 502 to a user behind a reverse proxy with no health check.
A service can report zero errors while its actual user experience is 100% failure because the user-facing path is in a different code path the metrics do not cover.

What black-box monitoring is

A black-box signal probes the system from outside, treating it as an opaque box:

An HTTP probe (blackbox_exporter with http_2xx module) periodically requests https://shop.example.com/checkout from a probe host outside the application’s trust zone. The probe sees only what an external client sees.
A TCP probe opens a TCP connection to a port and validates that the handshake succeeds.
A DNS probe issues a query and validates the answer.
An ICMP probe pings a host.
An external synthetic user journey steps through a multi-page flow that emulates a real customer.

Black-box signals are:

Realistic. They see what a user sees — the full path through DNS, load balancer, TLS, application.
Trust-zone breaking. They run from a different network/identity than the service itself, catching misconfigurations the service’s own metrics cannot.
Slow-changing signal. A probe that runs once per minute detects gross outages, not 0.1% regressions.

They are not a substitute for white-box:

A black-box probe cannot tell you which service in the path is slow. It can tell you the path is slow.
A black-box probe cannot tell you why. It can tell you the problem is at a hop.
A black-box probe does not detect internal-only failures (e.g., background job queue saturation that does not affect latency).

How they map to the three signals

Question	Source	Signal
Did the request fail?	The service’s own counter (white-box)	metric (`http_requests_total{status="5xx"}`)
Did the external user hit an error?	The blackbox probe (black-box)	metric (`probe_success{...} = 0`)
What was the request body?	The service’s structured log (white-box)	log
Did the service respond at all from the user’s perspective?	The probe’s success metric (black-box)	metric
What was the dependency latency?	The service’s tracing instrumentation (white-box)	trace
What is the user-experienced latency?	The probe’s probe_duration_seconds (black-box)	metric

White-box and black-box telemetry combine into one investigation. A white-box alert (“error rate increasing”) plus a black-box probe failure (“probe_success == 0”) is more actionable than either alone.

When only one is in place

Common failure shapes:

Only white-box. A reverse proxy is misconfigured; the service is healthy; the user sees 502. White-box metrics stay green because the service is fine.
Only black-box. The service queues requests internally; the probe succeeds because each request returns in time. Internal queue length grows unbounded.
Both, misaligned. A blackbox probe target is wrong; an alert fires “users cannot reach us” while users are unaffected. The opposite: a service-wide failure that the probe does not cover.

The fix in each case is correlated white-box + black-box at the same observation points, with alerts grounded in user-impact metrics (SLOs) rather than service-internal metrics.

What happens during a network partition

White-box metrics may go silent: the scraper cannot reach the service. Black-box probes from a different network zone may continue, or may not, depending on whether the partition is bidirectional. A platform with only white-box monitoring is blind during the partition. A platform with only black-box monitoring learns the partition is user-affecting — but cannot diagnose which subnet or link is impaired.

Production guidance

Run black-box probes from at least two distinct networks (a public probe, an internal probe). The two perspectives differ during partial outages.
Probe the user-facing path, not the service’s own port. The user-facing path is what the user sees; the service’s internal port is a different question.
Anchor alerting on user-impact metrics, not service-internal metrics. The SLO part of the course returns to this — Part XXII.

Verification

You should be able to answer:

What is the operational difference between white-box and black-box monitoring?
Why does a platform with only one of the two have an observability gap?
How does each map to the three telemetry signals?
What failure mode does each gap exhibit?