Break/Fix: Monitoring System Fails During Business Outage · Break/Fix · Observability

Reported symptoms

21:14 on a Friday. Customer support takes three calls in four minutes: checkout fails at the payment step. The checkout error-rate alert pages at 21:16. A Sev1 is declared at 21:19 and an incident commander is appointed.

The IC opens the pinned Sev1 dashboard. It does not load. Grafana returns 502.

The fallbacks do not work either. The runbook lists four, and the response team works through them in order over the next eleven minutes. The replica Prometheus, which exists precisely for this, times out. The pre-exported dashboard JSON is on the team file share, and the share will not mount. The read-only API token is valid, and a curl straight at the primary Prometheus times out too. The paper runbook is in the on-call bag; its print date is eleven months ago and two of its three contact numbers ring out.

The alerting path is fine. Pages keep arriving throughout, on time, correctly grouped. Whatever is wrong, it is not Alertmanager, and that fact makes the platform outage feel local and unimportant.

Some people can see, and some cannot. By 21:30 somebody notices that the two engineers who happen to be at the office can load Grafana. The three at home cannot. This is filed as a VPN problem.

Everything inside says checkout is healthy. An engineer with cluster access checks: the checkout service returns 200 on its health endpoint, up is 1 for every instance, and its request rate, error rate and latency are all normal. The blackbox probes for the checkout endpoint report success for the entire incident.

At 21:56, in the absence of anything else to try, the checkout deploy from 20:52 is rolled back. Customers still cannot check out. At 22:04 the team begins looking at the payment provider.

Evidence provided

The first useful measurement is the same request made from two places. Everything the team had tried so far was from one side of the perimeter.

Read-only / Safesame service, same path, two vantage points

$ curl -s -o /dev/null -w 'laptop grafana %{http_code} in %{time_total}s\n' \
--max-time 10 https://grafana.example.com/api/health
ssh jump-01.internal.example.com \
"curl -s -o /dev/null -w 'jumphost grafana %{http_code} in %{time_total}s\n' \
 --max-time 10 http://grafana.internal.example.com:3000/api/health"

laptop grafana 502 in 0.312s
jumphost grafana 200 in 0.041s

Illustrative output

The primary Prometheus behaves identically, and so does the replica that was supposed to be the answer when the primary does not.

Read-only / Safethe fallback resolves to the same address as the primary

$ for host in prometheus.example.com prometheus-replica.example.com; do
printf '%s ' "$host"
curl -s -o /dev/null -w '%{http_code}\n' --max-time 10 \
  "https://$host/api/v1/query?query=up" || echo timeout
done
dig +short prometheus.example.com prometheus-replica.example.com

prometheus.example.com 000
prometheus-replica.example.com 000
203.0.113.40
203.0.113.40

Illustrative output

From inside, the application is untouched. This is the evidence that made the team look at the payment provider, and read correctly it points the other way.

Read-only / Safecheckout is serving normally to everyone who reaches it

$ ssh jump-01.internal.example.com "curl -sG \
http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=sum by (code) (rate(http_requests_total{job=\"checkout\"}[5m]))' \
| jq -r '.data.result[] | .metric.code + \" \" + .value[1]'"

200 41.882
500 0.017

Illustrative output

And the synthetic check that exists to catch exactly this situation reported success throughout.

Read-only / Safethe probe passes, and it runs inside the perimeter

$ ssh jump-01.internal.example.com "curl -sG \
http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=probe_success{job=\"blackbox_checkout\"}' \
| jq -r '.data.result[] | .metric.instance + \" \" + .value[1]'"
kubectl -n monitoring get pods -l app=blackbox-exporter -o wide

https://checkout.example.com/health 1
NAME                       READY   STATUS    NODE
blackbox-exporter-0        1/1     Running   node-04.internal

Illustrative output

Two facts complete the picture. The team file share is published through the same edge tier as Grafana and the Prometheus endpoints. And the change calendar records a change to that edge tier applied at 21:09, five minutes before the first customer call.

Work the evidence before reading on

Two services are broken. They share no code, no database and no team.

Every check that runs from inside the perimeter is green, and every check that runs from outside it fails. Before naming a component, what is the smallest statement that covers both results?
The replica Prometheus was written into the runbook as the fallback for the primary being unavailable. It resolves to the same address. What was the fallback actually protecting against, and what did the author believe it protected against?
probe_success is 1 for the checkout endpoint while no customer can complete a checkout. The probe is not lying. What is it measuring, and what would have had to be true about the probe for its answer to be about customers?
Pages arrived normally all evening. Which direction does a notification travel, and which direction does a dashboard request travel?

Before continuing: checkout and the observability platform failed within the same minute. List everything they have in common. If that list has one entry, what is the correct next action, and why is it not opening a dashboard?

Root cause

1. One fault, two outages, one shared hop

Checkout was healthy for the entire incident. So was Grafana, so was Prometheus, so was the replica. Nothing in either system malfunctioned.

A single shared edge tier fronts the customer path to checkout and every human entry point into the observability platform: Grafana, the primary Prometheus, the replica Prometheus and the team file share. The change applied to that tier at 21:09 broke inbound routing through it. From outside, everything behind it became unreachable. From inside, nothing changed at all.

That is the whole fault, and it explains every symptom without needing a second cause. The office engineers could load Grafana because their traffic reaches the internal network without crossing the edge. Pages continued to arrive because notifications travel outbound and never traverse the inbound path. The application metrics were normal because the application was normal. Rolling back the 20:52 deploy changed nothing because that deploy was never involved.

2. The co-occurrence was the diagnosis and it was treated as an obstacle

Two services with nothing in common failed in the same minute. On this platform they share exactly one thing.

That inference was available at 21:19, from information already in the room, and it requires no telemetry whatsoever. It was not made because of how the failure was framed: the platform being down registered as an impediment to investigating checkout rather than as a fact about checkout. The response spent its first forty minutes trying to restore a view of a service that was fine, and its next twenty rolling back a deploy chosen because it was the most recent change anyone could see — while the change that mattered was in a calendar nobody thought to open, because the edge tier was not on anyone’s mental list of checkout dependencies.

This is the specific failure of judgement the scenario exists to train. When the monitoring platform fails at the same moment as the thing it monitors, that is not bad luck. It is a measurement, and it is usually the most precise one available, because shared failure implies shared dependency and the list of shared dependencies is short enough to enumerate on a whiteboard.

3. Every fallback shared the dependency it was meant to survive

The runbook documented four fallbacks and all four failed for the same reason.

The replica Prometheus was built to answer when the primary does not. It sits behind the same edge and resolves to the same address, so it can only answer when the primary is broken and the edge is not — which is a much narrower class of failure than the document implies, and not the class that occurred.

The exported dashboard JSON was on the team file share, published through the same edge. The direct API path terminated at the same edge. The paper runbook, the one fallback with no network dependency at all, was eleven months old and had drifted: two of three contact numbers were wrong.

None of this was ever tested. The pack was assembled as a list of artefacts — a replica exists, a snapshot exists, a token exists, a printout exists — rather than as a set of paths, and nobody had ever asked which network each one crosses. A fallback that has never been exercised is a design intention, and this incident is the first and only test it was given.

Under the hood

Two mechanisms explain why the platform looked healthy from every angle the team could reach, and both are worth understanding because they generalise well beyond this incident.

Notifications and dashboards travel in opposite directions. Alertmanager makes an outbound connection to the notification provider; a dashboard request is an inbound connection from a browser. An inbound edge failure leaves the outbound path completely intact, so pages keep arriving through a total loss of visibility. Continuing pages therefore prove that rule evaluation and notification work. They prove nothing about whether a human can see anything, and on this evening they actively reassured the team that the platform was mostly fine.

A blackbox probe measures the path it runs on. This one runs on a pod inside the perimeter, so it exercises the internal route to checkout, which was healthy throughout. Its probe_success of 1 is an accurate answer to the question it was asked. The trouble is that the question the team believed it was asking — can a customer reach checkout — is a different question that no instrument in this platform was positioned to answer. The vantage point of a synthetic probe is not a deployment detail; it is the definition of what the probe means. A probe that cannot see the edge cannot report an edge failure, and it will report success right through one.

Rolling back a deploy to see if it helps

At 21:56 the checkout deploy from 20:52 was rolled back. It was the only change anyone could see, the incident was forty minutes old, and doing something felt better than doing nothing.

Consider what that action costs during this particular incident. It is a change made with no hypothesis, in a system whose observability is degraded, so the outcome cannot be measured properly either way. If the rollback had appeared to help, the apparent improvement would have been noise and the team would have stopped looking — which is the worse branch, not the better one. And if the rollback had itself caused a problem, that problem would have arrived in a response that had already lost the ability to see problems arriving.

The instinct behind it is sound: at forty minutes with no progress, the pressure to act is real and should not be dismissed. The right expression of that instinct is to act on the one piece of evidence in the room that nobody had used — two unrelated services failing in the same minute — rather than on the most recent change that happened to be visible. Changing the system you cannot observe is the one move that reliably makes an incident longer.

Resolution

Name the meta-incident explicitly and give it a separate owner. The IC keeps the business incident; one named person owns restoring a read path and reports on a fixed cadence. Without that split, the entire response team debugs Grafana and nobody works the customer problem.
Ask what the two failing services share, before opening anything else. On this platform the list has one entry, and reaching it takes a minute. Do this even when the dashboard is available, because it is faster than a dashboard when it works.
Get a read path that avoids the broken hop rather than working down the fallback list in order. A jump host inside the perimeter reaches the Prometheus and Alertmanager APIs directly, and text answers to the right questions beat panels you cannot load.
Establish the direction of the fault from two vantage points. The same request from inside and from outside, against the application and against the platform, converts a confused incident into a one-sentence statement about a hop.
Take the edge change from 21:09 to the team that owns that tier, with the two-vantage-point evidence attached. Diagnosing what the change did is their work, not the response team work, and handing it over with evidence is faster than investigating it without access.
Appoint a scribe and keep the evidence ledger by hand. The usual tooling for it is behind the same edge, and a timeline reconstructed from memory afterwards is not a timeline.
Do not roll back application deploys speculatively. It is a change without a hypothesis in a system you cannot currently observe, and its main effect is to add a variable.
Tell customer support and the status page what is true: the service is healthy and the path to it is broken. That is a different message, a different expected duration, and a different set of questions from the ones support are currently answering.

Verification

Checkout completes end to end from an external vantage point, on a network that is not the office network. Every internal check was green throughout this incident and not one of them was about a customer.
The platform is reachable from that same outside position: Grafana loads, the primary Prometheus answers, the replica answers, and the file share mounts. Verify these separately from the business path — they failed together, but they are not guaranteed to recover together.
Each of the four documented fallbacks has a named network path written next to it. Any fallback that shares a hop with the primary is struck from the document rather than left in it with a caveat.
The blackbox probe now runs from outside the perimeter, and it can fail. Break the edge in staging and confirm probe_success goes to zero. A probe that has only ever returned 1 has never been tested.
A meta alert reaches the on-call rotation with the production Alertmanager deliberately stopped. If it does not, the meta alert path shares fate with the thing it is meant to report on.
The paper runbook has a print date inside the cover within the last quarter, and every contact number in it has been dialled.
The incident record shows two timestamps: when the meta-incident was named, and when a read path was restored. Those are the numbers the next drill is trying to reduce, and they cannot be recovered later.

Prevention

Put the observability platform on a different path from the product it watches. A shared edge tier is the cheapest possible guarantee that the platform will be unavailable exactly when it is needed, and no amount of redundancy behind that tier compensates for one shared hop in front of it.
Design the fallback pack as independent paths, not as a list of artefacts, and record the path next to each entry. A replica on a separate network. A dashboard snapshot on the device the engineer is physically holding rather than on a file share. A token that works against something reachable. A printed runbook with a print date on the inside cover.
Rehearse it quarterly. The rehearsal is the only thing that finds a shared dependency before an incident does, and it is cheap: block the edge in a drill and time how long the team takes to reach a read path. An unrehearsed fallback is a document, not a capability.
Probe from outside the perimeter, from more than one network. An internal probe measures a route no customer uses and will report success through a complete customer outage, which is worse than having no probe at all because it is trusted.
Route meta alerts through a separate Alertmanager on a separate channel. A platform outage has to be able to page even when the production alerting path is part of the outage, and the two paths should meet only at the notification provider.
Write the reasoning into the incident process itself: when two unrelated services fail in the same minute, enumerate what they share before investigating either one. It costs a minute, it needs no tooling, and on this evening it was worth ninety.

What if the edge had failed and checkout had genuinely been broken too?

This is the harder version, and it is worth thinking through because the response has to change.

The co-occurrence argument gets weaker: two faults at once means the shared dependency explains part of the picture and something else explains the rest. The discipline is to keep the meta- incident owner separate anyway, restore a read path first, and then work the application problem with instruments rather than with inference. Inference is what you use while you are blind; it is not a substitute for measurement once you can see again.

What does not change is the order. Establish a read path before investigating, because an investigation conducted without instruments produces conclusions nobody can check, and those conclusions tend to survive into the incident review as facts. The team that spends ten minutes getting to a jump host and then investigates will finish sooner than the team that starts guessing at minute zero, even though the second team looks faster for the first nine minutes.

And one thing gets easier: with two real faults, the rollback instinct is more likely to be right. That is precisely why it needs a hypothesis attached to it. Being right by accident during one incident is what teaches a team to be wrong on purpose during the next.

Why this matters

The monitoring system going down at the same moment as the thing it monitors is treated as bad luck. It is almost never bad luck. It is a measurement, and it is frequently the most precise one available, because two systems fail together when they share something and the list of things they share is short.

The practical consequence is a rule about where the platform sits. Every dependency the observability stack shares with production — an edge tier, a cluster, an identity provider, a DNS resolver, a storage backend — is a condition under which the platform is guaranteed to be missing at the worst possible time. Those dependencies are worth enumerating deliberately, because each one is a sentence of the form: when this breaks, we lose the service and our ability to look at it in the same instant.

The second consequence is about fallbacks. A fallback is not an artefact; it is a path. A replica that shares a network with its primary is a copy, not a fallback. A snapshot on a file server is a backup of a document, not a fallback. The only way to tell the difference is to trace the path each one uses and then break the primary on purpose and watch. Every fallback in this runbook looked correct on paper and none of them was, and the gap between those two statements is exactly one rehearsal that nobody had ever scheduled.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. One fault, two outages, one shared hop

2. The co-occurrence was the diagnosis and it was treated as an obstacle

3. Every fallback shared the dependency it was meant to survive

Resolution

Verification

Prevention