Why Observability Exists

A production incident arrives at 03:00. Users report that checkout fails for roughly one in three attempts. CPU, memory and disk dashboards look normal. An engineer opens Grafana. Three dashboards and a wall of green panels later, they are no closer to an answer.

This is what the word observability exists to prevent. A system is observable when its operators can answer questions about it from its telemetry without shipping new code first. Monitoring is the subset that watches for pre-defined failure conditions. Observability is the superset that lets you investigate the conditions you did not pre-define.

What monitoring is

Monitoring is pre-defined. You decide in advance which signals matter, you instrument them, and you alert on thresholds. The classic example is the Nagios check: a host is up, a port is open, disk is below 90%. The checks answer the questions you remembered to ask.

The strength of monitoring is that it is cheap. The weakness is that it cannot answer a question you did not anticipate. When the unknown failure mode appears, monitoring is silent.

What observability is

Observability is the property that lets you reconstruct the internals of a running system from its outputs. A system is more observable when its outputs are richer and more correlated. Three signals make up the bulk of production telemetry today:

                  Production System
                         |
            +------------+-------------+
            |            |             |
         Metrics        Logs         Traces
            |            |             |
       Prometheus       Loki          Tempo
            \            |             /
             +-----------+-------------+
                         |
                      Grafana
                         |
                  Investigation
                         |
                  Alert / Action

• Metrics — numeric values sampled over time. Cheap to store, good for alerting and dashboards. A single instance of a counter tells you “how many” or “how long” but does not tell you about the specific request.
• Logs — discrete events with structured or unstructured content. Captured per-service. Good for “what happened” but expensive at scale and unhelpful if correlation metadata is missing.
• Traces — the journey of a single request through a distributed system. Trace spans capture parent / child relationships and per-span timings. Good for “where did the latency come from” and “which dependency failed”.

The three signals are not interchangeable. Each one answers questions the others answer only with great effort.

Symptoms vs causes

When a user reports “checkout is broken,” that is a symptom. The cause is somewhere upstream. The investigation proceeds from the symptom outward:

Symptom (user-visible failure)
    |
+--- Slack channel reports
    |
+--- Health probe fires
    |
+--- Service-level metric crosses threshold
    |
+--- Dependency metric crosses threshold
    |
+--- Application logs show error pattern
    |
+--- Trace shows slow span
    |
+--- Host metric shows resource pressure
    |
+--- Capacity / change / external dependency observed at same time
    |
Root cause

A symptom is observable. A cause must be inferred and verified. Telemetry that only describes the symptom leaves the investigation at the top of the tree. Telemetry that reaches host and dependency metrics, structured logs, and traces reaches the bottom.

Why “we have a dashboard” is not enough

Three failure shapes appear repeatedly in teams that collect metrics without designing for investigation:

1. The unknown failure mode. A deployment that introduces memory pressure only after eight hours. A race condition that only appears under sustained traffic. A certificate that expires only on the third Tuesday of the month. None of these had a dashboard. The team discovered the dashboard was useless the moment the failure appeared.
2. The unowned dashboard. Wallpaper. Forty panels of green. Each panel was someone’s idea; nobody is responsible for any of them now. When the question appears, no engineer knows which panel to open.
3. The alert that fires but does not tell you what is wrong. “CPU above 80% for 5 minutes.” The alert pages someone, but does not say which service, which host, which dependency. The on-call engineer is paid to wake up and start looking.

None of these problems are fixed by another panel. They are fixed by signal richness (more dimensions, structured logs, traces), correlation (trace IDs in logs, exemplars in metrics), and ownership (a name attached to every dashboard and alert).

What “evidence-driven investigation” looks like

The same incident, investigated the right way:

1. Define symptom. “User-visible checkout failure for ~30% of attempts.”
2. Quantify impact. Inspect the user-visible metric (HTTP 5xx rate, transaction count, business counter).
3. Form a hypothesis. The payment-service dependency is timing out. Disk I/O on the database host is spiking. A recent deployment introduced a regression. Pick the most likely one.
4. Find evidence. Open the dependency-latency panel. Open the database-host I/O panel. Open the change log. Open the application’s structured logs filtered by level=error and the deployment’s correlation ID.
5. Test the hypothesis. Does the database have elevated I/O latency that started at 02:50? Did a deployment run at 02:45? Does the trace of a single failed checkout show 4.2 s waiting on payment-svc?
6. Locate root cause. The payment-svc dependency started returning 503s after a deploy at 02:45; the deployment introduced a new connection-pool configuration that does not match the database’s max_connections.

No dashboard panel predicts this incident. The investigation succeeds because the telemetry is rich, correlated, and recent.

What a sysadmin should take from this lesson

1. Monitoring is pre-defined. Observability is post-hoc. Production observability is the property that lets the second one exist.
2. Metrics, logs and traces are not interchangeable. They answer different classes of question. A healthy platform has all three and a documented way to correlate them.
3. Symptoms are observable. Causes must be inferred. Rich telemetry gives more pivot points between the two.
4. Dashboards are not the goal. Investigation is.

The remaining lessons in Part I refine this mental model (short-lived vs durable telemetry, white-box vs black-box, context and causality) before Parts II and III introduce the practical disciplines (SLIs/SLOs, USE/RED) and Parts IV onward build the actual metrics / logs / traces stack.

What happens if the metrics are wrong?

The whole investigation collapses. A wrong metric labels the wrong service as the culprit, sends the on-call engineer down the wrong path, and may delay the actual fix by hours. The “first check the metric is right” step is reflexive, not optional, for every investigation.

What happens if logs have no correlation IDs?

Logs become searchable strings. Filtering for “what happened during this user’s request” requires full-text search across gigabytes of log lines per minute. The cost of log shipping balances against the cost of finding nothing.

What happens if traces never make it into the platform?

The “which dependency was slow” question reverts to inference from symptoms. The investigation becomes detective work in metric panels and logs. Time-to-resolution rises. Sometimes the answer never appears.

Verification

You should now be able to answer:

• What is the operational difference between monitoring and observability?
• What does each of metrics, logs, and traces uniquely answer?
• Why is a “symptom” different from a “cause”?
• Why does a wall of green panels not make a system observable?

Quiz