The three pillars of observability · Docker & Containers

Every production incident is a question, and the three pillars are three shapes of answer.

Most treatments of this subject stop there, which leaves out the part that actually decides your architecture: the three have completely different cost curves. Doubling your traffic doubles your log bill and your trace bill and does approximately nothing to your metrics bill. Adding one label to one metric can cost more than a year of logs.

This lesson is the comparison with the numbers in it.

What each one answers

Metrics answer how much and how many. They are numbers sampled on a schedule, pre-aggregated by the thing producing them. Request rate, error rate, queue depth, memory.current from the container’s cgroup. A metric is cheap to store, fast to query over months, and cheap enough to evaluate an alert rule against every fifteen seconds.

Logs answer what happened. Discrete events with arbitrary context: this request, this SKU, this error, this stack trace. Rich, searchable, and expensive per event.

Traces answer where did the time go. One request’s path through every service, with a duration on each hop. Sampled, per-request, and the only signal that crosses a container boundary intact.

	Metrics	Logs	Traces
Unit	a time series	an event	a request tree
Aggregate questions	yes	slowly	no — sampled
Individual request	no	yes	yes, if sampled
Causal ordering across services	no	no	yes
Cost scales with	cardinality	traffic × bytes	traffic × sampling × spans
Retention typically	months	days to weeks	days
Good for alerting	yes	rarely	no

Choosing, by question

"Is anything wrong right now?" — metrics. An alert must evaluate in seconds against a bounded amount of data, which is what a metric is and what a log search is not.
"How often does this happen, and is it getting worse?" — metrics. Trend over months is exactly what a time series is for.
"What exactly happened to this one request?" — logs, found by the correlation ID from the previous lesson.
"Which of the five containers was slow?" — traces. Nothing else records the causal tree.
"Why was that container slow?" — traces to find the hop, then logs for that trace ID to find the reason.
"Has this ever happened before?" — metrics if you thought to record it, logs if you kept them long enough, and neither if the answer is a trace older than a week.

The last one is the honest one. Retention differs by an order of magnitude between the pillars, so “has this happened before” is answerable at very different depths depending on which signal captured it — and that is a design decision you make in advance, not during the incident.

The cost of each, with numbers

Take a modest Docker host: 20 containers, 200 requests per second, each request touching 4 services.

Metrics. A Prometheus time series costs a few kilobytes of memory while it is active, and a compressed sample costs on the order of two bytes on disk. With 20 containers exporting roughly 300 series each plus node_exporter and cAdvisor, call it 15,000 active series scraped every 15 seconds:

15,000 series x 4 samples/min x 60 min x 24 h  = 86.4M samples/day
86.4M x ~2 bytes                               = ~170 MB/day
90 days retention                              = ~15 GB

Fifteen gigabytes for a quarter of history. Notably, 200 requests per second appears nowhere in that calculation. Ten times the traffic produces exactly the same number of series and the same storage.

Logs. One structured JSON line per request per service, at a realistic 400 bytes once you include the correlation fields:

200 req/s x 4 services x 400 bytes  = 320 KB/s
320 KB/s x 86,400 s                 = ~27 GB/day uncompressed
compressed ~10:1                    = ~2.7 GB/day
14 days retention                   = ~38 GB

Ten times the traffic is ten times that. And 400 bytes per line is optimistic — a debug-level line with a stack trace is several kilobytes, which is how a single misconfigured log level turns 27 GB into 400 GB overnight.

Traces. At 10% head sampling, with each request producing about 8 spans across 4 services, and a span costing roughly 500 bytes:

200 req/s x 0.10 x 8 spans x 500 bytes  = 80 KB/s
                                        = ~6.9 GB/day uncompressed
compressed                              = ~1 GB/day
7 days retention                        = ~7 GB

The sampling rate is the whole lever. At 100% sampling that is 69 GB a day before compression, which is why sampling is not an optimisation but a precondition. What you give up by sampling is the subject of the sampling lesson later in this part.

Pillar	Per day	Retention	Total	Doubles when
Metrics	~170 MB	90 days	~15 GB	you double cardinality
Logs	~2.7 GB	14 days	~38 GB	you double traffic
Traces (10%)	~1 GB	7 days	~7 GB	you double traffic or the sampling rate

Cardinality: the one that bites without warning

Metrics look cheap right up until they are not, and the mechanism is multiplication.

Why one label multiplies rather than adds

Prometheus stores a separate time series for every unique combination of label values. The series count for a metric is the product of the distinct value counts of its labels, not the sum.

http_requests_total{service, method, status, endpoint}
  10 services x 5 methods x 8 statuses x 20 endpoints = 8,000 series

Add one more label with 50 distinct values and it is 400,000. Add a label whose values are unbounded — a user ID, a full URL with a query string, a request ID — and there is no bound at all.

The Prometheus guidance is explicit: keep a metric’s cardinality below 10, and above 100 “investigate alternate solutions such as reducing the number of dimensions or moving the analysis away from monitoring”. Its own worked example describes a double-digit number of millions of series as “too much for the current implementation”.

The failure is not graceful. Each active series holds memory in the head block, so ingestion pushes memory steadily upward until the container hits its memory.max and is OOM-killed. On restart Prometheus replays the write-ahead log, rebuilds the same series, and is killed again — a crash loop that survives every restart, because the cause is the data rather than the process.

Container churn is a cardinality source nobody plans for

The unbounded-label failure above is at least deliberate — somebody added the label. This one arrives on its own.

cAdvisor and similar exporters label container metrics with the container ID and often the image tag. Both change on every deploy. Every docker compose up -d destroys twenty containers and creates twenty new ones with twenty new IDs, so every deploy mints a fresh set of series and orphans the old ones.

The old series are not deleted. They stay in the head block until they age out of the retention window, so active series grow with your deploy frequency, not with your container count. A team deploying twenty times a day with twenty containers adds 400 container identities a day, each with the dozens of series cAdvisor exports.

The symptom is a Prometheus whose memory climbs on a sawtooth that tracks the deploy calendar rather than traffic, and which is fine for months and then is not.

The mitigations, in order of how much they help:

Drop the churning labels in metric_relabel_configs at scrape time — id, container_id, and the full image including the tag. Keep the Compose service name, which is stable across deploys and is the label you actually query by.
Shorten retention if the history is not being used. The default fifteen days of a high-churn series set is usually fourteen days of data nobody will query.
Measure it, so the growth is a graph rather than an outage. The command below gives you the numbers.

Read-only / Safemeasure your own cardinality— The TSDB status endpoint names the worst offenders directly.

PROM=prometheus

# Total active series, plus the top offenders by metric and by label.
docker compose exec -T "$PROM" wget -qO- 'http://localhost:9090/api/v1/status/tsdb?limit=15' | python3 -m json.tool

# The single number to trend. If this climbs while traffic is flat,
# something is churning.
docker compose exec -T "$PROM" wget -qO- http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_series | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["result"][0]["value"][1])'

seriesCountByMetricName tells you which metric to fix and labelValueCountByLabelName tells you which label is responsible. Those two lists together turn “Prometheus is using a lot of memory” into a specific relabelling rule, usually in under five minutes.

Read-only / Safemeasure your own log volume— What the containers on this host are actually producing right now.

# Bytes written by each container's json-file log, largest first.
sudo du -sh /var/lib/docker/containers/*/*-json.log 2>/dev/null | sort -rh | head -n 10

# Lines per minute from the noisiest container, measured rather than guessed.
CONTAINER=api
docker logs --since 1m "$CONTAINER" 2>&1 | wc -l

# Average bytes per line, which is the multiplier in the sizing above.
docker logs --since 1m "$CONTAINER" 2>&1 | awk '{n++; b+=length($0)+1} END {print b/n" bytes/line"}'

Multiply those two numbers by your container count and your retention and you have your log bill. Almost nobody has done this, and the result is routinely an order of magnitude away from the estimate.

The pillars compose because each is blind exactly where another sees.

Question	Metrics	Logs	Traces
Which request failed?	no — only the rate	yes	yes, if sampled
How often does this rare failure occur?	yes	only if you count them yourself	no — sampling distorts it
What is the aggregate error rate?	yes	slowly and expensively	no
Where did the latency go?	no — only that it went	no	yes
What was the value of that variable?	no	yes	in span attributes
What happened three months ago?	yes	usually gone	gone

Two of those are worth stating explicitly because they are the ones people get wrong in opposite directions:

You cannot compute a reliable error rate from traces. They are sampled. A 10% sample tells you about 10% of requests, and if the sampler is tail-based and biased toward errors, the proportion in your trace store is not the proportion in reality at all.
You cannot alert on logs the way you alert on metrics. A log query scans events; a metric query reads a compressed series. At scale the log query is slower than the alert interval, and an alert rule that cannot finish before it next fires is not an alert rule. If you want to alert on something that only appears in logs, count it into a metric at the collector and alert on the counter.

What to build, and in what order

Metrics first, for everything you ship. They are the cheapest, the only thing you can reliably alert on, and the only thing that will still be there in three months. Start with the four golden signals per service: rate, errors, duration, saturation.
Structured logs next, with the correlation fields. Unstructured logs are only slightly better than no logs once there are more than three containers, because you cannot join them.
Traces once there are more than two services in a request path. With a single container a trace tells you little that a log timestamp does not. With five, it is the only thing that answers the question.
Then connect them — the exemplar from metric to trace, the trace ID in every log line. The pivots are where the value is; each pillar alone is worth a fraction of the three together.

Verification that can fail

Read-only / Safethe top cardinality offenders— If one metric name dominates this list, that is where to start.

$ docker compose exec -T prometheus wget -qO- 'http://localhost:9090/api/v1/status/tsdb?limit=5' | jq -r '.data.seriesCountByMetricName[] | [.value, .name] | @tsv'

   184213 container_network_receive_bytes_total
  92104 container_fs_writes_bytes_total
  91882 container_cpu_usage_seconds_total
   8014 http_request_duration_seconds_bucket
   3200 http_requests_total

Illustrative output

Three cAdvisor metrics accounting for 95% of the series is the churn pattern from the callout above, and the fix is a relabel rule rather than more memory.

Read-only / Safeprove the three pillars are actually connected— Each of these can fail independently, and each failure is invisible from the others.

# 1. Metrics: is Prometheus scraping every target it should be?
docker compose exec -T prometheus wget -qO- http://localhost:9090/api/v1/targets | python3 -c 'import json,sys; [print(t["labels"]["job"], t["health"]) for t in json.load(sys.stdin)["data"]["activeTargets"]]'

# 2. Logs: is at least one container emitting a parseable trace ID?
docker compose logs --since 5m --no-log-prefix api | grep -c 'otelTraceID' || echo 'NO TRACE IDS IN LOGS'

# 3. Traces: did the collector accept anything in the last interval?
docker compose exec -T otel-collector wget -qO- http://localhost:8888/metrics | grep 'receiver_accepted_spans'

All three passing is the minimum bar. Any one failing means one third of your incident response is unavailable, and you will find out during the incident rather than now.

What if you can only afford one pillar?

A real constraint on a small Docker host, and the answer is not “the most powerful one”.

Take metrics. Not because they answer the most questions — they answer the fewest — but because of what they uniquely enable:

They are the only pillar you can alert on at a useful evaluation interval. Without metrics you find out about incidents from users.
They are the only one you can afford to keep for months, and capacity planning, trend analysis and “was it always like this?” all need months.
Their cost does not scale with traffic, so a metrics budget you size once stays sized.

The other two have partial free substitutes and metrics do not. Without a log pipeline you still have docker logs on the host for whatever the rotation kept — degraded, but present. Without traces you still have timestamps and, on a small stack, a tolerable amount of reasoning. Without metrics there is no substitute for a number that existed yesterday.

The order that follows for a constrained host: metrics with a bounded label set, then local or json-file log rotation configured so logs survive long enough to be useful without filling the disk, then traces if and when the request path crosses more than two containers.

And a warning about the cheapest-looking option: taking logs only, and computing rates by grepping them, is the trap. It looks free because the logs are already there. It ends in a log volume nobody sized, an alerting path that cannot evaluate fast enough, and no history, because retention was chosen by whatever fitted on the disk.

Knowledge check

Why this matters

The pillars are usually presented as a taxonomy, which makes them sound like three interchangeable ways of looking at the same data. They are not. They have different cost curves, and choosing between them is mostly an economic decision made under a budget.

The two numbers to carry away are the ones that behave in opposite directions. Log cost scales with traffic; metric cost scales with cardinality. That is why a busy service with three well-chosen labels is cheap to monitor and expensive to log, and why a quiet service with a per-user label can take down a Prometheus that a thousand times the traffic would not have troubled.

Everything else follows from measuring your own numbers rather than inheriting somebody’s estimate — and both measurements in this lesson take about a minute.

Where next

The next lesson is the plumbing that carries all three: running the OpenTelemetry Collector on a Docker host, and choosing between the topologies that decide what happens when it is unavailable.