Labels, Dimensions, and Time Series

A time series is the unique combination of metric name and label set. Adding a label with N values multiplies the series count by N. The label design is the discipline that keeps Prometheus memory and storage predictable.

This lesson covers the parts of a time series, the rules for safe label design, and the operational meaning of each label.

Anatomy of a time series

http_requests_total{method=GET, route=/checkout, status=200}  4096  @1700000000
   |                   |                                |                |        |
   |                   +-- label set                    |                + sample  + timestamp
   +-- metric name                                                   (value)

Five parts:

1. Metric name. The Prometheus convention is <unit>_<noun>_<verb>. Examples: http_requests_total, node_memory_MemAvailable_bytes, http_request_duration_seconds_bucket.
2. Label set. Zero or more key-value pairs. Order does not matter; the set is the same regardless of order.
3. Value. The sample. A counter is an integer; a gauge is float; a histogram has multiple components.
4. Timestamp. The scrape-time (or push-time, for Pushgateway) millisecond-precision timestamp.
5. Metadata. Help, unit, optional labels like le="..." for histograms.

How labels become series

Adding a label multiplies the series count:

http_requests_total                                  — 1 series
http_requests_total{method="GET"}                    — 1 series
http_requests_total{method="GET",status="200"}       — 1 series
http_requests_total{method="POST",status="500"}      — 1 series
http_requests_total{method="POST",status="200"}      — 1 series
http_requests_total{method="POST",status="500"}      — 1 series
                                                   = 5 total

Five label combinations = five time series. Each takes memory to index and disk to store. The metric itself has 5 series; the same metric with three status codes and three methods might have 9; with 50 status codes, 150.

The cardinality design decision is a memory and storage decision: how many distinct combinations of labels do you want the platform to track?

Label design rules

Five rules:

1. Distinct operations. Each label should identify a distinct operation, environment, host, instance, route, or status code. Labels are for aggregation, not for tagging arbitrary data.
2. Bounded cardinality. The number of distinct values for each label must be small and known. “User ID” is unbounded and dangerous. “Route” with 30 distinct paths is bounded and useful.
3. Operational alignment. Labels should map to operational dimensions: route, instance, environment, region. The label set is the answer to “how will I pivot in a dashboard?”
4. Coherence. All series of one metric should share most labels. A requests_total and request_duration_seconds with mismatched label sets cannot be joined cleanly.
5. No secrets. Labels are visible in URLs (/api/v1/query) and may be exposed. Do not label with secrets, PII, or internal hostnames that should not be public.

What labels map to operationally

The label set is the dashboard’s pivot dimensions. A good label set is the answer to “in an incident, what will I want to filter the metric by?”

Common operational labels:

• route. HTTP endpoint. Tells you which path the user hit. Cardinality should be bounded by the API design.
• method. HTTP verb. Tells you GET vs POST vs DELETE. Bounded by HTTP.
• status. HTTP status code class. Bounded by HTTP.
• instance. The producer of the metric (host, container, pod). Bounded by fleet size.
• region. Cloud region. Bounded by deployment.
• env. Production, staging, dev. Bounded to ~3.
• version. Application version. Bounded by deployment frequency.
• dependency. The downstream service. Useful for tracing.

A label like user_id, request_id, session_id, UUID is unbounded — adding a new label like this typically explodes the series count by orders of magnitude. Part IV on cardinality covers this in production depth.

How many series is too many?

A small Prometheus can comfortably hold ~1 million active series. A medium one can hold 10 million. The label design budget per metric is a design-time choice:

• Node metrics. ~5000 series across ~1000 hosts × ~5 collectors with a few labels each.
• Application request metrics. Bounded by route × status × method × instance. ~10000 series for a mid-size service.
• Histograms. × bucket count — a 12-bucket histogram produces 12 + 2 series.
• Sum across the platform. Depends on fleet size; design the label budget in advance.

A team that has not designed the label budget discovers it during the cardinality incident, not before.

Operational reading

The label set is what the operator uses to pivot the metric in the dashboard. The label set is what the alert fires on. The label set is what the incident response uses to bound the search.

A good label design:

• Always has instance (so the operator can find which host is affected).
• Always has route (so the operator can find which endpoint is affected).
• Has version (so the operator can bound the search to a recent deploy).
• Has bounded cardinality.

A bad label design:

• Has user_id (operator cannot filter; cardinality explodes).
• Has request_id (operator cannot reuse the dashboard; every request becomes a new series).
• Has error_message (operator cannot read; metric is cardinality-bloated and unreadable).

The label design is the dashboard design. The two are inseparable.

Production guidance

• Design the label budget at metric instrumentation time. Discuss every new label in a code review.
• Test for cardinality before deploying a new label. Use the tsdb tool or promtool to estimate series count.
• Use recording rules to compute aggregations that would otherwise consume excessive series (e.g. sum the per-route counter into a fleet-wide counter).
• Document the label set in the instrumentation guide.
• Avoid labels with high cardinality or unbounded values.

Verification

You should be able to answer:

• What are the parts of a Prometheus time series?
• How do labels multiply series count?
• Why are high-cardinality labels dangerous?
• What does a “good” label design look like?

Quiz