Metric Naming Conventions

An on-call engineer opens Grafana at 02:00 to graph request rate for the payment platform. One team exported http_requests_total. A second team exported requests_total. A third service, written a year later, exports http_server_requests_count. Three names, one concept. The dashboard has panels for two of them; the alert covers one. Nobody noticed for six months, because every panel looked plausible on its own.

Metric naming conventions exist to kill this class of quiet failure. A Prometheus metric name is not a display string. It is the primary key by which PromQL, recording rules, alerts and dashboards find your data. Naming is API design, and like any API it is expensive to change once consumers depend on it.

What it is

The Prometheus naming convention is a set of rules, documented in the upstream “Metric and label naming” guidance, that the ecosystem has standardised on. A well-formed name has four parts:

[namespace]_[name]_[unit][_suffix]

node_cpu_seconds_total
|    |   |       |
|    |   |       +-- type suffix: _total marks a counter
|    |   +---------- base unit: seconds, never milliseconds
|    +-------------- what is measured, snake_case
+------------------- namespace: which exporter or library

The rules that matter operationally:

1. snake_case. Lowercase letters, digits, underscores. The name must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*. Colons are reserved for recording rule output names — never use them in instrumentation.
2. Base units. Seconds, not milliseconds. Bytes, not kilobytes. Ratios, not percentages. http_request_duration_seconds, not http_request_duration_ms. Threshold maths written against one unit family is wrong by a factor of 1000 against another.
3. Type suffixes. A counter ends in _total. A histogram or summary named http_request_duration_seconds exposes three series families: _bucket, _sum and _count. A gauge takes no suffix. Never put _total, _bucket, _sum or _count on a gauge.
4. Namespace prefix. The first word says who produced it: node_ (node_exporter), go_ (Go runtime), process_ (process collector), http_ (HTTP instrumentation), prometheus_ (Prometheus itself). The prefix prevents collisions between exporters and tells the reader where to look for documentation.

Real names you will meet daily: node_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_size_bytes, http_requests_total, go_goroutines, process_resident_memory_bytes, scrape_duration_seconds, up.

Why a sysadmin cares

Naming is not cosmetics. Four production systems consume metric names directly:

• PromQL selection. rate(http_requests_total[5m]) selects by name. If half your services call the counter something else, the query silently sees half your traffic. No error is raised; the answer is simply wrong.
• Type inference by humans and tools. The _total suffix tells every reader — and every linter — that rate() is the correct function. A counter named http_requests invites someone to graph it raw and wonder why the line only ever climbs.
• Histogram plumbing. histogram_quantile(0.99, ...) only works on _bucket series. Rename the histogram and the latency panels go blank while the alert still evaluates the old name.
• Threshold units. An alert written as http_request_duration_seconds > 0.5 is meaningless against a metric exported in milliseconds. The alert fires constantly or never; both outcomes train the team to ignore it.

The operational failure is almost never a crash. It is a wrong answer delivered confidently at 02:00.

How it works

Names are written at three layers, and each layer has its own lever:

1. Instrumentation. The application or exporter source code chooses the name. Client libraries in Python and Java append _total to counters automatically; the Go client expects you to supply the full name yourself. Histograms create the _bucket, _sum and _count families from one base name.
2. The exposition. Every scrape returns the text format with # HELP and # TYPE metadata lines. The name in the exposition is what Prometheus stores.
3. Ingestion rewriting. metric_relabel_configs in the scrape config can rename or drop metrics as they arrive. This is the sysadmin’s escape hatch for an exporter you cannot patch.

A quick reference for reading names in the wild:

Good                                   Bad
node_cpu_seconds_total                 node_cpu                     (no unit, no type)
node_filesystem_avail_bytes            node_filesystem_avail_kb     (non-base unit)
http_requests_total                    httpRequestsTotal            (camelCase)
http_request_duration_seconds          http_request_duration_ms     (non-base unit)
temperature_celsius                    temp                         (vague, no unit)
queue_depth                            queue_depth_items_total      (gauge with _total)

How to configure it

You rarely “configure” a name directly; you choose exporters that follow the conventions and you normalise the ones that do not. The realistic levers in prometheus.yml:

scrape_configs:
  - job_name: legacy-app
    static_configs:
      - targets: ['10.0.0.21:8080']
    metric_relabel_configs:
      # The legacy exporter ships a camelCase counter. Normalise it
      # at ingestion so dashboards only ever know one name.
      - source_labels: [__name__]
        regex: 'httpRequestsTotal'
        target_label: __name__
        replacement: 'http_requests_total'
      # A metric nobody queries: drop it and keep the series budget.
      - source_labels: [__name__]
        regex: 'go_memstats_.*'
        action: drop

Treat ingestion-time renaming as a compatibility shim, not a design. The right fix is always in the exporter or instrumentation; the relabel rule is what you deploy while you wait for that fix. Write the team’s conventions down — a one-page instrumentation guide that says “counters end in _total, durations in _seconds, sizes in _bytes, prefix with the service name” prevents the drift that relabel rules then have to paper over.

How to validate it

Three checks, in order of increasing depth.

# 1. Lint the exposition of any exporter before it goes live.
curl -s http://localhost:9100/metrics | promtool check metrics

Illustrative output for a badly-named exporter:

requests counter metrics should have "_total" suffix
request_duration_milliseconds use base unit seconds instead of milliseconds
HttpLatency metric names should be written in snake_case not camelCase

# 2. Confirm the name arrives in Prometheus after a reload.
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=http_requests_total' | jq '.data.result | length'

# 3. Confirm the TYPE metadata is what you expect.
curl -s http://localhost:9100/metrics | grep -E '^# TYPE http_requests_total'

The outputs confirm the name, the type, and that the series flows end to end. If step 1 is clean and step 2 finds nothing, the problem is scraping or relabelling, not naming.

How it can fail

1. Same concept, different names. Two services export the same measurement under http_requests_total and requests_total. Symptom: fleet-wide dashboards undercount; an alert on one name is blind to the other service. Nothing errors.
2. Counter without the suffix. http_requests is a counter but reads as a gauge. Symptom: promtool check metrics flags it, OpenMetrics tooling normalises or rejects it, and engineers graph the raw ever-climbing line instead of a rate.
3. Unit mismatch. One exporter ships _milliseconds, another _seconds. Symptom: thresholds written for one unit are wrong by 1000x against the other; alerts either storm or sleep.
4. Values encoded in the name. http_requests_get_checkout_total embeds what should be labels (method, route) into the name. Symptom: you cannot aggregate with sum by (route); every new route creates a new metric; the only query tool left is regex over __name__.
5. Rename without dual-emission. A service update renames jobs_failed to job_failures_total. Symptom: every panel and alert on the old name shows “No data” within five minutes of the deploy; history is split across two names.
6. Exporter upgrade renames. Exporters have renamed metrics across major versions — node_exporter 0.16 renamed node_cpu to node_cpu_seconds_total, among many. Symptom: after an upgrade, recording rules referencing old names evaluate to nothing and the dashboards they feed go stale.

How to troubleshoot it

Order matters; each step halves the search space.

1. Does the exporter emit the name? curl -s http://target:9100/metrics | grep the_name. If absent, the problem is instrumentation, not Prometheus.
2. Is the exposition well-formed? Pipe it through promtool check metrics. Lint failures here predict confusion later.
3. Does Prometheus hold the series? Query the API for the name. If absent but present at the exporter, check up for the target and then your metric_relabel_configs — a rename or drop rule is the usual suspect. Compare scrape_samples_scraped against scrape_samples_post_metric_relabeling for the target to see how much relabelling removed.
4. Was it renamed recently? Query both the old and the new name over the last week. The crossover point is the deploy or upgrade that renamed it.
5. Which alerts and dashboards consume it? Grep your rules and dashboard JSON for the name before approving any rename.

Security implications

Metric names and their # HELP text are served unauthenticated on every exporter /metrics endpoint by default. The exposition discloses your internal architecture: service names, versions (go_info, prometheus_build_info, node_exporter’s node_uname_info), and which subsystems exist. Names can also leak directly — a metric named after an internal customer or a secret subsystem is public to anyone who can reach the port. Restrict exporter ports with network policy, and never encode sensitive identifiers into names or label values.

Performance implications

The name is stored once per series in the index, but it travels on every sample in every scrape body and every query response. Long, descriptive names are cheap; names with embedded values are not — they multiply series count, which is the real memory and disk driver. A rename is a churn event: the old series family goes stale and the new one is created, so a mass rename across a fleet shows up as a visible bump in head series and WAL traffic for a few hours.

Production guidance

• Adopt the upstream conventions verbatim; do not invent a local dialect. The value of the convention is that every engineer and every linter already knows it.
• Lint every exporter in CI with promtool check metrics before the scrape config referencing it is merged.
• Rename via dual-emission: ship the new name alongside the old, migrate consumers, then remove the old name in a later release.
• Keep renames out of metric_relabel_configs where possible; use relabelling as a temporary shim with a ticket attached, not as permanent plumbing.
• Prefer official, well-maintained exporters. Their names are the de-facto standard that dashboards on the internet already assume.

Verification

You should now be able to answer:

• What do the suffixes _total, _bucket, _sum and _count tell you about a metric’s type?
• Why does Prometheus require base units in names, and what breaks when two exporters disagree on milliseconds versus seconds?
• What happens to series, dashboards and alerts when a metric is renamed without dual-emission?
• Which command lints an exposition for naming violations before it reaches production?

Quiz