Custom Metrics Responsibly

A checkout service is producing RED metrics. The on-call can answer “is the service slow?” and “is the dependency slow?” in thirty seconds. The question the on-call cannot answer is “is the business healthy?” — the conversion rate, the cart abandonment rate, the payment failure rate by category. The RED metrics are the system; the business metrics are the outcome.

The custom metric is the discipline that bridges the two. The discipline is not “add a metric for every domain event.” The discipline is “add a metric only when the metric earns its place in the dashboard, the alert, or the runbook.” The team that follows the discipline has a Grafana that fits on one screen; the team that does not has a wall of orphan panels that no one trusts.

What it is

A custom metric is a metric that the application emits in addition to the RED, USE, and dependency metrics. Custom metrics are business metrics — the counters and histograms that describe the outcome of the service, not the runtime of the service.

The four common shapes are:

1. Business counters — counts of business events: checkout_attempt_total{status="success"}, signup_completed_total{plan="pro"}, payment_processed_total{currency="eur"}.
2. Business histograms — distributions of business values: cart_value_eur_bucket, order_processing_seconds_bucket.
3. Business gauges — current state of a business resource: cart_items_pending_count, inventory_level_units{item="sku123"}.
4. SLO indicators — the underlying metric of an SLO: slo:checkout_latency_p95:burn_rate_5m.

The discipline that distinguishes responsible custom metrics from irresponsible ones is bounded cardinality. Every label must have a bounded value set; every metric must have an owner; every metric must be consumed by at least one dashboard or alert.

The canonical reference for the discipline is the Prometheus metric and label naming guide and the OpenTelemetry Metrics API. The two together describe the contract of a custom metric: the name, the type, the unit, the labels, the owner.

Why a sysadmin cares

The single biggest predictor of a Grafana instance that breaks under load is the cardinality of the custom metric set. The team that has 10,000 custom metrics at 10 labels each is running a Prometheus that uses 100 GB of memory. The team that has 100 custom metrics at 5 labels each is running a Prometheus that uses 5 GB.

Three operational problems disappear when custom metrics are responsible:

1. The “Prometheus is OOMKilled” problem. A unbounded label on a single metric can grow the series count by 10x. The Prometheus memory follows. The fix is the bounded label; the prevention is the cardinality review.
2. The “no one owns this metric” problem. A metric that no one consumes is a metric that no one maintains. The owner is the discipline that keeps the metric from rotting.
3. The “duplicate metric for the same thing” problem. Two services that emit checkout_total with different labels produce two panels that the operator cannot trust. The contract is the discipline that prevents the duplication.

How it works

The mental model is the contract. A custom metric has six fields:

  custom metric contract
  +-- name        -- checkout.completed.count
  +-- type        -- counter
  +-- unit        -- 1
  +-- description -- Count of completed checkouts
  +-- labels      -- { payment_method, currency, region }
  +-- owner       -- team: payments

The contract is published before the metric is emitted. The contract is reviewed by the platform team. The metric is instrumented with the OpenTelemetry SDK. The metric is validated in staging before the production rollout.

The cardinality budget is the constraint. The platform typically allocates a per-team cardinality budget (e.g. 100,000 series per team). The contract’s labels must respect the budget. The platform team runs a cardinality review against the staging Prometheus and rejects contracts that exceed the budget.

The lifecycle is the discipline:

  define  ->  instrument  ->  validate  ->  retire

The retire phase is the most often missed. A metric that is no longer consumed should be removed from the SDK. The metric continues to consume the cardinality budget until it is removed.

Under the hood

How to configure it

The custom metric is configured in three places: the SDK at service startup, the Collector pipeline, and the Prometheus alerting / recording rule.

1. The SDK at service startup

A Python service with OpenTelemetry 0.110.x emitting a custom business metric:

# app.py -- custom business metric
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import (
    ExplicitBucketHistogramAggregation,
    View,
    DropAggregation,
)
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "checkout",
    "service.version": "1.4.2",
})

# SEVERITY: CONFIGURATION -- the views are the contract
views = [
    # Drop the user_id attribute on every metric
    View(
        instrument_name="*",
        attribute_keys={"user_id"},
    ),
    # Drop the order_id attribute on the order latency
    View(
        instrument_name="order.processing.duration",
        attribute_keys={"order_id"},
    ),
    # Custom bucket boundaries for order.processing.duration
    View(
        instrument_name="order.processing.duration",
        aggregation=ExplicitBucketHistogramAggregation(
            boundaries=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
        ),
    ),
]

provider = MeterProvider(
    resource=resource,
    metric_readers=[reader],
    views=views,
)
metrics.set_meter_provider(provider)

meter = metrics.get_meter("checkout", "1.4.2")

# Custom business counter
checkout_completed = meter.create_counter(
    "checkout.completed.count",
    unit="1",
    description="Count of completed checkouts",
)

# Custom business histogram
order_duration = meter.create_histogram(
    "order.processing.duration",
    unit="s",
    description="Wall-clock duration of order processing",
)

# In the checkout handler, after the charge succeeds:
checkout_completed.add(1, {
    "payment_method": payment.method,
    "currency": payment.currency,
    "region": customer.region,
})

The two views enforce the contract. The user_id and order_id attributes are dropped. The order.processing .duration histogram has explicit bucket boundaries that match the service’s latency range.

The payment_method label is bounded: card, wallet, bank_transfer. The currency label is bounded: eur, usd, gbp. The region label is bounded: eu-west-1, us-east-1, us-west-2. The cardinality is 3 * 3 * 3 = 27 series — well within the per-team budget.

2. The Collector pipeline

The Collector enforces the cardinality budget as a runtime invariant:

# /etc/otelcol-contrib/config.yaml
processors:
  attributes/custom_drop:
    actions:
      - key: user_id
        action: delete
      - key: order_id
        action: delete
      - key: session_id
        action: delete
      - key: request_id
        action: delete
  batch:
    timeout: 10s

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
    resource_to_telemetry_conversion:
      enabled: true

service:
  pipelines:
    metrics:
      processors: [attributes/custom_drop, batch]
      exporters: [prometheusremotewrite]

The four delete actions are the safety net. Even if the SDK view is missing, the Collector drops the unbounded attributes.

3. The Prometheus alerting and recording rule

The custom metric is consumed by exactly one alert and one dashboard panel. The contract is published in the runbook.

# /etc/prometheus/rules/checkout.rules
groups:
- name: checkout_business
  interval: 30s
  rules:
  # Recording rule for the business rate
  - record: checkout:completed:rate5m
    expr: |
      sum by (payment_method, currency, region) (
        rate(checkout_completed_count[5m])
      )

  # Alert on the conversion rate
  - alert: CheckoutConversionDrop
    expr: |
      checkout:completed:rate5m
        /
      on(region) group_left()
        checkout:attempt:rate5m
      < 0.6
    for: 30m
    labels:
      severity: ticket
      team: payments
    annotations:
      summary: 'Checkout conversion rate below 60% for 30 minutes'
      runbook_url: 'https://runbooks.example.com/checkout/conversion'

The alert is on the conversion rate — the ratio of completed checkouts to attempts. The 30-minute for: filters transient dips. The severity: ticket label routes it to the working-hours backlog.

The promtool validator confirms the rule syntax:

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/checkout.rules

Expected output:

SUCCESS: /etc/prometheus/rules/checkout.rules

How to validate it

Validate that the custom metric is live with three checks.

1. The metric is emitted and the cardinality is bounded.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=count%20by%20(payment_method)%20(checkout_completed_count)'

Expected output:

payment_method
card             1
wallet           1
bank_transfer    1

Three values, one series each. The cardinality is bounded.

2. The conversion rate is computed.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=checkout:completed:rate5m'

Expected output:

payment_method, currency, region
card, eur, eu-west-1     85.3
card, usd, us-east-1     72.1
wallet, eur, eu-west-1   12.4

The conversion rate is per payment method, currency, and region. The dashboard joins the three dimensions.

3. The label budget is respected.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=count(checkout_completed_count)'

Expected output:

27

The total series count is 27 — 3 payment_method * 3 currency * 3 region. The platform’s per-team budget is 100,000 series; the metric consumes 27.

How it can fail

Four specific failure shapes appear regularly in custom metric instrumentation:

1. Unbounded cardinality. The developer adds a user_id label to the custom metric. The metric emits one series per user. The Prometheus memory climbs; the dashboard loads slowly; the cardinality review rejects the metric. Symptom: count(checkout_completed_count) grows by tens of thousands per hour.
2. Missing owner. The metric is emitted by the checkout service. The dashboard is owned by the analytics team. The alert is owned by the payments team. When the metric breaks, no one fixes it. The metric is orphaned. Symptom: the metric is present but the alert is silent.
3. Duplicated semantics. Two services emit checkout_total with different labels. The dashboards show two different values. The on-call does not know which to trust. The contract is not enforced. Symptom: two panels with the same name but different values.
4. Orphan metrics. The metric is emitted but no dashboard or alert consumes it. The metric consumes the cardinality budget without contributing to the investigation. Symptom: the metric is present in the Prometheus but no panel references it.

How to troubleshoot it

When a custom metric says something the operator does not believe, the diagnostic order is:

1. Confirm the contract is published. The metric must have a name, type, unit, description, labels, and owner. The contract is the source of truth.
2. Confirm the cardinality is bounded. Run count(metric_name) and confirm the series count is under the per-team budget. If the count is unbounded, the labels are leaking.
3. Confirm the metric is consumed. Run promtool query series and search for the metric in the alerts and dashboards. If no panel or alert references the metric, the metric is an orphan.
4. Confirm the SDK view is enforced. The view must drop the unbounded attributes. The attributes/custom_drop Collector processor is the safety net.
5. Confirm the alert is wired. The metric should have at least one alert. The alert should fire when the metric is broken.

Security implications

The custom metric is the highest-risk part of the metric set. The labels can leak:

• User identifiers — user_id, session_id, request_id are unbounded and may contain PII. The view should drop these attributes.
• Request payloads — db.statement, url.full contain the full request. The Collector should drop these attributes.
• Authentication tokens — auth_token is occasionally added as a label. The view should drop this attribute.

The custom metric should be reviewed for sensitivity before the cardinality review. The sensitivity review is the human check; the cardinality review is the machinery check.

The custom metric should be restricted to operators with a recorded purpose. The combination of metric name, labels, and dashboard is enough to identify the service’s workload shape.

Performance implications

The custom metric is cheap to emit. The counter is a single increment per event. The histogram is a single record per event. The OTLP exporter serialises the data every 10 seconds.

The platform cost is the cardinality. A metric with N labels and K values per label emits K^N series. A metric with 5 labels and 10 values per label emits 100,000 series. The Prometheus memory follows.

The on-call cost is the dashboard. The custom metric panel is a single value per series. The dashboard query is cheap.

The cardinality review is the lever. The platform’s per-team budget is the constraint. The contract is the implementation.

Production guidance

• The custom metric has a contract. Name, type, unit, description, labels, owner. The contract is published before the metric is emitted.
• The cardinality is bounded. Every label has a bounded value set. The view enforces the bound. The Collector is the safety net.
• The metric is consumed. Every metric has at least one dashboard or alert. The orphan metric is the failure shape.
• The metric has an owner. The owner is paged when the metric breaks. The owner is the discipline that prevents the orphan.
• The metric is retired. The metric is removed from the SDK when it is no longer consumed. The retire phase is the most often missed.
• The cardinality review is in CI. The pre-deploy check runs count(metric_name) against the staging Prometheus and rejects contracts that exceed the budget.

Verification

You should now be able to answer:

• What are the six fields of a custom metric contract, and which one is most often missed?
• What is the cardinality trap of a custom metric, and how is the trap prevented at the SDK and the Collector?
• Why is the owner field the most important one in the contract, and what does an orphan metric look like?
• What is the lifecycle of a custom metric, and why is the retire phase the most often missed?
• Where does the cardinality review run in CI, and what is the pre-deploy check?

Quiz