Business vs Observability Metrics

A checkout service degrades at 02:14. CPU on the order host climbs to 78 percent; the 5xx error rate climbs to 4 percent; p99 latency on the /checkout route crosses 1.8 s. Every technical SLO is amber. The on-call engineer pages out, the team convenes, the incident is logged. The work is real.

At 02:51 the head of growth messages the same engineer: “Refunds are spiking. Are we eating a chargeback wave?” Nobody on the incident call has a panel that answers the question. The funnel counter that should report completed checkouts is sampled, not continuous; the revenue counter is a daily ETL job that runs at 06:00. By the time the team could answer the question, the chargeback has already arrived at the payment processor.

This is the gap between technical observability and business metrics. The first describes the system. The second describes the business the system serves. A platform that only has the first leaves the second question to guesswork and to a different team that probably uses a different tool.

What it is

Technical observability metrics are the RED/USE numbers that describe a running system: request rate, error rate, duration, utilisation, saturation, errors. They are property of the service. The service team owns them; the on-call engineer reads them to investigate.

Business metrics are the numbers that describe the outcome of the service: orders started, orders completed, refunds issued, revenue captured, active subscriptions, churn events. They are property of the business. The product team, the finance team, and the growth team read them; the on-call engineer reads them when a business impact statement is needed in an incident.

The two answer different questions. They share the same platform because the only way to correlate a business event with the technical event that caused it is to keep them in the same TSDB.

    Technical observability               Business metrics
    ----------------------------           ----------------------------
    "How is the checkout svc?"            "Are we losing money?"
    HTTP rate, error rate, latency        orders_started{...}
    p99, CPU, memory, I/O                 orders_completed{...}
                                          revenue_captured_usd{...}
                                          refunds_issued_usd{...}
    Owned by: service team                Owned by: product / growth
    Cadence: per-second, per-scrape       Cadence: per-event, per-minute
    Cardinality: low (handful of labels)  Cardinality: high risk
    Time horizon: hours, days             Time horizon: days, months
                                          often also real-time

Why a sysadmin cares

Three reasons get the operator’s attention in a way the product manager’s slide does not:

1. Incidents need a business-impact line. Every major incident review asks “what was the customer impact?” The answer is a number from the business counter, not a guess from the technical counter. The team that has the answer wins the review.
2. A business counter is the only signal that aligns SLO work with revenue. A 99.9% latency SLO on a route that nobody purchases through is invisible to the business. A 99.0% SLO on the checkout route is a revenue decision. The counter is the evidence.
3. The business counter is the alert that pages the CEO. When the executive team wants a single panel, it is the funnel counter, not the CPU counter. The operator who can produce that panel during an incident is the one who gets a seat at the prioritisation meeting.

None of this is the on-call engineer’s job description. All of it is the on-call engineer’s actual job. The platform that supports both is the platform the operators build.

How it works

The mental model is two axes crossing:

    System-oriented                          User-oriented
    ----------------------------------------------------------------
    |                                                              |
    |   RED/USE metrics                Funnel metrics              |
    |   http_requests_total            orders_started              |
    |   http_request_duration_seconds  orders_completed            |
    |   node_cpu_seconds_total         orders_refunded             |
    |                                                              |
    |   ----------------------------------------------------------------
    |                                                              |
    |   Resource metrics                Outcome metrics             |
    |   node_memory_Active_bytes       revenue_captured_usd        |
    |   kube_pod_cpu_usage             subscriptions_active        |
    |   disk_io_now                    churn_events                |
    |                                                              |
    ----------------------------------------------------------------
    Short retention (hours)                Long retention (months)
    + dashboard + alert                   + analytics + report

The left half is what the platform engineer works on. The right half is what the business works on. The intersection is where incident impact lives.

The technical question “is the system healthy?” is not the same as the business question “is the business healthy?”. A system can be at 100% availability and the business can be at zero revenue because the system is serving a product that does not sell. A business can be at peak revenue and the system can be at 100% capacity waiting for a queue that has been growing for nine minutes. The two questions are orthogonal.

How to configure it

The configuration is two halves: a Prometheus scrape job for the technical metrics that already exist, and an OpenTelemetry Collector or Grafana Alloy pipeline that turns the business event stream into a Prometheus counter.

# /etc/prometheus/prometheus.yml
# Technical metrics: the application exposes its /metrics endpoint
# and Prometheus scrapes it. No change from the standard pattern.
scrape_configs:
  - job_name: checkout-service
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets:
          - checkout-service:8080
        labels:
          service: checkout
          tier: api

The business stream is different. The order events live in a Kafka topic; the application emits them as part of normal processing. The collector summarises them and emits a Prometheus counter:

# /etc/alloy/config.alloy
# Grafana Alloy pipeline: consume the orders.completed Kafka topic,
# aggregate per minute, and expose the counter for Prometheus.
otelcol.receiver.kafka "orders" {
  brokers          = ["kafka:9092"]
  topic            = "orders.completed"
  encoding         = "otlp_json"
  group_id         = "alloy-orders-aggregator"
}

otelcol.processor.attributes "order_shape" {
  actions: [
    {
      key:            "metric_name"
      action:         "insert"
      value:          "orders_completed_total"
    },
    {
      key:            "metric_description"
      action:         "insert"
      value:          "Count of completed orders, per plan and country."
    },
  ]
}

otelcol.exporter.prometheus "orders" {
  endpoint         = "0.0.0.0:9095"
  add_metric_suffixes = false
  resource_to_telemetry_conversion = true
}

otelcol.auth.basic "scrape" {
  username = sys.env("ALLOY_SCRAPE_USER")
  password = sys.env("ALLOY_SCRAPE_PASSWORD")
}

prometheus.scrape "orders" {
  targets = ["localhost:9095"]
  job     = "orders-business"
  scrape_interval = "30s"
}

The Prometheus side then adds the business counter to the same TSDB as the technical counter:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: orders-business
    metrics_path: /metrics
    scrape_interval: 30s
    static_configs:
      - targets:
          - alloy-orders:9095
        labels:
          pipeline: business
          source: kafka

The result is one TSDB that holds both the technical counter http_requests_total{service="checkout"} and the business counter orders_completed_total{plan="pro",country="GB"}. The correlation is a single PromQL query.

How to validate it

# Is the technical counter still being scraped?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=up{job="checkout-service"}' | jq .

# Expected:
# { "status": "success", "data": { "resultType": "vector",
#   "result": [ { "metric": {"job": "checkout-service", "instance": "..."},
#                "value": [ ..., "1" ] } ] } }

# Is the business counter showing up?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=orders_completed_total' | jq .
# Expected: a non-empty vector

# Is the business counter *moving*?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=rate(orders_completed_total[5m])' | jq .
# Expected: a non-zero float

# Is the label cardinality bounded?
curl -s http://localhost:9090/api/v1/series/ \
  --data-urlencode 'match[]=orders_completed_total' \
  | jq '.data | length'
# Expected: at most a few hundred series, not millions

# Sanity check: are the two worlds talking to the same time?
# A technical RED metric vs the business counter, last 1 hour
sum(rate(http_requests_total{service="checkout",status=~"2.."}[5m]))
  / on() group_left()
sum(rate(orders_completed_total[5m]))

# A ratio close to 1.0 means the technical "2xx" counter and the
# business "completed" counter agree on scale. A ratio of 10 means
# one of them is wrong by an order of magnitude.

How it can fail

1. Business metrics live in a different system. Symptom: the on-call engineer asks “how many orders failed in the last 10 minutes?” and is told to ask the analytics team. The analytics team’s answer arrives in 24 hours. The incident is over; the post-mortem has no business impact line. The fix is one platform.
2. High-cardinality labels on a business counter. Symptom: orders_completed_total{user_id="..."} with ten million series. Prometheus OOMs at the head block; the WAL replay takes 45 minutes on restart. The label set was inherited from the event schema and never bounded. The fix is the label contract: cardinality is owned by the platform team, not the event schema.
3. Business counter dropped because the collector pod was scaled to zero. Symptom: a Kafka lag alert fires, but the orders_completed_total counter stops incrementing for 20 minutes while the collector restarts. The fix is a minimum replica count of one on the business pipeline, with a PDB.
4. Time mismatch between the technical counter and the business counter. Symptom: the technical counter reports the failure at 02:14; the business counter reports the lost revenue at 02:51. The 37-minute gap is the difference between an incident and a chargeback. The fix is event-time, not ingest-time, on the business counter.
5. A business counter that no-one owns. Symptom: a panel shows a rate of zero. Nobody knows whether the counter is broken or whether the business is genuinely at zero. The fix is the owner field on the dashboard and the runbook entry on the metric.
6. The technical counter overstates the business outcome. Symptom: HTTP 2xx rate is 99.7 percent; orders completed is 94 percent. The difference is users who reached the 2xx page and then abandoned the funnel. The technical team celebrates the SLO; the business team declares a revenue incident. The fix is to define the SLO on the business counter, not the HTTP counter.

How to troubleshoot it

1. Is the business counter being scraped? Check up{job=...} for the business job. If up==0, the collector is down; the counter is stale.
2. Is the counter still moving? Compute rate(orders_completed_total[5m]). If zero, the upstream stream is silent; check the Kafka consumer lag.
3. Is the label set bounded? Run count by(__name__)(orders_completed_total) and compare against the documented cardinality budget. If the budget is exceeded, the next scale-out is the failure.
4. Is the time alignment correct? Compare the business counter’s last sample time to the equivalent technical counter. If they lag by more than a minute, the pipeline is the bottleneck.
5. Is the business counter the right counter? Check the definition against the product spec. A counter that “completed” = “paid” is different from a counter that “completed” = “delivered”. The team that disagrees with the definition is the team that disagrees with the incident.

Security implications

• Business counters can leak PII. A counter labelled orders_completed_total{user_id="..."} is a PII leak. The cardinality contract must exclude user identifiers; the Prometheus relabel_configs should drop the label at ingest if the contract is violated.
• Business counters expose revenue. The revenue_captured_usd counter is a sensitive number. The Prometheus instance that holds it should sit behind the same access controls as the underlying revenue database. Grafana teams with finance-team-only access, not org-wide.
• The collector pipeline is a write path. A misconfigured Alloy that allows remote writes from arbitrary sources is a metric injection vulnerability. The prometheus.scrape job should restrict the source by network and by basic auth.

Performance implications

The dominant cost variables are the same as for any other Prometheus counter, but the budget is tighter:

Technical counter cardinality   bounded by service's label space
                                (usually < 10k series per service)

Business counter cardinality    bounded by business's label space
                                (usually < 50k series; lower is
                                better; 100k is the cliff)

Business counter retention      longer than technical (months vs
                                days) because the funnel is
                                what the growth team reports on

Pipeline ingest                 business counters often come
                                through a collector, not a
                                direct application scrape; the
                                collector is the bottleneck

The trade-off is between label richness and query speed. A business counter with five high-cardinality labels is a dashboard dream and a Prometheus memory nightmare. The right answer is to design the label set with the people who will query it, and to enforce the contract at ingest.

Verification

You should now be able to answer:

• What is the operational difference between a technical observability metric and a business metric?
• Why does a business counter need a cardinality contract that is independent of the event schema?
• Where should the business counter pipeline live: in the application, in a collector, or in a warehouse?
• Who owns the label set of a business counter?
• What is the first symptom that the business counter pipeline is broken?

Quiz