Revenue Impact Metrics

A major incident starts at 14:02. The checkout service returns 5xx for 22 percent of requests. The on-call engineer restores service at 14:38. The incident review is on Friday.

The technical review says: “MTTR 36 minutes; one deployment rolled back; one customer-visible feature disabled.” The business review asks for the number. The platform engineer opens the funnel dashboard and the RED dashboard; both are green until the moment of the incident. Neither panel says “revenue lost”.

The engineer who has the answer has built a recording rule that multiplies the failed-order rate by the average ticket size, and that rule has been computing for the last 36 minutes. The engineer who does not have the answer is in the room, looking at a dashboard, trying to back-compute the number from the funnel counter and a guess at the ticket size. The room prefers the first engineer.

This is the operational value of revenue impact metrics. The number is the language the executive team speaks. The technical team that produces the number during the incident is the team that earns the right to talk about the next quarter’s budget.

What it is

Revenue impact is the financial cost of an incident, expressed in the same currency as the rest of the P&L. The arithmetic is straightforward:

  Revenue lost per minute
    = order_rate_per_minute
    * average_ticket_size
    * failure_rate

  Revenue lost per incident
    = revenue_lost_per_minute
    * duration_minutes
    * recovery_factor  (the partial recovery from
                         retries and reactivated sessions)

  Total cost per incident
    = revenue lost
    + recovery cost (engineering hours, vendor credits)
    + churn cost (customers who did not return)
    + trust cost (deferred customers, negative reviews,
                  support load)

The four components are not equally estimable. The first is arithmetic from the metrics. The second is a time entry. The third is a cohort analysis on the next month’s data. The fourth is a survey or a sentiment analysis. The only one the operator can produce during the incident is the first.

The arithmetic is more subtle than the formula suggests. The order rate is not constant; the average ticket size is not constant; the failure rate is not constant. The revenue lost per minute is a moving estimate that the recording rule updates every 15 seconds. The dashboard panel shows the running total.

Why a sysadmin cares

Three reasons the operator cares about a metric the executive team cares about:

1. The metric is the input to capex prioritisation. When two services are competing for an upgrade and the executive team has to choose, the service whose downtime is most expensive wins. The engineer who can produce the revenue-impact number wins the argument.
2. The metric is the alert that pages the executive team. A revenue-loss alert is the only alert that bypasses the on-call routing and goes directly to the leadership channel. The threshold is a financial threshold, not a latency threshold.
3. The metric is the post-mortem input. “What was the customer impact?” is the first question of every incident review. The answer is a number, not a story. The engineer who has the number writes the post-mortem in half the time.

None of these reasons justify a dashboard that nobody reads. The metric earns its place when the on-call engineer uses it during the incident, not when the leadership team discovers it in the next quarter’s review.

How it works

The recording rule that multiplies the three inputs and produces the running total is the operational core of the metric. The inputs are counters, the output is a gauge, and the recording rule evaluates every 15 seconds.

  inputs                                       output
  --------------------------------------------- --------------
  orders_started_total{plan, country}    *     revenue_lost_usd
  ticket_size_usd{plan, country}                (gauge, sum over
  failure_rate{plan, country}                    labels, per
                                                   scrape)

The failure rate is derived from the funnel:

  failure_rate = 1 - (orders_completed_total / orders_started_total)

The ticket size is the average revenue per completed order, computed as a recording rule over the last 24 hours:

  ticket_size_usd{plan, country}
    = revenue_captured_usd{plan, country}
      / orders_completed_total{plan, country}

The revenue lost per minute is the product of the three, and the running total is the integral over the incident window.

How to configure it

The configuration is three recording rules, one for the failure rate, one for the ticket size, and one for the revenue lost per minute. The fourth rule, the running total, is the dashboard panel.

# /etc/prometheus/rules/revenue_impact.yml
groups:
  - name: revenue_impact
    interval: 15s
    rules:
      # Stage 1: failure rate per stage transition (canonical
      # shape: started -> completed). The ratio is bounded 0..1.
      - record: failure_rate:orders_started_to_completed:ratio
        expr: |
          1 - (
            sum by (plan, country) (rate(orders_completed_total[5m]))
            /
            sum by (plan, country) (rate(orders_started_total[5m]))
          )

      # Stage 2: ticket size per plan, per country, 24-hour window.
      # The window is long enough to smooth the per-order variance;
      # short enough to refresh each day.
      - record: ticket_size_usd:revenue_per_completed_order:24h
        expr: |
          sum by (plan, country) (increase(revenue_captured_usd[24h]))
          /
          sum by (plan, country) (increase(orders_completed_total[24h]))

      # Stage 3: revenue lost per minute, per plan, per country.
      - record: revenue_lost_usd_per_minute:impact
        expr: |
          sum by (plan, country) (rate(orders_started_total[5m]))
          *
          avg by (plan, country) (ticket_size_usd:revenue_per_completed_order:24h)
          *
          failure_rate:orders_started_to_completed:ratio

      # Stage 4: the running total over the incident window. The
      # dashboard panel computes this in Grafana with
      # $__range * avg_over_time(), so a separate rule is not
      # required.

The dashboard panel is then a single query:

# The running total of revenue lost over the dashboard time
# range. The factor 1/60 converts the per-minute rate to a
# total over the range.
sum(revenue_lost_usd_per_minute:impact) * ($__range / 60)

The number that the panel shows is the answer the executive team wants.

How to validate it

# Is the failure rate in the expected range?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=failure_rate:orders_started_to_completed:ratio' \
  | jq '.data.result[0].value[1]'
# Expected: a float between 0 and 1; baseline is whatever the
# historical steady-state is.

# Is the ticket size sensible?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=ticket_size_usd:revenue_per_completed_order:24h' \
  | jq '.data.result[0].value[1]'
# Expected: a positive float, comparable to the product catalogue
# average.

# Is the revenue lost per minute computing?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=revenue_lost_usd_per_minute:impact' \
  | jq '.data.result[0].value[1]'
# Expected: a non-negative float, zero when the funnel is healthy.

# During an incident, the running total over the last 30 minutes
sum(revenue_lost_usd_per_minute:impact) * 30

# When the incident is over and the failure rate has returned
# to baseline, the running total stops growing.

# Is the recording rule evaluating on the expected interval?
curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[] | select(.name == "revenue_impact") | .interval'
# Expected: 15s

How it can fail

1. The ticket size is computed over a too-short window. Symptom: the ticket size oscillates with the per-minute order mix; the revenue lost per minute is noisy at 15-second windows. The fix is a longer window (24 hours) for the ticket size recording rule.
2. The ticket size is the wrong ticket size. Symptom: the team has a plan of “pro” and “enterprise”, but the ticket size is averaged across both. The revenue lost per minute is wrong by an order of magnitude. The fix is to compute the ticket size per plan and per country.
3. The failure rate is computed with a too-short window. Symptom: the failure rate spikes during a one-minute scrape blip and the revenue lost per minute spikes with it. The fix is a 5-minute window for the failure rate, not a 1-minute window.
4. The revenue lost per minute is exported as a cumulative counter, not a rate gauge. Symptom: the next incident review tries to subtract the previous incident’s value and gets a meaningless number. The fix is to make the metric a gauge with the per-minute rate, and compute the running total in the dashboard.
5. The recording rule is missing the sum by (plan, country). Symptom: the result is a single series with no labels; the dashboard cannot slice by country. The fix is to add the sum by (plan, country) and re-test.
6. The revenue impact number is used in a public post-mortem before the data is verified. Symptom: the number is wrong by a factor of 10; the post-mortem is reprinted; the executive team publicly walks back the figure. The fix is to publish the methodology alongside the number.

How to troubleshoot it

1. Is the failure rate recording rule evaluating? Check /api/v1/rules and confirm the rule is in the revenue_impact group.
2. Is the ticket size recording rule evaluating? Check the same endpoint; confirm the rule’s last evaluation time.
3. Are the orders counters still incrementing? up\{job="orders- business"\}.
4. Is the revenue lost per minute matching the manual estimate? Compute the product manually for a 5-minute window; compare. A discrepancy of more than 10 percent means one of the inputs is wrong.
5. Is the recording rule interval matching the scrape interval? A 15-second recording rule on a 30-second scrape produces a recording rule that evaluates on stale data. Match the intervals.

Security implications

• Revenue impact is a sensitive number. The revenue_lost_usd_per_minute metric is a financial figure. The Prometheus instance that holds it should sit behind the same access controls as the underlying revenue database.
• The ticket size metric is sensitive. A ticket_size_usd label set across plan and country is implicitly a price book. The metric should be view-only to the on-call team and write-only to the recording rule.
• The post-mortem is a public document. The revenue impact number published in the post-mortem is the figure the press reproduces. The methodology must be in the document, not in a separate email.

Performance implications

The recording rule is cheap. The dominant cost is the cardinality of the output series, which is the same as the input series. A retailer with 50 plans and 30 countries has 1,500 series for the failure rate, the ticket size, and the revenue lost per minute. That is a small TSDB; the rule evaluator on a single Prometheus 2.55.x host completes the evaluation in a few milliseconds.

The trade-off is the recording rule interval. A 15-second interval matches the scrape interval and produces a real-time panel. A 5-minute interval smooths the panel but loses the granularity the on-call engineer needs to see the recovery moment. The choice is operational, not technical.

Verification

You should now be able to answer:

• What is the formula for revenue lost per minute, and what units are the three inputs?
• What is the difference between gross revenue impact and net revenue impact, and why does the post-mortem need both?
• Why should the ticket size recording rule use a 24-hour window while the failure rate recording rule uses a 5-minute window?
• What is the right metric type for the revenue lost per minute: counter, gauge, or histogram?
• What is the first symptom that the revenue impact recording rule is broken?

Quiz