User Journey Metrics

A growth engineer opens the funnel dashboard at 09:00 on a Monday morning. The dashboard says: “Conversion started -> paid fell from 32 percent to 21 percent over the weekend.” The on-call engineer who picked the message up at 09:01 sees the technical counter for the checkout service is at 99.7 percent availability and p99 latency is 800 ms. There is no incident. There is, clearly, a problem.

The first hour of the investigation is spent trying to find a technical cause. The HTTP service is healthy. The database is healthy. The dependency chain is healthy. The conversion funnel is broken because the funnel is the metric that disagrees with the RED metrics. The user starts the checkout, reaches the payment page, and does not proceed. The technical counter counts the arrival; the business counter counts the completion. The counters have diverged.

This is the daily reality of user journey metrics. The funnel is the place where the business measures itself; the counter is the place where the platform engineer hears the business.

What it is

A user journey is a sequence of named stages a user passes through on the way to a business outcome. The metrics are the per-stage counts and the conversion rates between adjacent stages.

  visit           signup          activation       checkout         payment
  the             on the          on the           started          completed
  landing         platform        the platform     on the           on the
  page                                                          platform
   |               |                |                |                |
   v               v                v                v                v
 visits_total  signups_total  activations_total  orders_started_total  orders_completed_total
       \              |                |                |                  /
        \             |                |                |                 /
         \            v                v                v                /
          +-- conversion_rate{from="visit",to="signup"}                /
                       \              |                |               /
                        \             |                |              /
                         +-- conversion_rate{from="signup",to="activation"}
                                          \              |             /
                                           \             |            /
                                            +-- conversion_rate{from="activation",to="started"}
                                                            \         /
                                                             \        /
                                                              +----- conversion_rate{from="started",to="completed"}

A funnel has two fundamentals:

• Stages are counters, not gauges. A stage is an event that happens at a moment in time. The counter is the cumulative number of times the event has happened. The rate is the frequency of the event per second.
• Conversion is a ratio between two rates. It is not a counter. It is the derivative of one counter over the derivative of another. A fall in conversion is a fall in the ratio of two rates, not a fall in any single counter.

The distinction matters. A panel that shows the absolute number of orders is misleading; the number can climb while the conversion rate falls because the top of the funnel is climbing faster than the bottom. The growth team is interested in the ratio; the on-call engineer is interested in the absolute number when the absolute number is the alert.

Why a sysadmin cares

Funnel metrics are the operationally important output of every business counter. The reasons the operator should care:

1. The funnel is the alert that the executive team reads. When the CEO wants to know if the company is OK, the answer is the funnel. The operator who can produce the funnel panel during an incident is the one who earns a seat at the prioritisation meeting.
2. The funnel is the metric that disagrees with the technical metrics. The technical RED metrics say “the service is healthy”. The funnel says “the conversion is at 50 percent of its baseline”. The disagreement is the truth; the operator who only knows the technical metrics cannot resolve it.
3. The funnel is the input to capex and engineering prioritisation. The team that owns the funnel owns the prioritisation. The operator who can talk about the funnel can talk about the prioritisation.

The engineer who reads the funnel as a habit, not as a reaction to a question, is the engineer who fixes the conversion fall before the executive team notices it.

How it works

The instrumentation pattern is per-stage counters with a consistent label set across the stages. The label set is the business’s view of the user; the stage is the business’s view of the journey.

  Application                    Edge collector                 Prometheus
  ------------                   --------------                 -----------
  events emitted:                 bootstrap:                     queries:
   orders.started                 Alloy / OTel Collector         rate(stage_total[5m])
   orders.paid                    subscribes to the topic         - the per-second
   orders.completed               emits a Prometheus counter       rate of the stage
   orders.refunded                for each stage
   signup.created
   signup.activated
  emit-each-event:                 emit-each-interval:           - the canonical
   send to Kafka topic              sum the events                  business metric

The key design decision is the label set. The same labels must be present on every stage so the conversion rate can be computed by dividing the rates:

# The conversion rate from started to completed, last 5 minutes,
# per country and plan
sum(rate(orders_completed_total[5m])) by (country, plan)
  /
sum(rate(orders_started_total[5m])) by (country, plan)

The ratio is meaningful only when the labels on the numerator and denominator are identical. A label on the numerator that is not present on the denominator turns the ratio into noise.

How to configure it

The configuration is the Alloy pipeline that consumes the business events and the Prometheus scrape job that ingests the counter.

# /etc/alloy/config.alloy
# Stages: started, paid, completed, refunded.
# Each stage is its own Kafka topic; the pipeline subscribes to
# all four and emits a Prometheus counter for each.
otelcol.receiver.kafka "orders_started" {
  brokers          = ["kafka:9092"]
  topic            = "orders.started"
  encoding         = "otlp_json"
  group_id         = "alloy-orders-started"
}

otelcol.receiver.kafka "orders_paid" {
  brokers          = ["kafka:9092"]
  topic            = "orders.paid"
  encoding         = "otlp_json"
  group_id         = "alloy-orders-paid"
}

otelcol.receiver.kafka "orders_completed" {
  brokers          = ["kafka:9092"]
  topic            = "orders.completed"
  encoding         = "otlp_json"
  group_id         = "alloy-orders-completed"
}

otelcol.receiver.kafka "orders_refunded" {
  brokers          = ["kafka:9092"]
  topic            = "orders.refunded"
  encoding         = "otlp_json"
  group_id         = "alloy-orders-refunded"
}

otelcol.processor.attributes "shape" {
  actions: [
    {
      key:    "metric_name"
      value:  "stage_counter"
      action: "insert"
    },
  ]
}

otelcol.processor.transform "rename" {
  error_mode: "ignore"
  trace_statements: []
  metric_statements: [
    {
      context: "metric"
      statements: [
        # Rename the metric to the per-stage counter name. Each
        # receiver has its own pipeline; the receiver's "topic"
        # attribute is the stage name.
        'set(metric.name, "orders_" + attributes["topic"] + "_total") where attributes["topic"] != nil',
        'set(metric.description, "Count of orders reaching stage " + attributes["topic"]) where attributes["topic"] != nil',
      ]
    },
  ]
}

otelcol.exporter.prometheus "orders" {
  endpoint = "0.0.0.0:9095"
  add_metric_suffixes = false
  resource_to_telemetry_conversion = true
}

prometheus.scrape "orders" {
  targets    = ["localhost:9095"]
  job        = "orders-business"
  scrape_interval = "30s"
}

The Prometheus side then ingests the counter with the same label set as the technical counter:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: orders-business
    metrics_path: /metrics
    scrape_interval: 30s
    static_configs:
      - targets: [alloy-orders:9095]
        labels:
          pipeline: business
          source: kafka

The funnel is then a single PromQL query:

# The conversion rate from started to completed, last 5 minutes,
# per country
sum(rate(orders_completed_total[5m])) by (country)
  /
sum(rate(orders_started_total[5m])) by (country)

How to validate it

# Are the four stages all being counted?
for stage in started paid completed refunded; do
  curl -s http://localhost:9090/api/v1/query \
    --data-urlencode "query=orders_${stage}_total" \
    | jq -r '.data.result[0].value[1] // "missing"'
done
# Expected: four numeric values, all non-zero

# Is the conversion rate sane?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(orders_completed_total[5m])) / sum(rate(orders_started_total[5m]))' \
  | jq '.data.result[0].value[1]'
# Expected: a float between 0 and 1 (e.g. 0.32)

# Is the label set bounded?
curl -s http://localhost:9090/api/v1/series/ \
  --data-urlencode 'match[]=orders_completed_total' \
  | jq '.data | length'
# Expected: at most a few hundred series

# Is the collector alive?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=up{job="orders-business"}' | jq .
# Expected: up == 1

# Does the funnel agree with the RED metrics?
# Ratio of 2xx checkout to orders started, last 5 minutes
sum(rate(http_requests_total{service="checkout",status=~"2.."}[5m]))
  /
sum(rate(orders_started_total[5m]))

# A ratio close to 1.0 means the technical "2xx" arrival agrees
# with the business "started" counter. A ratio of 0.5 means half
# the 2xx arrivals are not starting a funnel.

How it can fail

1. The funnel stages use inconsistent labels. Symptom: the conversion rate query returns an empty result; the operator cannot compare countries. The fix is the label contract: every stage has the same label set.
2. The funnel is a gauge, not a counter. Symptom: the operator applies rate() to a metric that is a snapshot of the current funnel size; the rate is meaningless. The fix is to redefine the metric as a counter and emit the cumulative count, not the current value.
3. The funnel pipeline is sampled at 1 percent. Symptom: the counter is low-resolution; the conversion rate is noisy at five-minute windows; the alert fires on the noise, not the signal. The fix is to emit the counter at the full event rate, then aggregate at query time.
4. The funnel is computed across the wrong time window. Symptom: the conversion rate at 09:00 is computed over the last 24 hours; the technical incident started at 02:14 and ended at 02:51. The funnel does not show the incident. The fix is to query the funnel at the incident’s time window, not the business’s morning window.
5. The funnel counter has a user_id label. Symptom: the counter has million-series cardinality; Prometheus is at the edge of memory. The fix is the relabel rule at the collector that drops the user identifier.
6. The funnel pipeline is decoupled from the application release. Symptom: a deployment introduces a new orders.partial_refund event; the pipeline does not know about it; the funnel counter is missing the new stage. The fix is a pipeline deployment for each new event, not a silent dependency.

How to troubleshoot it

1. Is the funnel counter being scraped? up\{job="orders- business"\}. If zero, the collector is down; the funnel is stale.
2. Are all stages moving? Run the validation loop above. If one stage is missing, the upstream event is missing; the application team is the next call.
3. Is the conversion rate crossing the alert threshold? Compute the ratio and compare against the last 30 days. If the ratio has fallen, the funnel metric is the symptom.
4. Is the funnel disagreement with the technical metrics real? Compare the technical 2xx rate to the business started rate. If the two diverge, the funnel is the truth.
5. Is the cause known? Open the structured logs for the affected stage. The investigator looks for the error pattern or the change log entry that coincided with the divergence.

Security implications

• Funnel labels can leak PII. A user_id label has unbounded cardinality and is a PII leak. The cardinality contract must exclude user identifiers; the relabel rule must drop them at ingest.
• Funnel metrics expose revenue. The orders_completed_total counter, multiplied by ticket size, is revenue. The Prometheus instance that holds it should sit behind the same access controls as the underlying revenue database.
• The Kafka topic is the source of truth. If the topic has PII restriction, the collector that subscribes to it inherits the restriction. The collector security context must be reviewed.

Performance implications

The performance cost of a funnel is dominated by the cardinality of the label set, not by the number of stages:

Stages   Cardinality   Total series
  4         10             40
  4        100            400
  4      1,000          4,000
  4     10,000         40,000    <- the cliff
  4    100,000      400,000    <- redesign the labels

The right answer is to design the label set with the people who will query it, and to enforce the contract at ingest. A funnel with five high-cardinality labels is a dashboard dream and a Prometheus memory nightmare.

Verification

You should now be able to answer:

• What is the difference between a stage counter and a stage gauge, and why does the funnel care?
• Why must the per-stage labels be identical across all stages of a funnel?
• What does it mean when the funnel conversion rate falls while the technical RED metrics are healthy?
• Where should the funnel counter pipeline live, and what is the cardinality budget?
• What is the first symptom that the funnel pipeline is broken?

Quiz