A/B Test Metrics

A growth engineer ships an experiment at 09:00 on Tuesday. The hypothesis: a new checkout button copy will increase the conversion rate from 32 percent to 35 percent. The experiment is configured at the edge, the variant is assigned at the session, and the conversion event is logged at the success page. The experiment is set to run for 14 days.

By 09:23 on Tuesday, the dashboards show the new variant at a conversion rate of 41 percent. The on-call engineer is paged because the metric has crossed the alert threshold. The growth engineer is paged because the result is “too good to be true”.

The 41 percent is a sample ratio mismatch. The variant assignment service is sending 80 percent of new sessions to the treatment arm and 20 percent to the control. The conversion rate of the treatment arm is inflated by the over-representation of high-intent users. The experiment is not a winner; the assignment is broken.

The right response is to debug the variant assignment, not to declare the experiment a win. The team that has the live counter and the confidence to call the metric wrong is the team that does not ship a broken experiment to production.

What it is

A/B test observability is the live monitoring of an experiment as it runs. The metrics are:

• Exposure — the number of sessions that have been assigned to a variant. The counter is the ground truth for “did the variant receive traffic?”
• Conversion — the number of sessions that completed the target action, per variant. The counter is the ground truth for “did the variant produce the outcome?”
• Guardrail metrics — the technical RED metrics per variant. The counters confirm that the variant is not degrading the service.

These are not the analysis metrics. The analysis metrics are p-value, confidence interval, minimum detectable effect, and expected loss. Those are the output of a statistics engine, not a Prometheus query.

The two workflows are intentionally separate:

  Live monitoring (Prometheus + Grafana)       Analysis (stats engine)
  ----------------------------------------     ------------------------
  exposure counter per variant                 expected sample size
  conversion counter per variant               p-value, confidence interval
  guardrail counters per variant               minimum detectable effect
  live panel every 15 s                        final report at 14 days
  alert on exposure imbalance                  stop on p < 0.05 + sample size
  alert on guardrail regression                reject on guardrail miss

The reason the two are separate is that A/B test analysis is a statistics discipline, not a metrics discipline. The signals Prometheus produces are the inputs to the analysis; the analysis itself is a statistics engine that knows how to account for sample size, multiple comparisons, novelty effects, and peeking.

Why a sysadmin cares

Three reasons the operator cares about experiments:

1. The experiment is a production deployment. A new variant is a code path that real users are running. The same observability discipline applies: does the variant hit the SLO, does the error rate climb, does the latency regress? The guardrail metrics are the operator’s signal.
2. The experiment is a label on the RED metrics. A new variant is a new value of the variant label on every RED counter. The label is a free cardinality axis; the team that does not bound it produces a memory cliff.
3. The experiment is the canary. Sometimes the experiment is the canary. The team that uses the experiment counters as the production canary signal is the team that catches the regression before the rollout completes.

The operator who treats the experiment as a label, not as a feature, is the operator who handles the cardinality discipline correctly.

How it works

The instrumentation pattern is two counters per stage plus a variant label. The label set is the union of the experiment identifier and the variant identifier.

  Counter                              Labels
  -----------------------------------  --------------------------------
  experiment_exposure_total            experiment_id, variant
  experiment_conversion_total          experiment_id, variant
  http_requests_total                  variant (if present)
  http_request_duration_seconds        variant (if present)

The variant label is what makes the existing RED metric sliceable by experiment. The label needs to be added to the RED metrics before the experiment starts, by the application team, and agreed in the experiment design doc.

The conversion counter is the business counter for the experiment’s target action. The counter is the same shape as the funnel counter from the previous lesson, with the variant label added.

The live monitoring panel is then two queries:

# Exposure per variant, last 5 minutes
sum by (variant) (rate(experiment_exposure_total[5m]))

# Conversion rate per variant, last 5 minutes
sum by (variant) (rate(experiment_conversion_total[5m]))
  /
sum by (variant) (rate(experiment_exposure_total[5m]))

The panel also shows the cumulative exposure and the cumulative conversion rate:

# Total exposure per variant, since experiment start
sum by (variant) (experiment_exposure_total)

# Cumulative conversion rate per variant
sum by (variant) (experiment_conversion_total)
  /
sum by (variant) (experiment_exposure_total)

The cumulative view is the canary. The cumulative exposure ratio between variants should be approximately 50/50 for a two-arm experiment. A 60/40 ratio is a sample ratio mismatch; the assignment is broken.

How to configure it

The configuration is the experiment counter pipeline plus the Prometheus scrape job.

# /etc/alloy/config.alloy
# Grafana Alloy pipeline: consume the experiment exposure and
# conversion events from Kafka, and emit Prometheus counters.
otelcol.receiver.kafka "experiment_exposure" {
  brokers          = ["kafka:9092"]
  topic            = "experiment.exposure"
  encoding         = "otlp_json"
  group_id         = "alloy-experiment-exposure"
}

otelcol.receiver.kafka "experiment_conversion" {
  brokers          = ["kafka:9092"]
  topic            = "experiment.conversion"
  encoding         = "otlp_json"
  group_id         = "alloy-experiment-conversion"
}

otelcol.processor.attributes "experiment" {
  actions: [
    {
      key:    "metric_name"
      value:  "experiment_event_total"
      action: "insert"
    },
  ]
}

otelcol.processor.transform "label_renames" {
  error_mode: "ignore"
  trace_statements: []
  metric_statements: [
    {
      context: "metric"
      statements: [
        'set(metric.name, "experiment_" + attributes["event_type"] + "_total") where attributes["event_type"] != nil',
      ]
    },
  ]
}

otelcol.exporter.prometheus "experiment" {
  endpoint = "0.0.0.0:9095"
  add_metric_suffixes = false
  resource_to_telemetry_conversion = true
}

prometheus.scrape "experiment" {
  targets    = ["localhost:9095"]
  job        = "experiment-business"
  scrape_interval = "30s"
}

The application side, for the RED metrics with the variant label, is a label propagation in the request context:

# Python example: prometheus_client middleware that adds the
# variant label to every RED counter.
from prometheus_client import Counter, Histogram
from flask import request, g

REQUESTS = Counter(
    "http_requests_total",
    "HTTP requests",
    ["method", "status", "handler"],
)

LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency",
    ["method", "status", "handler"],
)

@app.before_request
def _capture_variant():
    # The variant is set by the edge / feature-flag service
    g.variant = request.headers.get("X-Experiment-Variant", "control")

@app.after_request
def _record(response):
    REQUESTS.labels(
        method=request.method,
        status=response.status_code,
        handler=request.url_rule.rule if request.url_rule else "unknown",
    ).inc()
    return response

The variant label is added at the application layer; the cardinality cost is bounded by the cardinality contract for the RED metrics.

How to validate it

# Is the exposure counter being scraped?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=experiment_exposure_total' \
  | jq '.data.result | length'
# Expected: at least 2 series (control + treatment)

# Is the conversion counter being scraped?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=experiment_conversion_total' \
  | jq '.data.result | length'
# Expected: at least 2 series

# Is the exposure ratio between variants sane?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (variant) (experiment_exposure_total)' \
  | jq '.data.result[] | {variant: .metric.variant, value: .value[1]}'
# Expected: a ratio close to 50/50 for a two-arm experiment

# Is the conversion rate per variant being computed?
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (variant) (rate(experiment_conversion_total[5m])) / sum by (variant) (rate(experiment_exposure_total[5m]))' \
  | jq '.data.result[] | {variant: .metric.variant, value: .value[1]}'
# Expected: a float between 0 and 1 per variant

# Sample ratio mismatch alert: the ratio of exposure between
# control and treatment should be close to 1.0.
abs(
  sum(experiment_exposure_total{variant="info"})
  /
  sum(experiment_exposure_total{variant="info"})
  - 1
) > 0.05

How it can fail

1. Sample ratio mismatch. Symptom: the exposure ratio between variants is 60/40 when the assignment service is configured for 50/50. The fix is to debug the variant assignment service; the experiment is not a valid test.
2. Novelty effect. Symptom: the treatment variant wins dramatically in the first 48 hours, then converges to the control. The fix is to wait until the experiment has run for the planned duration; do not stop the experiment on the early signal.
3. Peeking. Symptom: the team checks the cumulative conversion rate every hour and stops the experiment on the first statistically significant signal. The false positive rate is 20 percent at five peeks. The fix is to fix the sample size in advance and stop only when the planned sample size is reached.
4. Unbounded label cardinality. Symptom: the experiment_id label has hundreds of unique values from past experiments. The cumulative view is meaningless. The fix is to drop the experiment_id label from the counter after the experiment ends; the label is short-lived.
5. Missing control. Symptom: the team forgot to assign a control arm; the experiment is a single arm. The conversion rate is a single number, not a comparison. The fix is to require the control arm in the experiment design doc.
6. The analysis is done in Prometheus. Symptom: the team uses PromQL to compute the p-value and the confidence interval. The numbers are wrong because PromQL does not have the statistical primitives for the analysis. The fix is to use a statistics engine (Optimizely, Eppo, in-house) and feed it the raw counts from Prometheus.

How to troubleshoot it

1. Is the exposure counter being scraped? Check up\{job= "experiment-business"\}. If zero, the collector is down.
2. Is the exposure ratio between variants sane? Compute the ratio and compare to the assignment service’s configuration. A mismatch is the first failure to check.
3. Is the conversion rate per variant matching the assignment? If the conversion rate differs by a factor of 2 in the first hour, the assignment is wrong.
4. Is the variant label on the RED metrics? If missing, the team cannot see the per-variant technical impact. The fix is to add the label at the application layer.
5. Is the analysis engine producing the same raw counts as Prometheus? The analysis engine reads the raw counts from the warehouse; Prometheus has the live counts. The two should agree at the time of the experiment’s end.

Security implications

• Experiment data can leak PII. The experiment_id label on a request counter can be correlated with the user agent and the IP address to identify the user in the experiment. The cardinality contract must exclude user identifiers; the relabel rule must drop them at ingest.
• Experiment metadata is sensitive. The hypothesis, the variant definitions, and the conversion goal are business strategy. The Prometheus instance that holds them should sit behind the same access controls as the strategy doc.
• The variant assignment service is a write path. A misconfigured service that allows arbitrary variant values is a metric injection vulnerability. The relabel rule at the collector should restrict the variant to a known set.

Performance implications

The performance cost of a single experiment is bounded by the number of variants and the number of concurrent experiments. The cardinal cost model is:

  Variants   Concurrent   Series per counter
    2           1                  2
    2           5                 10
    5           5                 25
    5          20                100
    5          50                250    <- the practical ceiling

The ceiling is the limit on the cardinality budget. The team that ships 50 concurrent experiments with 5 variants each has committed 250 series to the experiment counter. The team that ships 50 concurrent experiments with 5 variants and the experiment_id label has 250 * 50 = 12,500 series. The multiplier is the trap.

The convention that prevents the trap is: experiment_id is short-lived. The label is dropped from the counter after the experiment ends. The Prometheus retention window is the natural expiring mechanism.

Verification

You should now be able to answer:

• What is the difference between A/B test monitoring and A/B test analysis, and why must the two be separated?
• What is sample ratio mismatch, and what is the first alert that catches it?
• What is the cardinality cost of the variant label on the RED metrics, and how is it bounded?
• Why is computing the p-value in Prometheus the wrong workflow?
• What is the first symptom that the variant assignment service is broken?

Quiz