Multi-Step Journey Probes · Observability

A customer reports “checkout is broken.” The on-call opens the dashboard. The HTTP probe is green. The TLS probe is green. The DNS probe is green. The application’s own metrics are green: CPU, memory, request rate, error rate. The customer is convinced the application is broken; the dashboard is convinced the application is fine. Both are right. The customer’s flow is “add to cart → enter address → choose shipping → enter payment → confirm.” The HTTP probe hits /healthz. The two answers are different questions about different surfaces.

The multi-step journey probe is the synthetic signal that asks “does the user-visible flow work?” — at the cost of being expensive, brittle, and harder to maintain than a single HTTP probe.

What it is

A multi-step journey probe is a scripted sequence of HTTP or browser interactions that drives a user flow end to end. The script opens a session, authenticates, performs the flow’s steps, and asserts the outcome at each step. The script runs on a schedule from one or more vantage points; the result is recorded as a metric.

The two implementations in the Prometheus / Grafana ecosystem are:

k6 (HTTP). A Go-based load-testing tool with a JavaScript test script. The script issues HTTP requests with cookies, headers, and assertions; the result is a set of metrics that can be pushed to a Pushgateway or to a remote-write receiver. k6 is fast, cheap, and faithful to HTTP semantics; it does not execute JavaScript or render the page.
Playwright (browser). A Node.js-based browser automation framework. The script drives a headless Chromium, Firefox, or WebKit through the user flow; the result is a set of metrics and screenshots. Playwright is faithful to the user’s actual browser; it is more expensive than k6 by an order of magnitude.

The blackbox_exporter’s http_2xx module is not the right shape for either. The HTTP module issues a single request; the journey script issues many. The right shape is a separate prober (k6, Playwright, or a custom microservice) whose results are pushed.

The journey is recorded in Prometheus via the Pushgateway (pull-mode Prometheus cannot pull from a browser or from a scripted loop). The Pushgateway is a single point of failure; the discipline is to push metrics with job and instance labels that distinguish the journey, the step, and the vantage point.

Why a sysadmin cares

The journey is the synthetic signal that closes the gap between “the boundary is up” and “the user can complete the flow.” Three production questions map onto it:

Does the user-visible flow work? A journey that exercises login, search, and checkout asserts the stitched flow, not the individual hops.
Is the flow fast enough for the SLA? The journey records duration per step; the dashboard surfaces the step that drifts.
Does the flow work from the user’s vantage point? A journey that runs from three regions catches the regional failures that single-vantage-point probes miss.

The journey is also the right shape for a canary gate. “Block the rollout if the checkout journey is red for two consecutive runs” is a single rule that covers more surface than any single HTTP probe.

The cost is real. A journey is ten to one hundred times more expensive than a single HTTP probe: it issues more requests, holds sessions, and may run a full browser. The right cadence is minutes to tens of minutes, not seconds.

How it works

The script lives outside the exporter. The exporter is not in the loop. The mental model has three moving parts: the journey runner, the Pushgateway, and Prometheus.

  journey runner (cron)                Pushgateway       Prometheus
  -------------------------            -----------       ----------
  09:00:00  journey.checkout           |                |
    |--- step 1: GET /login -------->  |                |
    |<-- 200 + session cookie ----    |                |
    |--- step 2: POST /auth -------->  |                |
    |<-- 200 + auth token --------    |                |
    |--- step 3: GET /cart --------->  |                |
    |<-- 200 + cart id -----------    |                |
    |--- step 4: POST /checkout ---->  |                |
    |<-- 200 + order id -----------   |                |
    |                                  |                |
    |  push:                           |                |
    |    journey_success{step="4"} 1   |---> scrape --->|
    |    journey_duration_seconds{step="4"} 4.2         |
    |    journey_failure_reason{step="2"} "401"         |
    v                                  v                v
  exit code 0 / 1                  labelled metrics    time-series store

Two design choices dominate:

Pull mode vs push mode. The journey runner owns the timing; Prometheus cannot pull from a browser. Push mode via the Pushgateway is the right shape. The Pushgateway’s honor_labels: true is mandatory; otherwise the labels get rewritten and the journey-step dimension is lost.
Step labels. The journey records journey_success and journey_duration_seconds per step. The label set must distinguish the journey (journey="checkout"), the step (step="4"), and the vantage point (region="eu-west-1"). Cardinality rises with the number of journeys times the number of steps times the number of vantage points; budget accordingly.

How to configure it

Below is a production-shaped k6 script for the canonical “login → add to cart → checkout” journey, paired with the Prometheus scrape job for the Pushgateway.

// /etc/journeys/checkout.js
import http from 'k6/http';
import { check, group } from 'k6';
import { Counter, Trend } from 'k6/metrics';

export const options = {
  thresholds: {
    'journey_failure': ['count==0'],
    'journey_duration': ['p(95)<8000'],
  },
};

const journeySuccess = new Counter('journey_success');
const journeyFailure = new Counter('journey_failure');
const journeyDuration = new Trend('journey_duration_seconds');

export default function () {
  const base = 'https://api.example.com';
  const start = Date.now();

  group('01_login', function () {
    const res = http.post(`${base}/v1/auth`, JSON.stringify({
      username: __ENV.JOURNEY_USER,
      password: __ENV.JOURNEY_PASS,
    }), { headers: { 'Content-Type': 'application/json' } });
    const ok = check(res, {
      'login status 200': (r) => r.status === 200,
      'login has token': (r) => r.json('token') !== undefined,
    });
    if (!ok) { journeyFailure.add(1); return; }
  });

  group('02_add_to_cart', function () {
    const res = http.post(`${base}/v1/cart`, JSON.stringify({
      sku: 'TEST-SKU-1',
      quantity: 1,
    }), { headers: { 'Content-Type': 'application/json',
                     'Authorization': `Bearer ${__ENV.JOURNEY_TOKEN}` } });
    const ok = check(res, {
      'cart status 200': (r) => r.status === 200,
      'cart has id': (r) => r.json('cart_id') !== undefined,
    });
    if (!ok) { journeyFailure.add(1); return; }
  });

  group('03_checkout', function () {
    const res = http.post(`${base}/v1/checkout`, JSON.stringify({}), {
      headers: { 'Content-Type': 'application/json',
                 'Authorization': `Bearer ${__ENV.JOURNEY_TOKEN}` },
    });
    const ok = check(res, {
      'checkout status 200': (r) => r.status === 200,
      'checkout has order_id': (r) => r.json('order_id') !== undefined,
    });
    if (!ok) { journeyFailure.add(1); return; }
  });

  journeySuccess.add(1);
  journeyDuration.add((Date.now() - start) / 1000);
}

# Run the journey from cron.
JOURNEY_USER="$JOURNEY_USER" JOURNEY_PASS="$JOURNEY_PASS" \
JOURNEY_TOKEN="$JOURNEY_TOKEN" \
  k6 run --out json=/tmp/journey-$(date +%s).json /etc/journeys/checkout.js

# Parse and push the step-level metrics.
journey-push /tmp/journey-*.json

The journey runner pushes the metrics to the Pushgateway. The Pushgateway is scraped by Prometheus.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: pushgateway_journey
    honor_labels: true
    scrape_interval: 60s
    scrape_timeout: 10s
    static_configs:
      - targets: ['pushgateway.internal:9091']
        labels:
          service: checkout-journey
          env: prod

honor_labels: true is mandatory. Without it, Prometheus rewrites the labels and the journey-step dimension is lost.

How to validate it

# 1. Run the journey by hand.
JOURNEY_USER="$JOURNEY_USER" JOURNEY_PASS="$JOURNEY_PASS" \
JOURNEY_TOKEN="$JOURNEY_TOKEN" \
  k6 run /etc/journeys/checkout.js
# ...
# checks.................: 100.00% ✓ 9  ✗ 0
# data_received..........: 12 kB  4.0 kB/s
# journey_duration_seconds...: avg=4.12  min=3.91  med=4.05  max=4.81  p(95)=4.81

# 2. Confirm the Pushgateway has the metric.
curl -s http://pushgateway.internal:9091/metrics | grep journey_
# journey_failure{group="01_login",...} 0
# journey_failure{group="02_add_to_cart",...} 0
# journey_failure{group="03_checkout",...} 0
# journey_duration_seconds{group="01_login",...} 0.412
# journey_duration_seconds{group="02_add_to_cart",...} 0.821
# journey_duration_seconds{group="03_checkout",...} 2.871

# 3. Confirm Prometheus has the metric.
journey_success{service="checkout-journey",env="prod"}
# {group="01_login",region="eu-west-1"} 1
# {group="02_add_to_cart",region="eu-west-1"} 1
# {group="03_checkout",region="eu-west-1"} 1

# 4. Validate the Pushgateway config.
promtool check config /etc/prometheus/prometheus.yml
# SUCCESS

# 5. Confirm the cron entry.
crontab -l | grep journey
# */5 * * * * /usr/local/bin/journey-run checkout

A green journey with the expected step metrics and a green Pushgateway scrape are the headline assertions. The journey-step dimension is what makes the dashboard useful.

How it can fail

Step 1 fails, steps 2 and 3 are never reached. The dashboard shows journey_success=0 for step 2, but the real failure is on step 1. The step-1 metric is missing. Symptom: step 2 red, step 1 silent; the on-call chases the wrong step.
Session cookie expires between steps. A long journey exceeds the session-cookie lifetime. Step 2 fails on 401. The script does not refresh the session. Symptom: step 2 red, step 1 green, the application is fine.
Pushgateway is down. The journey runs successfully but the metrics are not pushed. Prometheus shows up{job="pushgateway_journey"}=0. Symptom: dashboard is missing data, the journey is green, no one notices.
honor_labels: false. Prometheus rewrites the group label. The step dimension is lost; the dashboard shows one row per journey, not one per step. Symptom: cardinality is too low; the on-call cannot identify the failing step.
Credentials in the script. The script reads the password from an environment variable; the environment is logged in the cron output. Symptom: credentials in the platform’s log stream.
The journey is the wrong journey. The script drives the application through a flow that no real user follows (because the user is shown a redesigned UI). The journey is green; the user is on a different path. Symptom: journey green, real users red.
The cadence is wrong. The journey runs every minute and the application cannot sustain the load. The journey itself becomes the cause of the outage. Symptom: latency rises during the journey, customers see the journey traffic on their dashboards.
Browser drift. Playwright upgrades. The script that worked on Chromium 119 fails on Chromium 121 because the page selector changed. Symptom: journey red on a version bump that is otherwise harmless.

How to troubleshoot it

The order matters because the boundary at which the failure lives determines the remedy.

Is the journey runner alive? systemctl status journey-runner (or the equivalent). If this fails, the runner is the boundary.
Run the journey by hand. k6 run /etc/journeys/checkout.js. Look at the per-step pass/fail output. The script tells you which step failed; do not reason from the summary alone.
Inspect the per-step metrics on the Pushgateway. journey_failure{group="..."}. The group label is the step that failed.
Compare the journey latency to baseline. A drift from 4 s to 12 s on a single step is a brownout; the next failure is a timeout.
Cross-check with the HTTP probe for the same target. HTTP green, journey red means the boundary is up but the flow is broken. HTTP red, journey red means the boundary is the boundary.
Cross-check with the application’s structured logs. The journey’s correlation ID (a header injected by the script) appears in the application’s logs; the logs show the error.
Confirm the cron entry, the Pushgateway scrape, and the Prometheus retention. A green journey that disappears from the dashboard is a pipeline problem, not a journey problem.

Security implications

The journey runner holds credentials: a session cookie, an API token, a payment instrument. Those credentials must be:

Stored in a secret manager, not in the runner’s environment file.
Rotated regularly. A journey that runs every five minutes uses a credential thousands of times per day; the credential’s lifetime must reflect that.
Scoped narrowly. The test user must be able to perform the journey; the test user must not be able to perform anything else. The application should treat the test user as a real user for the journey’s path and as a limited user for everything else.

The journey runner’s logs must redact the credentials. A credential that ends up in the platform’s log stream is a credential that must be rotated. The runner’s correlation header (used to find the journey’s request in the application’s logs) must not contain the credential.

The journey script is code. Review it on every change. A journey that exfiltrates a credential is not a journey that the platform wants running.

Performance implications

A journey is more expensive than an HTTP probe by an order of magnitude. The cost is:

Runner CPU. Each journey allocates the script’s runtime, executes the HTTP requests, and parses the responses. A modern four-core runner handles roughly 10 concurrent journeys before saturating.
Network egress. Each journey is many requests from the runner host to the target. A 10-step journey is 10x the egress of a single HTTP probe.
Target load. Each journey is real load on the application. The cadence must be low enough that the journey does not become the cause of the outage.
Browser cost (Playwright). Each Playwright journey launches a headless browser; the browser cost dominates. A single Playwright journey is roughly 200 ms of browser startup plus the per-step interaction time.

The right cadence is minutes to tens of minutes for HTTP journeys, tens of minutes to hours for browser journeys. A 5-minute HTTP journey is appropriate for the checkout flow. A 30-minute browser journey is appropriate for the homepage render.

Production guidance

Pick the journey deliberately. The journey that backs an SLO must be the journey that real users follow. A journey against a deprecated UI is a journey that does not assert the production path.
Pick the cadence deliberately. The journey is expensive. The right cadence is the cadence that catches the failure mode within the SLA minus the journey cost. 5 minutes is typical for HTTP journeys; 30 minutes for browser journeys.
Treat the journey script as code. Review changes. Test in staging. Roll forward with the application.
Use the Pushgateway with honor_labels: true. The step dimension is the dashboard’s value.
Cross-check the journey with the HTTP probe. A green HTTP and a red journey is a flow problem, not a boundary problem.
Negotiate synthetic carve-outs with downstream providers. A journey that triggers rate limits is a journey that pages the on-call for the wrong reason.
Review the journey on every browser or framework upgrade. A version bump that breaks a selector is a version bump that pages the on-call.

Verification

You should now be able to answer:

What does a multi-step journey probe actually exercise that a single HTTP probe does not?
Why is the Pushgateway the right shape for journey metrics, and what does honor_labels: true protect?
When is k6 the right tool, and when is Playwright?
Why is the cadence for a journey measured in minutes rather than seconds?
What is the difference between a step-1 failure that masks steps 2 and 3, and a genuine step-2 failure?