Exemplar Format

At 11:14 a team migrated a Go service to a new release of the Prometheus client library. The new release changed the default exemplar suffix from one space after the # to two spaces. The OpenMetrics text parser, which is strict about whitespace under the -002 parser mode, rejected the entire scrape. The team’s metrics panel showed “no data” for the service. The trace pipeline was unaffected. The on-call engineer spent 40 minutes reading the scrape format spec before spotting the difference. The fix was a configuration flag on the client library to revert the spacing.

The OpenMetrics exemplar format is a fixed, deterministic suffix. This lesson is about what every byte in the suffix means, what the parser enforces, and what the operator should know before touching a client library config.

What it is

The exemplar is appended to the end of a histogram bucket line in the OpenMetrics text format. The full grammar, in the order it appears on the wire:

<metric_name>{<label_pair>,<label_pair>,le="<bound>"} <value> [<timestamp>] # {<exemplar_label>=<exemplar_value>,...} <exemplar_value> [<exemplar_timestamp>]

A real line from a /metrics endpoint:

http_request_duration_seconds_bucket{method="POST",route="/checkout",status="200",le="1.0"} 18434 # {trace_id="1bf0bbd05e6f4f8c8c0e8e3a2b1c0d1e",span_id="7890abcdef012345"} 0.948 1715638801.456

The five parts of the suffix, in order:

1. — a single space between the bucket value and the # comment marker. Required.
2. # — the OpenMetrics comment marker, followed by a single space. The # is the only character that introduces an exemplar; the parser will not parse an exemplar without it.
3. {trace_id="...",span_id="..."} — the exemplar label set. trace_id is required by the OpenMetrics spec; span_id is optional but emitted by every mainstream library. Additional labels may be added by the producer.
4. <value> — the value of the observation that produced the exemplar. Required.
5. <timestamp> — the timestamp in seconds, with sub-second precision. Optional; defaults to the scrape timestamp when omitted.

The exemplar is not a comment. The # is the exemplar marker in OpenMetrics, and the parser treats it as syntax. The format is text-only; the proto and protobuf encodings serialize the same fields under different delimiters.

Why a sysadmin cares

The format is the contract between the producer and the Prometheus server. Both ends must agree on every byte. The parser is strict: a malformed line fails the whole scrape of the whole family, not just the malformed exemplar. A team that ships a custom exporter with a hand-rolled exemplar serializer can take down the histograms for the service without the trace pipeline ever knowing.

The format also encodes the cost. The bytes per exemplar are not just the trace_id; they include the label set, the value, the timestamp, and the separator characters. A team that knows the format knows how to estimate the storage cost of enabling exemplars on a high-cardinality histogram.

How it works

The mental model is a histogram bucket line with an optional trailer. The trailer is independent of the bucket. The bucket is a cumulative counter; the trailer is a point-in-time observation. The two can be inspected separately.

http_request_duration_seconds_bucket{
  method="POST",
  route="/checkout",
  status="200",
  le="1.0"
} 18434                                                    <-- bucket value
  # {                                                       <-- exemplar marker
    trace_id="1bf0bbd05e6f4f8c8c0e8e3a2b1c0d1e",
    span_id="7890abcdef012345"
  }
  0.948                                                    <-- exemplar value
  1715638801.456                                           <-- timestamp

The OpenMetrics parser walks the line left-to-right. The trailer is parsed only when the bucket line is well-formed and the parser is in exemplar mode (set by the Accept header on the request). The Prometheus client library sets the flag to enable exemplar parsing on every scrape by default in 2.55.x; the operator does not need to negotiate it.

The label set is the carry-forward

The exemplar carries the labels that are also on the bucket line. A label added to the exemplar that is not on the bucket is dropped by the parser. A label on the bucket that is not on the exemplar is preserved on the bucket side and not duplicated on the exemplar side.

The convention is that the exemplar label set is the same as the bucket label set, plus trace_id and span_id. The producer does not need to re-emit the bucket labels; the spec assumes the bucket is the source of truth for them. The Prometheus server reconstructs the full label set at query time.

The parse is all-or-nothing

The OpenMetrics parser parses the exemplar trailer as a single unit. If the trailer is malformed — a missing closing brace, an unescaped quote, a non-numeric value — the parser rejects the whole bucket line. The bucket value is accepted but the exemplar is dropped. The line is logged with the malformed-trail error; the scrape continues.

The Prometheus server, however, has a stricter mode. If the malformed trailer is preceded by other malformed bytes in the same scrape, the server may reject the entire scrape. The observability team should not rely on the lenient parser mode for production. Run the linter.

The value is the observation, not the bucket

The numeric value in the trailer is the value of the observation that produced the exemplar, not the value of the bucket counter. The bucket counter is 18434. The exemplar value is 0.948. The two are unrelated. The exemplar value answers “how long did this request take”; the bucket value answers “how many requests have crossed this boundary so far”.

The timestamp is the observation time

The trailer timestamp is when the exemplar was recorded. It is not the scrape timestamp. The Prometheus server uses the trailer timestamp to bucket the exemplar into the correct 15-minute retention window. The scrape timestamp is the time the data arrived.

How to configure it

The format is not negotiated; it is the spec. The operator configures the endpoints, not the bytes.

1. Acknowledge the format on the Prometheus server.

The Prometheus server is started with the exemplar flag. The parser detects the format from the # { trailer on the scrape response. No additional configuration is required on the server side.

2. Configure the producer to emit the format.

Most client libraries emit the format by default. The edge cases:

// Go: prometheus/client_golang (verified on 1.20.x)
import "github.com/prometheus/client_golang/prometheus"

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "Time spent handling HTTP requests.",
        Buckets: prometheus.DefBuckets,
        // Exemplars are recorded by default when an active
        // span context exists. The trailer is emitted in
        // the OpenMetrics format.
    },
    []string{"method", "route", "status"},
)

# Python: prometheus_client (verified on 0.20.x)
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Time spent handling HTTP requests',
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
    labelnames=('method', 'route', 'status'),
    # enable_exemplars defaults to True (0.18+).
    # The library attaches the active OpenTelemetry trace
    # context to the bucket trailer when one is available.
)

# OpenTelemetry Collector: histograms_to_exemplars processor
# (PROCESSOR block — drop into the pipelines section)
processors:
  histograms_to_exemplars:
    # The processor scans incoming OTLP histogram metrics for
    # paired exemplar attachments and forwards them as
    # OpenMetrics exemplar trailers on the Prometheus exporter.
    # No additional configuration is required for the format.

3. Validate the format with the Prometheus text parser.

# READ-ONLY
promtool check metrics /tmp/check.txt

Where /tmp/check.txt is a copy of the scrape response. The command parses the file with the same parser that Prometheus uses on the scrape. A malformed trailer is reported as text format parsing error in line N.

How to validate it

Three layers of validation, each catching a different class of format error.

1. The trailer is present on the bucket line.

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | grep -c '# {trace_id'

Expected output: a non-zero count. If the count is zero, either the trailer is missing or the trace ID label is spelled differently.

2. The trailer is well-formed.

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | grep '# {trace_id' \
  | head -1

Expected output: a line that ends with a closing brace, a numeric value, and an optional timestamp. If the line ends in the middle of the trailer or the value is non-numeric, the producer is broken.

3. The trailer parses cleanly.

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | grep '# {trace_id' > /tmp/exemplars.txt
promtool check metrics /tmp/exemplars.txt

Expected: empty output and exit status 0. Any non-empty output is a parse error. The line number is reported.

4. The Prometheus query API returns exemplars.

# READ-ONLY
curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="1.0"}' \
  --data-urlencode 'start=2026-08-13T11:00:00Z' \
  --data-urlencode 'end=2026-08-13T11:30:00Z' \
  | jq '.data[0].exemplarLabels'

Expected: an object with trace_id and span_id keys. If the keys are absent, the format is parsed but the labels have been dropped.

How it can fail

Six failure modes, ordered by frequency.

1. Whitespace drift. A custom exporter emits # {trace_id (two spaces after the #). The Prometheus parser, in default mode, rejects the trailer. Symptom: the histogram shows values on the panel; the exemplars are absent; the scrape response reports text format parsing error.
2. Unescaped quote in a label. A producer includes a label value with a " inside (a user name, a JSON string). The trailer becomes trace_id="abc"def". The parser rejects the trailer at the second quote. Symptom: the bucket line is dropped; the metric disappears from the panel until the producer is fixed.
3. Missing timestamp separator. A producer emits the value and timestamp without a space between them. The parser reads them as a single number and the timestamp is silently truncated. Symptom: the exemplar is stored with a bad timestamp; the Grafana diamond is offset horizontally.
4. Trailer on a non-histogram metric. A producer emits a # {trace_id=...} trailer on a counter. The parser rejects the trailer on a non-histogram line. Symptom: the counter scrape continues, but the trailer is dropped with a parse warning.
5. Newline embedded in a label value. A producer includes a \n in a label value. The trailer wraps onto a second line. The parser sees a new line and ends the current metric. Symptom: the next metric on the response is misaligned; the parse fails at the second line.
6. Multi-line trailer. A producer emits two # {trace_id} trailers on the same bucket line. The parser parses the first and ignores the second. Symptom: the trace from the second observation is lost; the bucket appears correctly.

How to troubleshoot it

Steps in order from cheapest to most expensive.

1. Inspect the raw scrape. curl /metrics | grep. The trailer is on the line, or it is not. The shape is visible by eye.
2. Run promtool check metrics on the scrape. The tool parses with the same parser Prometheus uses. The output names the line and the byte offset of the error.
3. Check the library version. A breaking change in the exemplar format is rare but documented. The library changelog is the right place to look.
4. Check the Accept header. The Prometheus server sends Accept: application/openmetrics-text or the legacy format. The exemplar format is only emitted in the OpenMetrics format. If the client library is configured to emit the legacy format, the trailer is absent.
5. Check the trace context propagation. If the trailer is well-formed but the trace_id is empty, the active span context is missing. The histogram is updated; the trace ID is not populated.

Security implications

The trailer is a public line. The same curl the operator uses to read the format is the one an attacker uses to read the trace IDs. The label set on the bucket is the label set on the exemplar. A label that resolves to a user, customer, or session becomes a unique exemplar per user. The exemplar is not a log line but the same privacy discipline applies.

The trailer is also a target for injection. The # { marker is the boundary; if an attacker can inject a label value that contains a closing brace or a newline, they break the format. The OpenMetrics text format is not designed to be parsed in a sandbox with the attacker holding the input. Treat the producer as trusted and the endpoint as protected.

Performance implications

The trailer is small in bytes. The cost per exemplar on the wire is roughly:

• trace_id="..." — 32 hex characters plus 12 bytes of framing = 44 bytes.
• span_id="..." — 16 hex characters plus 11 bytes of framing = 27 bytes.
• Separator characters — 5 bytes for # {, }, ,, space.
• Value and timestamp — 20 bytes average.
• Total — approximately 100 bytes per trailer.

The appender file stores the trailer plus the bucket label set. A bucket label set with three labels of average length adds 60 bytes. Total per exemplar: 160 bytes. A platform with 10,000 active histogram-bearing series, scraped at 15-second intervals, accumulates 40,000 exemplars per minute. The appender file grows by 6.4 MB per minute at peak, capped by the 15-minute retention at 96 MB.

The wire cost is paid on every scrape. A 100-byte trailer per bucket per scrape is a 1.2 MB per second overhead on a 10,000-bucket histogram. The CPU cost on the Prometheus server is the parse cost; the bandwidth cost is the trailer serialisation.

The trade-off is paid in appender size and bandwidth for the single-click pivot from a metric to a trace. The cost is bounded by the cardinality of the histogram label set, not by the cardinality of the trace.

Production guidance

• Pin the client library version. The format is stable in OpenMetrics, but the libraries emit the format with version-specific quirks. Pin the version in the service’s dependency manifest.
• Validate the format in CI. Run promtool check metrics on a canned scrape response as part of the exporter contract test. The check fails the build before the format reaches the platform.
• Restrict the label set. The exemplar carries the bucket label set. Strip PII from the label set; the exemplar is stripped with it.
• Monitor the parse errors. The Prometheus server emits prometheus_target_scrape_pool_exemplar_failed_total for parse failures. Alert on any non-zero rate.
• Plan for the wire cost. The bandwidth cost of the trailer is roughly 1% of the bucket wire cost. Document the cost in the capacity plan.

Verification

You should now be able to answer:

• What are the five parts of the OpenMetrics exemplar suffix in order?
• Which field is required by the spec and which fields are optional?
• Why is the trailer value different from the bucket value?
• What happens when the parser encounters a malformed trailer?
• How many bytes does an exemplar trailer cost on the wire?

Quiz