Span Status

An alert fires: HTTP 5xx rate above 1% for checkout. The on-call engineer opens Grafana and pastes \{ service.name = "checkout" && status = error \} into TraceQL. Within ten seconds the engineer has the failing traces and the failing spans; within a minute the engineer is staring at the exact exception message. The alert pointed at the symptom; the status filter pointed at the cause. That is what span status buys you: the cheapest possible filter for “which traces failed”.

What it is

OpenTelemetry defines three span status values:

• UNSET — the default. The span has no opinion on success or failure. Use this when the span completed without any error the application cares about.
• OK — the span completed successfully, even if the HTTP / RPC status would otherwise suggest an error. Used sparingly: the canonical “I checked, this is fine” signal.
• ERROR — the span failed. Use this when the unit of work did not produce the result the caller asked for.

The status is a single field with an optional human-readable message. It is not a severity scale; it is a tri-state. The OpenTelemetry specification is explicit that UNSET is the default and that an uncaught exception does not automatically flip the status to ERROR — the application is responsible.

Why a sysadmin cares

The status field is the cheapest filter in TraceQL. Every Tempo search index entry includes the status; every Grafana dashboard keyed on errors uses it; every alert that pages on error rate uses it. The cost of getting it wrong is paid at incident time:

• Every span left at UNSET when it should be ERROR makes the error-rate dashboard miss real failures. The alert does not fire when it should.
• Every span set to ERROR when it succeeded makes the same dashboard noisy. The alert pages for healthy requests.

Span status is the contract between the application’s notion of success and the platform’s notion of failure. If the contract is honoured, the platform is correct. If it is not, the platform is wrong about the system it is watching.

How it works

A typical HTTP server span looks like this:

span: POST /checkout
  kind:        SERVER
  status:      ERROR          <- the only field that says "this failed"
  status.message: "upstream payment-svc returned 503"

  events:
    - exception          <- the diagnostic context for the same failure
      attributes:
        exception.type:       "UpstreamUnavailable"
        exception.message:    "503 Service Unavailable from payment-svc"
        exception.stacktrace: "Traceback ..."

Two facts matter:

• status = ERROR is what makes the span findable by TraceQL. Without it, the exception event exists but the search index does not flag the span as a failure.
• The exception event is what gives the failure context. Without it, the status message is the only human-readable signal, and the on-call engineer has no stack trace.

Set the status; emit the exception event. They are not interchangeable. The status is the summary; the event is the detail. A span with only status = ERROR and no exception event is a failure the engineer cannot root-cause. A span with only an exception event and status = UNSET is a failure the search index does not surface.

Auto-instrumentation for HTTP servers in most languages sets status = ERROR automatically when the response status code is 5xx and leaves it UNSET for 2xx / 3xx / 4xx. Custom business logic must set it explicitly.

Under the hood

How to configure it

Application side — Python SDK, setting status explicitly on business failure:

from opentelemetry import trace
from opentelemetry.semconv.trace import SpanAttributes
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("checkout.charge")

with tracer.start_as_current_span("payment.charge") as span:
    span.set_attribute(SpanAttributes.RPC_SYSTEM, "grpc")
    span.set_attribute(SpanAttributes.RPC_SERVICE, "payment.PaymentService")

    try:
        result = charge_card(...)
    except UpstreamUnavailable as exc:
        # 1. Record the exception event -- the diagnostic detail
        span.record_exception(exc)

        # 2. Set the span status -- the searchable summary
        span.set_status(Status(StatusCode.ERROR, "upstream payment-svc returned 503"))

        # 3. The exception still propagates; the caller decides what to do
        raise

# HTTP server span example: status is set automatically by the
# auto-instrumentation when the response is 5xx. To override
# (for example, to mark a 200 as ERROR when business logic failed):
with tracer.start_as_current_span("POST /checkout") as span:
    if business_validation_failed():
        span.set_status(Status(StatusCode.ERROR, "cart contains out-of-stock items"))

The convention set_status after record_exception is the canonical sequence. Reverse the order and you have the same effect; the order does not matter on the wire but does matter to the human reading the code.

Collector side — the transform processor can rewrite status based on attributes. The common case is upgrading a 5xx to ERROR if a downstream SDK forgot to:

# /etc/otelcol/config.yaml
processors:
  transform/status_from_http:
    trace_statements:
      - context: span
        statements:
          # If the span is an HTTP server span with a 5xx status code
          # and the span status is still UNSET, set it to ERROR.
          - set(status.code, "Error") where
              attributes["http.response.status_code"] != nil and
              attributes["http.response.status_code"] >= 500 and
              status.code == "Unset"
  batch:
    timeout: 5s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform/status_from_http, batch]
      exporters: [otlp/tempo]

This is a backstop, not a substitute for fixing the application SDK. Every backstop is a sign that some instrumentation library is missing the convention.

How to validate it

# 1. Generate a request that fails.
TRACE=$(curl -s -X POST http://shop.internal/checkout \
  -H 'Content-Type: application/json' \
  -d '{"cart_id":42,"force_error":true}' | jq -r .trace_id)

# 2. Confirm the status is ERROR and the exception event exists.
curl -s -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE" \
  | jq '.batches[].scopeSpans[].spans[]
         | {name, status_code: .status.code, status_msg: .status.message,
            exception_event: ([.events[] | select(.name=="exception")] | length)}'

# Expected:
# { "name": "POST /checkout",
#   "status_code": "Error",
#   "status_msg": "upstream payment-svc returned 503",
#   "exception_event": 1 }
# { "name": "payment.charge",
#   "status_code": "Error",
#   "status_msg": "upstream payment-svc returned 503",
#   "exception_event": 1 }

# 3. Search Tempo for every failing trace in the last 15 minutes.
curl -s -G -u "$TEMPO_USER:$TEMPO_PASS" \
  --data-urlencode 'q={ service.name = "checkout" && status = error }' \
  --data-urlencode 'limit=20' \
  --data-urlencode 'since=15m' \
  http://tempo.internal:3200/api/search | jq '.traces | length'
# Expected: non-zero

# 4. Confirm UNSET is the default on successful paths.
curl -s -G -u "$TEMPO_USER:$TEMPO_PASS" \
  --data-urlencode 'q={ service.name = "checkout" && status = unset && duration > 100ms }' \
  --data-urlencode 'limit=20' \
  http://tempo.internal:3200/api/search | jq '.traces | length'
# Expected: a small number -- only healthy long-running paths.

How it can fail

1. Always UNSET. Auto-instrumentation is disabled, the application catches every exception silently, or the SDK initialisation was reset. Symptom: the error dashboard is flat; alerts that depend on status = error never fire; Tempo’s indexed spans are all UNSET.
2. ERROR set but no exception event. The application sets the status but never emits the exception event. Symptom: status = error traces exist but the trace UI shows no diagnostic detail; the on-call engineer has to read status.message to get any context.
3. Exception event but no status change. A library catches an exception, emits record_exception, but does not set the status. Symptom: the trace UI shows the exception but the search index does not flag the span as a failure; status = error filters miss the trace.
4. Status overwritten by a wrapper. A framework middleware sets status = OK on every span regardless of inner failures. Symptom: the leaf span has ERROR but the wrapping span has OK; the root span — the one Tempo indexes most aggressively — says nothing wrong.
5. HTTP 4xx counted as error. Auto-instrumentation that marks 4xx as ERROR inflates the error rate. Symptom: every legitimate client mistake (validation failure, bad auth token) appears in the error dashboard.
6. Status message contains PII. A developer puts the user’s email in the status.message. Symptom: a security review finds PII in trace data; the message field is indexed and exposed to anyone with Tempo read access.

How to troubleshoot it

Security implications

status.message is a free-text field that ends up indexed in Tempo’s search index for any service that uses it. It is rarely PII-sensitive — the convention is to put a short operational description (“upstream returned 503”, “validation failed”) — but the discipline is the same as for any other attribute: do not put user identifiers, emails, or session tokens in messages. A common leak is the upstream error message propagated verbatim; if the upstream echoes the user’s input into its error response, the status message becomes a mirror of the user’s data.

Performance implications

The status is a single byte on the wire and a single column in the search index. There is no meaningful cost from setting it correctly or leaving it UNSET. The cost is paid by not setting it: the alert that does not fire, the dashboard that does not update, the on-call engineer who cannot find the failing trace. Span status is the cheapest high-leverage field in the entire OpenTelemetry data model.

Production guidance

• Add a CI check that asserts every caught exception in the application is followed by both record_exception and set_status.
• Use the transform/status_from_http backstop in the Collector for HTTP spans that auto-instrumentation missed.
• Do not turn 4xx into ERROR by default. 4xx is the client telling you the request was wrong; that is normal traffic, not a platform failure.

Verification

You should now be able to answer:

• What are the three OpenTelemetry span status values, and which is the default?
• Why is status = ERROR not enough on its own to make a span fully informative?
• Write the TraceQL query that returns every failing trace for a given service in the last fifteen minutes.
• What does it mean operationally when every indexed span for a service has status = UNSET?
• When is status = OK the right choice on a span?

Quiz