Span Events

A trace is a 240 ms payment.charge span. The dependency looks slow, but the question is why. Opening the span shows three events along its timeline: at 18 ms a request.start event with the merchant ID; at 35 ms a fraud.scored event with the risk score; at 230 ms an exception event with a stack trace saying “connection reset by peer”. The 5 ms gap between fraud.scored and exception is where the dependency stopped responding. Without events, the span is a duration; with events, it is a story.

What it is

A span event is a timestamped, structured annotation that lives inside a span. Each event has:

• a name (exception, request.start, cache.hit),
• a timestamp (nanoseconds since epoch, drawn from the same monotonic clock as the parent span),
• a set of typed attributes (same key / value model as span attributes).

Events are emitted between the start and the end of the parent span; they appear as ticks on the span’s timeline. They are not separate signals; they are part of the span.

The OpenTelemetry semantic conventions reserve a few event names that every backend recognises:

• exception — emitted by language instrumentation when a thrown exception is caught; carries a stack trace under exception.stacktrace and the exception type and message.
• message — for messaging instrumentation; carries the message size and ID.
• Custom event names are common and encouraged. The convention is noun.verb (cache.miss, retry.attempt, db.slow).

Why a sysadmin cares

Events are what make a span legible. The duration tells you how long; the attributes tell you what; the events tell you the steps that happened inside. Three operational reasons to use events:

• Failure context. A status = error span with no exception event is opaque: it failed, but the on-call engineer has no reason. An exception event carries the type, message, and stack trace.
• Causal steps. A business workflow that involves several distinct actions inside one span — fraud.scored, risk.approved, ledger.posted — is a sequence of events; rendering them on the timeline is the difference between “this took 200 ms” and “this took 5 ms on fraud and 195 ms waiting for the ledger”.
• Structured annotation without a new log line. A span event does not go through the log pipeline; it lives where the trace lives. There is one correlation identifier by construction.

Events are not a replacement for logs. They are a complement: logs answer “what happened in the service over the last ten minutes”; events answer “what happened inside this one span”.

How it works

A span has a single timeline. Events appear on that timeline at the moment they happen. An exception event emitted from inside an HTTP handler looks like this on the wire:

span: payment.charge
  start: 14:22:11.000
  end:   14:22:11.240
  duration: 240 ms

  event: request.start     14:22:11.018   merchant_id=m_8821 amount=4299
  event: fraud.scored       14:22:11.053   risk_score=12 decision=allow
  event: exception          14:22:11.230   exception.type=ConnectionError
                                              exception.message="connection reset by peer"
                                              exception.stacktrace="Traceback ... "

The events are inside the span. Tempo renders them as ticks on the span’s bar; TraceQL filters against them as { span.events.exception.type = "ConnectionError" }.

Events differ from attributes in three ways:

• Attributes describe the whole span (“this was a POST to /charge with HTTP 200”). They are set once and immutable.
• Events describe moments within the span (“at 18 ms the request started; at 35 ms fraud scored it”). They have their own timestamps and their own attribute sets.
• Logs describe moments within the service (“at 14:22:11 the connection pool exhausted”). They have their own correlation identifier and live in Loki.

Under the hood

How to configure it

Application side — Python SDK emitting a custom event plus an exception event:

from opentelemetry import trace
from opentelemetry.semconv.trace import SpanAttributes
import traceback

tracer = trace.get_tracer("checkout.charge")

with tracer.start_as_current_span("payment.charge") as span:
    # Structured event with attributes
    span.add_event(
        "fraud.scored",
        attributes={
            "risk.score": 12,
            "risk.decision": "allow",
            "risk.model": "v3",
        },
    )

    try:
        result = charge_card(...)
    except ConnectionError as exc:
        # The SDK has a helper for the standard "exception" event
        span.record_exception(exc)
        # Optional: record additional structured context
        span.add_event(
            "payment.retry_scheduled",
            attributes={"retry.delay_ms": 250, "retry.attempt": 2},
        )
        raise

The record_exception(exc) helper emits an exception event with exception.type, exception.message, and exception.stacktrace attributes populated from the exception object. It does not set status = error; that is the caller’s responsibility (covered in the next lesson).

Collector side — no special configuration is needed. The Collector forwards events as-is.

How to validate it

# 1. Generate a request that emits an exception event.
TRACE=$(curl -s -X POST http://shop.internal/checkout \
  -H 'Content-Type: application/json' \
  -d '{"cart_id":42,"force_error":true}' | jq -r .trace_id)

# 2. Inspect the span's events.
curl -s -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE" \
  | jq '.batches[].scopeSpans[].spans[]
         | select(.name=="payment.charge")
         | {name, events: [.events[] | {name, time: .timeUnixNano,
                                       attrs: (.attributes | map({(.key): .value}) | add)}]}'

# Expected (illustrative):
# {
#   "name": "payment.charge",
#   "events": [
#     { "name": "fraud.scored",
#       "time": 1716183731053000000,
#       "attrs": { "risk.score": 12, "risk.decision": "allow" } },
#     { "name": "exception",
#       "time": 1716183731230000000,
#       "attrs": { "exception.type": "ConnectionError",
#                  "exception.message": "connection reset by peer" } }
#   ]
# }

# 3. Search for every trace that contains an exception event of a given type.
curl -s -G -u "$TEMPO_USER:$TEMPO_PASS" \
  --data-urlencode 'q={ span.events.exception.type = "ConnectionError" }' \
  --data-urlencode 'limit=20' \
  --data-urlencode 'since=15m' \
  http://tempo.internal:3200/api/search | jq '.traces | length'
# Expected: non-zero

Note that span.events.exception.type = ... searches by event attribute; Tempo must be configured to index event attributes for this to be fast. Otherwise the same query is correct but returns traces only after Tempo has loaded the full trace block for each candidate.

How it can fail

1. Event attribute explosion. A span emits one event per retry attempt, and each retry event carries the full request payload as an event attribute. Symptom: the trace block grows to tens of megabytes per span; Tempo ingests slowly; the object-store write rate spikes.
2. PII in events. A developer puts the user’s email into an event attribute. The email persists with the trace block for the retention period. Symptom: a security review finds PII in traces that was never sent to logs.
3. Events without context. An event is emitted but the parent span context is lost (a separate worker thread, an off-the-clock code path). Symptom: events appear in Tempo at the top level instead of nested in the span.
4. Event used as a log line. A developer treats add_event as a fancier log call and emits 200 events per request. Symptom: spans are 50 KB on the wire; ingest CPU rises; search-index size balloons.
5. Wrong event name. A custom event is named error instead of exception. TraceQL filters written by the rest of the team (events.exception.type) miss it. Symptom: dashboards keyed on the standard name return zero traces for the service.
6. Exception event but no stack trace. record_exception is called with a wrapped exception that the helper does not understand, or __traceback__ is not preserved. Symptom: the exception event has type and message but no exception.stacktrace, leaving the on-call engineer to guess at the call site.

How to troubleshoot it

Security implications

Events share the same retention and access profile as spans. The same discipline applies: no PII (no full SQL statements, no raw emails, no JWTs, no authorisation headers) in event attributes; no opaque blobs that might contain any of the above. The standard exception.stacktrace attribute is the common leak point — third-party libraries often include the request payload or the SQL string in the exception message. Treat stack traces as untrusted text.

Performance implications

Each event is a few hundred bytes to a few kilobytes on the wire, dominated by attributes. The cost is linear in event count per span. A practical budget:

• 0-5 events per span for typical request paths
• 1-2 events per span for error paths (exception plus one context event)
• Bounded event attribute cardinality (a few dozen attributes per event at most)

Above that, the trace block grows, ingest slows, and the search-index size rises. The right discipline is the same as for attributes: bounded event vocabulary, named owners, and periodic review.

Production guidance

• Use record_exception for exceptions; use add_event for application-level steps.
• Treat the exception event as the standard interface to errors. A span with status = error but no exception event is incomplete; a span with an exception event but status = UNSET is incomplete in the other direction.
• Do not put PII in event attributes. The retention matches the trace, which is longer than log retention by default.

Verification

You should now be able to answer:

• What three fields does a span event carry?
• How does a span event differ from a log line, operationally?
• What is the OpenTelemetry-standard name for an event that records a caught exception?
• Why does Tempo render events from the trace block rather than the search index?
• What is the practical upper bound on events per span before ingest cost becomes noticeable?

Quiz