The Critical Path

The trace shows seven spans. The longest one is 1.31 s. The on-call engineer assumes the longest span is the bottleneck and opens a ticket with payment-svc. The investigation finds that the payment-svc span was actually off the critical path; the request ran a parallel call to a recommendations service while waiting for payment-svc, and the payment-svc call overlapped with the recommendations call. The real bottleneck was the database query inside the user-profile lookup. The fix path is the critical path, not the longest span.

This lesson is the discipline of distinguishing the synchronous chain that bounds the request from the work that can be overlapped or skipped.

What it is

The critical path of a request is the sequence of spans whose durations add up to the request’s total duration. Every span on the critical path is a span whose work the request had to wait for; nothing on the path could be overlapped with anything else on the path.

A span is on the critical path if and only if:

1. The span is part of the parent chain from root to the slowest leaf.
2. The span’s duration is not fully covered by another sibling that ran in parallel.
3. The span is synchronous with respect to its parent — the parent cannot proceed until the span completes.

The complement is the non-critical work: spans whose duration is hidden inside a sibling that ran in parallel, or spans whose duration is hidden inside a wait that the parent already absorbed.

Trace: 8f1d...a3c  (POST /checkout, 1.42 s)

  T=0.00   api-gateway.checkout              [================] 1.42 s   critical
  T=0.00     checkout.cart.read              [===]            0.012 s   critical (serial)
  T=0.012    checkout.pricing.lookup         [=========]      0.083 s   critical (serial)
  T=0.095      pricing-svc.calc              [========]       0.078 s   critical (serial child)
  T=0.178    checkout.payment.charge         [==============] 1.31  s   critical (serial)
  T=0.178      payment-svc.charge            [==============] 1.30  s   critical (serial child)
  T=0.178      recommendations.suggest       [=====]          0.05  s   NOT critical (parallel)
  T=1.49     checkout.receipt.write          [===]            0.014 s   critical (serial)

The total duration is 1.42 s. The sum of the critical-path spans (cart.read, pricing.lookup, pricing-svc.calc, payment.charge, payment-svc.charge, receipt.write) is 0.012 + 0.083 + 0.078 + 1.31 + 0.014 = 1.509 s — but the receipt write happened after the payment charge, so the critical path is 0.012 + 0.083 + 0.078 + 1.31 + 0.014 = 1.497 s, of which 1.31 s was the payment-svc charge and the rest was the synchronous overhead. The recommendations call overlapped with the payment charge and is not on the critical path; its 50 ms of duration is hidden inside the parent’s wait.

Why a sysadmin cares

Three operational pains are specific to confusing the longest span with the bottleneck.

1. The fix that does not change user-visible latency. The team optimises the longest span; the user-visible latency is unchanged because the optimised span was off the critical path. The team concludes optimisations “do not work” when the truth is that they optimised the wrong span.
2. The dependency team that is wrongly accused. The dependency team is asked to fix the slowest call; the call was running in parallel with another call and the user-visible latency was bounded by a different call. The dependency team invests in latency reduction that improves no one’s experience.
3. The opportunity that is missed. The non-critical work is the candidate for parallelisation. A serialisation that was assumed to be necessary because the spans looked sequential is actually a join point that can be made non-blocking.

How it works

The critical path is computed from the span tree, the start times, and the durations. The algorithm:

1. Find the root span; its duration is the total request duration.
2. Walk the children of the root in time order. The first child’s start time is the root’s start time; the first child’s end time is the first child’s start time plus its duration.
3. For each child, check whether its start time and end time fall inside the parent’s wall-clock interval. If yes, the child is on the critical path of the parent. If no, the child’s work was already covered by an earlier sibling or by the parent’s own wait.
4. Recurse: a child that is on the critical path has its own children. Walk each child that is on the critical path.
5. The leaf at the end of the chain is the bottleneck — the span that took the most time on the critical path.

The Tempo trace UI performs this computation on the rendered trace and shades the critical-path spans in a darker colour than the off-path spans. The Tempo API exposes the same information through the trace structure; the cost is a single span tree walk per trace.

The TraceQL structural filter

The Tempo query language supports a structural operator that expresses the parent-child relationship directly:

{ resource.service.name = "checkout" } >> { name = "pg.charge.tx" }

The >> operator selects traces whose root span (or any span) has a descendant matching the right-hand set. The “descendant” is the critical-path child of the matching ancestor.

For the slowest-leaf query, the operator becomes:

{ resource.service.name = "checkout"
  && span.http.status_code = 504
  && duration > 1s
} >> { name =~ ".+charge.+" }

The query returns traces where the checkout path produced a 504, the checkout span itself took over 1 s, and one of the descendant spans is a charge call. The leaf that bounded the request is the rightmost match.

The pipeline operator | adds aggregation:

{ resource.service.name = "checkout" } >> { name =~ ".+charge.+" }
  | select(name, duration, http.url)

The aggregation returns the name, duration, and HTTP target of the charge call. The operator combines the structural relationship with the per-span attribute selection in a single query.

Parallel vs sequential children

The difference between parallel and sequential children is the difference between critical and non-critical work.

Sequential children (parent waits for each):

  Parent timeline:  [-----child A-----][-----child B-----]
  Parent duration:  child A + child B
  Critical path:    child A, child B

Parallel children (parent waits for the longest):

  Parent timeline:  [-----child A-----]
                    [---child B---]
  Parent duration:  child A
  Critical path:    child A
  Non-critical:     child B

In the parallel case, the parent’s duration is the longest child. The shorter child is off the critical path. Optimising the shorter child has no effect on the parent’s duration; optimising the longest child does.

The OTel SDK does not record the wall-clock relationship between sibling spans explicitly. The parent span records its own start and end; the child spans record their own start and end. The relationship must be reconstructed from the timestamps.

The async boundary

The asynchronous boundary is the place where a child span does not block its parent. In OTel, a span of kind CLIENT is synchronous by convention; a span of kind PRODUCER or CONSUMER is asynchronous. An asynchronous span represents work that is not on the critical path of the request that initiated it.

Sync client (blocks parent):
  span.kind = CLIENT
  parent.duration >= child.duration
  child is on critical path (unless overlapped with a sibling)

Async producer (does not block parent):
  span.kind = PRODUCER
  parent.duration is independent of child.duration
  child is off critical path of the parent request

The distinction matters when the trace shows a span that took 30 seconds but the parent request took 1 second. The 30-second span is asynchronous work that happened in the background; optimising it is a separate project from optimising the request latency.

Under the hood

How to configure it

The critical path is a read-side computation; there is no configuration to enable. The configuration that makes the computation meaningful is the instrumentation that records the wall-clock relationship correctly.

For a service that fans out into parallel work, the OTel SDK context propagation captures the parent-child relationship automatically. The discipline is to make sure every parallel unit has its own span and that the parent span’s duration is the actual wall-clock duration, not the sum of the children’s durations.

import asyncio
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def fetch_recommendations(user_id):
    with tracer.start_as_current_span("recommendations.suggest") as span:
        span.set_attribute("user_id", user_id)
        # Simulated work; off the critical path of the parent
        await asyncio.sleep(0.05)
        return ["item-1", "item-2"]

async def charge_payment(order_id):
    with tracer.start_as_current_span("payment-svc.charge") as span:
        span.set_attribute("order_id", order_id)
        await asyncio.sleep(1.3)
        return {"status": "ok"}

async def checkout(order):
    with tracer.start_as_current_span("checkout.process") as span:
        span.set_attribute("order_id", order["id"])

        # Critical path
        with tracer.start_as_current_span("checkout.payment.charge"):
            await charge_payment(order["id"])

        # Off the critical path (in a real system, fire-and-forget)
        asyncio.create_task(fetch_recommendations(order["user_id"]))

        # Critical path
        with tracer.start_as_current_span("checkout.receipt.write"):
            await write_receipt(order)

The discipline is that asyncio.create_task does not block the parent. The span for recommendations.suggest is started but the parent does not wait for it; the span’s duration is recorded when the task eventually completes, well after the parent has closed.

How to validate it

Two checks confirm the critical-path view is live.

# READ-ONLY: confirm the trace has a critical-path rendering.
tempo-cli trace show 8f1d...a3c \
  --include-critical-path 2>&1 \
  | jq '.resourceSpans[].scopeSpans[].spans[] | {name, duration, critical: .criticalPath}'

{"name": "api-gateway.checkout", "duration_ms": 1420, "critical": true}
{"name": "checkout.cart.read", "duration_ms": 12, "critical": true}
{"name": "checkout.pricing.lookup", "duration_ms": 83, "critical": true}
{"name": "checkout.payment.charge", "duration_ms": 1310, "critical": true}
{"name": "payment-svc.charge", "duration_ms": 1300, "critical": true}
{"name": "recommendations.suggest", "duration_ms": 50, "critical": false}
{"name": "checkout.receipt.write", "duration_ms": 14, "critical": true}

The recommendations.suggest span is marked as not on the critical path. The trace view confirms the visual shading matches the data.

The TraceQL query that returns only critical-path spans:

{ resource.service.name = "checkout" && duration > 1s }

Returns the root span and its critical-path child, the payment-svc charge. The non-critical span (recommendations.suggest) is in the trace but is not in the filtered result set because its duration is below the threshold.

How it can fail

Five failure modes specific to the critical-path view.

1. The parallel span that looks serial. A span fan-out executes sequentially in the code but is recorded with overlapping timestamps because the SDK mis-recorded the parent start time. Symptom: the trace view shades spans as on the critical path that are actually parallel. Cause: a custom SDK wrapper that calls span.end() late. The fix is to use the SDK’s standard context manager or with-style span lifecycle.
2. The async work that blocks the parent. A span of kind PRODUCER is recorded with the parent’s duration including the producer’s work. Symptom: the parent duration is inflated; the critical path includes work that was supposed to be asynchronous. Cause: the application awaits the producer’s future before closing the parent span. The discipline is to close the parent span before awaiting background work.
3. The synchronous span that is recorded as async. A span of kind INTERNAL is recorded with parent.duration shorter than child.duration. Symptom: the child “could not have run inside the parent” warning fires. Cause: a clock skew between the SDK and the collector, or a span that was recorded after the parent had already ended.
4. The critical path that changes shape under load. A request under load takes 1.5 s; under low load it takes 200 ms. The slowest span is different at the two loads. Symptom: the same logical request has different bottlenecks at different loads. Cause: under load, the application thread pool queues; the queue time is recorded as part of the first span that runs on the pool. The critical path includes the queue time as part of the first on-pool span; under low load the queue time is zero.
5. The critical path that is hidden by a generic span name. The slowest leaf is named process or work. Symptom: the on-call engineer cannot identify the bottleneck. Cause: a custom span was named with a generic placeholder. The discipline is the same as for the dependency view: use the OTel semantic conventions.

How to troubleshoot it

When the critical path looks wrong, the order matters.

1. Inspect the span kind. CLIENT and INTERNAL spans are synchronous with their parent; PRODUCER and CONSUMER spans are asynchronous. A span whose kind does not match its actual semantics produces a wrong critical path.
2. Inspect the start times. Two sibling spans whose start times are equal are parallel. Two siblings whose start times differ by the first’s duration are serial. The trace view shows the timeline; the critical-path computation matches it.
3. Inspect the parent duration. A parent whose duration is less than a child’s duration is recording the wrong clock values. Either the SDK clock has drifted or the span is not actually inside the parent.
4. Compare against the histogram. The metric histogram shows the population latency distribution; the trace shows one request. If the slowest leaf in the trace is not the dominant leaf across the population, the trace is not representative; the next-slowest leaf may be the real bottleneck.
5. Walk the timeline visually. The trace view’s timeline rendering is the ground truth. A span that is visually off the critical path but is computed as on the path is a bug; a span visually on the path but computed as off is a configuration issue.

Security implications

The critical-path view exposes the dependency chain. A trace that says “this request’s critical path was checkout → payment-svc → pg-primary” is useful for latency; it is also a map of the synchronous dependencies. The discipline is the same as for the dependency view:

• Treat the dependency map as semi-sensitive. The map reveals the internal call graph. The trace backend’s ACL should match the data classification of the most-sensitive downstream dependency.
• Apply redaction at the SDK before the span is exported. The critical-path view does not introduce a new leak surface; it is a reorganisation of the same spans. The redaction discipline is unchanged.

Performance implications

The critical-path computation is on the read path; the cost is a single tree walk per rendered trace. The cost is bounded by the depth and breadth of the span tree. For a typical request with 10-20 spans, the computation is a few microseconds.

The performance gain from identifying the right bottleneck is significant. Optimising a non-critical span is wasted effort; optimising the critical span produces a measurable latency reduction. The discipline is to spend the investigation time on the critical-path view before the optimisation time.

Production guidance

• Train the team to look for the critical path, not the longest span. The mental model is “what did the request have to wait for” rather than “what was the slowest span”. The two are different in the parallel case.
• Use the OTel span kind correctly. Synchronous work is CLIENT or INTERNAL; asynchronous work is PRODUCER or CONSUMER. The kind is the signal the critical-path computation uses to distinguish the two.
• Record wall-clock start times accurately. The critical-path computation depends on the parent and child start times being actual wall-clock values, not adjusted values. Use the SDK’s standard start_as_current_span or equivalent; do not manually set start times.
• Look for non-critical work as candidates for parallelisation. A span that is consistently off the critical path is a candidate for fire-and-forget or for being moved to a background worker.

Verification

You should now be able to answer:

• What two conditions must hold for a span to be on the critical path?
• Why is the longest span in a trace not always the bottleneck?
• What is the TraceQL operator that selects traces by ancestor / descendant relationship, and what does it return?
• How does the OTel span kind distinguish synchronous work from asynchronous work?
• What is the first thing to inspect when the critical path looks wrong?

Quiz