Service Mesh Enabled

The on-call engineer opens the service mesh dashboard on Wednesday morning. A new namespace was on-boarded to Istio on Monday. The mesh telemetry was left at the default. The tracing backend now sees twice the spans for every request to the new namespace: one set from the Envoy sidecar, one set from the application SDK. Tempo disk usage has climbed faster than the forecast. The team that owns the new namespace is unaware their service is doubling the platform’s volume.

This is the service-mesh-enabled trace volume incident. A service-mesh-enabled incident is a production event in which the service mesh sidecar and the application SDK both emit spans for the same request, multiplying the trace volume without multiplying the request rate. The proximate symptom is per-namespace trace rate doubling in step with a mesh rollout; the proximate cause is almost always the mesh tracing being enabled while the application tracing is left on, with no decision about which is the source of truth.

What a service-mesh-enabled trace volume incident is

A service-mesh-enabled trace volume incident is not the same as a healthy mesh. A healthy mesh has tracing enabled at the sidecar; the application SDK is disabled or configured to respect the sidecar’s context; one trace per request is emitted, and the sidecar’s spans are the source of truth for mesh-level metadata. An incident is when both sources are emitting: the sidecar exports its own spans for the request, and the application SDK exports its own spans for the same request, with the same trace identifier. Two parallel traces arrive at the collector. The collector buffers them as if they were one trace; the result is a trace with twice the expected span count, or two traces that share the same identifier but have different parent / child relationships.

The shape is recognisable: a step change in tempo_distributor_spans_received_total that correlates with a mesh namespace rollout, and a per-service span count that has roughly doubled for the rolled-out namespace.

Why a sysadmin cares

A service-mesh-enabled trace volume incident takes the tracing platform out of budget without any change to the application code. The application team did nothing wrong; their SDK has been emitting the same spans for months. The mesh team enabled a feature on the namespace; the platform ingests twice as many spans. The platform team is paid to find out why their volume budget is exceeded without any matching traffic change. The investigation crosses three teams and three dashboards.

The cost of the incident is paid in the next two hours of the on-call engineer’s night. The cost of the next incident is paid by everyone who on-boards a namespace to the mesh without making the source-of-truth decision.

How it works

A service-mesh-enabled trace volume incident manifests at the namespace boundary. The mesh sidecar and the application SDK both emit; the collector receives; the backend stores.

Application pod
   |
   +-- application SDK (opentelemetry SDK)
   |     |
   |     +-- emits spans for HTTP server, HTTP client, DB calls
   |     +-- exports via OTLP to localhost:4317 (sidecar) or
   |     |   directly to the cluster collector
   |
   +-- Envoy sidecar (istio-proxy / linkerd-proxy)
         |
         +-- emits spans for inbound and outbound HTTP
         +-- exports via OTLP to the mesh tracing collector

Two sources emit; one trace identifier is generated (the sidecar’s, since the mesh sees the request first); the application SDK inherits the trace identifier via W3C traceparent propagation from the sidecar; both sources emit spans under the same trace_id. The collector receives both sets and either merges them into one trace (if the parent / child links are consistent) or splits them into two traces (if the parent / child links disagree, which they will, because the sidecar’s view of the parent is the inbound request and the application’s view of the parent is its own incoming server span).

The most common shape is “sidecar emits, application emits, the trace is split into two by the collector”. Tempo stores two traces per request. The per-request span count looks right; the per-trace span count is roughly half of the expected.

Under the hood

How to configure it

The fix for a service-mesh-enabled trace volume incident is to pick one source of truth per service. The most common production pattern is “application SDK is the source of truth; sidecar tracing is disabled”.

# Istio: disable mesh tracing for the namespace.
# Apply via Istio Telemetry API or kubectl.
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: disable-mesh-tracing
  namespace: shop
spec:
  tracing:
  - providers:
    - name: none
    disableForPodOwners:
      # The application SDK is the source of truth.
      # The sidecar does not emit; the SDK inherits the
      # trace identifier via W3C traceparent.
      match:
        labels:
          app.kubernetes.io/part-of: shop

# Linkerd: disable proxy-level tracing via Helm values or
# Linkerd Viz annotation.
apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: checkout
  namespace: shop
spec:
  # Disable per-route tracing for the checkout service.
  routes: []

The application SDK then emits; the sidecar does not. The collector receives one trace per request.

# app.py  -- the application SDK is the source of truth
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "checkout",
    "service.namespace": "shop",
    "service.version": "1.42.0",
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://otel-collector:4317")
    )
)
trace.set_tracer_provider(provider)

How to validate it

When a namespace is suspected of doubling the trace volume, the first read is the per-namespace span rate and the sidecar’s own metric.

# READ-ONLY: per-service trace rate in Tempo by namespace.
# Tempo exposes a generated metric for trace count by
# service.namespace.
curl -s http://tempo:3200/metrics \
  | grep '^tempo_ingester_traces_per_service'

# READ-ONLY: Istio sidecar trace export rate. The Envoy
# sidecar emits a metric per pod for traces emitted.
kubectl exec -n istio-system deploy/istiod -- \
  curl -s http://localhost:15014/stats \
    | grep -E 'tracing|spans'

# READ-ONLY: application SDK trace rate. The OTel SDK
# exposes spans per second via its own metric, if the
# application is configured to emit a self-telemetry metric.
# Alternatively, the OTel Collector receiver counter
# identifies the source by the exporter's `otlp` source.
curl -s http://otel-collector:8888/metrics \
  | grep '^otelcol_receiver_accepted_spans'

If the per-namespace rate has roughly doubled and the sidecar’s trace export metric is also climbing, both sources are emitting. The fix is to disable one.

How it can fail

The six failure shapes that account for the great majority of service-mesh-enabled trace volume incidents:

1. Both sources enabled by default. Istio’s telemetry.v1.Tracing is enabled for the mesh control plane, and the application namespace has no override. The application SDK is also enabled. Both emit. Per-namespace rate roughly doubles in step with the namespace on-boarding.
2. Application SDK ignores sidecar context. The application’s OTEL_PROPAGATORS is not configured to respect W3C traceparent; the SDK generates its own trace identifier. The sidecar’s spans and the application’s spans have different trace_ids. Tempo stores two unrelated traces per request.
3. Two collectors, two traces. The sidecar exports to the Istio add-on collector; the application exports to the cluster collector. The traces have the same trace_id but different tenants. Tempo’s search index shows two distinct traces per request.
4. Sidecar emit-on-error only. A team configures the sidecar to emit only on error (telemetry.v1.Tracing.sampling: 100 only for status_code error). The application SDK is also configured with a probabilistic sampler at 100 percent. Errors are emitted twice; healthy traces are emitted once. Per-trace span count on errors is roughly double.
5. SDK upgrade changes propagation defaults. An OTel SDK upgrade changes the default propagator from W3C TraceContext to B3. The sidecar still emits W3C; the application emits B3. Trace identifiers diverge. Tempo stores two unrelated traces per request.
6. Sidecar meshConfig defaults re-enabled. A platform upgrade resets the mesh control plane’s tracing configuration to the default. A namespace that had tracing.providers: [none] in a Telemetry resource inherits the new default. The sidecar emits; the application emits; the volume doubles.

How to troubleshoot it

The diagnostic order is fixed: confirm the symptom, locate the namespace, locate the source, then act.

1. Confirm the symptom. Read tempo_distributor_spans_received_total and per-service span rate. Compare to the baseline. A rate that has doubled for a specific namespace is the textbook symptom.
2. Locate the namespace. Inspect the mesh telemetry resources. A namespace with no Telemetry override inherits the mesh control plane default. The default may be “sidecar emits”.
3. Locate the source. Read the sidecar’s tracing exporter metric and the application’s SDK exporter metric. The source whose metric is climbing is the second source.
4. Form a hypothesis. Identify when the per-namespace rate started climbing. The hypothesis is almost always correlated with a namespace on-boarding or a mesh upgrade in the change log.
5. Find evidence. Cross-reference with the mesh control plane events. The culprit is almost always a Telemetry resource that is missing or that was overridden by a recent upgrade.
6. Act. Apply a namespace-scoped Telemetry resource that disables sidecar tracing for the affected namespace, or disable the application’s SDK exporter. Watch the per-namespace rate fall back to the baseline.

Security implications

A service-mesh-enabled trace volume incident doubles the exposure of request data in the tracing backend. The sidecar’s spans carry HTTP target, status code, and duration. The application’s spans carry the same plus business-logic attributes. Two sources means two copies of the same data in the retention window. The fix is a single source of truth and a documented retention window.

A sidecar that emits to a third-party collector (the Istio add-on to Jaeger or Zipkin pattern) routes request data out of the cluster’s tracing backend. The fix is a documented routing decision; the mesh telemetry resources control the destination, not the source.

Performance implications

A service-mesh-enabled trace volume incident roughly doubles the per-request tracing cost. The sidecar emits two spans per request (inbound and outbound); the application emits the rest. Doubling the source count doubles the per-request span count; the per-byte cost is unchanged; the per-request cost is doubled. The distributor, ingester, and compactor all see the doubled rate; the only fix is at the source.

Production guidance

• Make the source-of-truth decision at namespace on-boarding. The decision is per service, not per platform.
• Document a Telemetry resource pattern. The pattern is “sidecar disabled, application SDK enabled” for services that have rich business logic; “sidecar enabled, application SDK disabled” for services that do not.
• Alert on per-namespace trace rate. A climb in step with a namespace on-boarding is the canonical signal that both sources are emitting.
• Pin the SDK propagator to W3C TraceContext. B3, Jaeger, and other propagators are not compatible with the sidecar’s default.
• Pin the mesh control plane tracing configuration. A platform upgrade that resets the mesh control plane default re-enables sidecar tracing for every namespace that does not have an override.

Verification

• What is the difference between “sidecar emits, SDK emits” and “sidecar emits, SDK disabled”?
• What is the diagnostic order when the per-namespace trace rate has doubled compared to the baseline?
• What is the most common cause of a service-mesh-enabled trace volume incident?
• Where in the mesh configuration is the source-of-truth decision enforced?

Quiz