Instrumentation Not Running

A trace lookup returns trace not found. The on-call engineer follows the missing-trace chain and lands on link A. The application was deployed three weeks ago in a routine release. The release notes mention no instrumentation change. The previous trace from the same user ID, from the previous release, is in Tempo. So spans were once being emitted; now they are not. The on-call engineer opens the running pod and runs printenv | grep OTEL_. The variables are gone. The deployment manifest was updated to a base image that strips OpenTelemetry env vars on container start. The SDK was never initialised in this build. Every later link is irrelevant; fix link A.

What it is

“Instrumentation not running” is the failure shape where the OpenTelemetry SDK is not active in the application process. The application emits zero spans, regardless of what the collector and exporter are doing. Three sub-shapes exist, distinguished by where in the SDK lifecycle the failure sits:

1. SDK not loaded. The SDK package is not on the classpath, not in node_modules, not in the Go module graph, or not required at startup. The process runs without any instrumentation hook.
2. SDK loaded but not initialised. The SDK package is present, but no TracerProvider has been constructed, or the global provider has been reset to the no-op default. The SDK is dormant.
3. SDK initialised but no exporter wired. A TracerProvider exists, a Tracer exists, spans are being created in code — but no BatchSpanProcessor and no exporter are attached. Spans accumulate in memory and are dropped on process exit.

The three sub-shapes have three different fixes. The wrong fix is to attach an agent when the SDK is loaded but not initialised; that turns a missing-spans symptom into an uninitialised SDK still doing nothing, with an extra process restarted for no reason.

Why a sysadmin cares

Link A is the most common cause of a missing trace in a production fleet. The investigation time matters: every minute spent on link D or link E before ruling out link A is a minute spent on the wrong layer. The diagnostic must be cheap and must be reflexive. The reflex is “confirm link A in under sixty seconds”.

Three patterns cause link-A failures most often:

• Container base image swap. A migration from a fat-base image to a slim image drops the SDK binaries. The application builds and deploys; the SDK is gone.
• Auto-instrumentation agent disabled by env var. A defensive ops change sets OTEL_SDK_DISABLED=true to silence telemetry during an incident; the env var is never unset.
• Manual-instrumentation Tracer never registered. A refactor removes the bootstrap code that calls SdkTracerProvider and sets the global. The Tracer calls in the codebase resolve to the no-op Tracer; no spans are created.

How it works

An OpenTelemetry SDK has four runtime phases:

  Phase 1: load       — SDK package is on disk / in module graph.
  Phase 2: initialise — TracerProvider constructed and set as global.
  Phase 3: instrument — Tracer.start() called by application code
                         or by an instrumentation hook.
  Phase 4: export     — BatchSpanProcessor flushes spans to the
                         configured exporter on schedule or on
                         shutdown.

Link A failure means at least one of phases 1, 2, or 3 has not completed. Phase 4 is link D, not link A; the collector can be perfectly healthy while link A is broken.

Auto-instrumentation vs manual

Auto-instrumentation attaches at phase 1, before the application’s own code runs. Manual instrumentation attaches at phase 2 or phase 3, in code. Auto-instrumentation is the dominant pattern for production fleets because it requires no per-service code change; the failure mode for auto-instrumentation is “agent not attached at start”.

A failure in the auto-instrumentation column is the deployment manifest, the env var, or the entrypoint command. A failure in the manual column is the bootstrap code.

SDK self-observability

Each SDK exposes self-observability metrics on a known endpoint:

  +-------------------+---------------------+----------------------+
  | Runtime           | Default port        | Counter to watch     |
  +-------------------+---------------------+----------------------+
  | Java              | 9464 (configurable) | otel.sdk.span.*      |
  | Python            | 9464 (configurable) | otel.sdk.span.*      |
  | Node              | 9464 (configurable) | otel.sdk.span.*      |
  | .NET              | 8888 (configurable) | otel.sdk.span.*      |
  | Go                | application-defined | otel.sdk.span.*      |
  +-------------------+---------------------+----------------------+

otel.sdk.span.started is the canonical “spans are being created” counter. A flat counter at zero is a clean signal that the SDK is loaded but no spans are being created.

How to configure it

There is no production configuration that enables the SDK “after the fact”; the SDK is configured at deployment time. The configuration examples below cover the four common auto-instrumentation patterns.

Java agent attached

  # Dockerfile
  ENV JAVA_TOOL_OPTIONS="-javaagent:/opt/otel/opentelemetry-javaagent.jar"
  ENV OTEL_SERVICE_NAME=checkout-svc
  ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
  ENV OTEL_METRICS_EXPORTER=prometheus
  ENV OTEL_EXPORTER_PROMETHEUS_PORT=9464
  COPY opentelemetry-javaagent.jar /opt/otel/

Severity: CONFIGURATION. Rebuild and re-deploy the container image.

Python opentelemetry-instrument

  # Dockerfile
  RUN pip install opentelemetry-distro[otlp]
  RUN opentelemetry-bootstrap -a install
  ENV OTEL_SERVICE_NAME=checkout-svc
  ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
  ENV OTEL_METRICS_EXPORTER=prometheus
  ENV OTEL_EXPORTER_PROMETHEUS_PORT=9464
  ENTRYPOINT ["opentelemetry-instrument", "--traces_exporter", "otlp", \
              "--metrics_exporter", "prometheus", \
              "python", "main.py"]

Severity: CONFIGURATION. Rebuild and re-deploy.

Node SDK via NODE_OPTIONS

  # Dockerfile
  ENV NODE_OPTIONS="--require @opentelemetry/instrumentation/auto"
  ENV OTEL_SERVICE_NAME=checkout-svc
  ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
  ENV OTEL_METRICS_EXPORTER=prometheus
  ENV OTEL_EXPORTER_PROMETHEUS_PORT=9464
  RUN npm install --save @opentelemetry/api \
                     @opentelemetry/sdk-node \
                     @opentelemetry/auto-instrumentations-node

Severity: CONFIGURATION. Rebuild and re-deploy.

Go manual bootstrap

  // main.go
  func main() {
      ctx := context.Background()
      exporter, err := otlptracegrpc.New(ctx,
          otlptracegrpc.WithEndpoint("otel-collector:4317"),
          otlptracegrpc.WithInsecure())
      if err != nil { log.Fatal(err) }
      res, _ := resource.New(ctx,
          resource.WithAttributes(semconv.ServiceName("checkout-svc")))
      tp := sdktrace.NewTracerProvider(
          sdktrace.WithBatcher(exporter),
          sdktrace.WithResource(res))
      otel.SetTracerProvider(tp)
      // ... application
  }

Severity: CONFIGURATION. Re-build and re-deploy the binary. Go auto-instrumentation does not exist; manual bootstrap is the only path.

Self-observability scrape

The Prometheus / Grafana Alloy scrape against the SDK’s self-observability endpoint:

  prometheus.scrape "checkout_sdk" {
    targets = [{
      __address__ = "checkout-svc:9464",
      job         = "otel-sdk",
      service     = "checkout-svc",
    }]
    forward_to      = [prometheus.remote_write.default.receiver]
    scrape_interval = "30s"
  }

Severity: CONFIGURATION. Apply the Alloy config and reload.

How to validate it

Severity: READ-ONLY.

Confirm SDK env vars

  kubectl exec deploy/checkout-svc -- printenv | grep '^OTEL_'
  # OTEL_SERVICE_NAME=checkout-svc
  # OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
  # OTEL_EXPORTER_OTLP_PROTOCOL=grpc
  # OTEL_METRICS_EXPORTER=prometheus

A missing env var is the cheapest link-A signal that exists.

Confirm the SDK startup banner

  kubectl logs deploy/checkout-svc --since=10m | \
    grep -iE 'opentelemetry|otel|tracerprovider'
  # OpenTelemetry Java agent v2.10.0 loaded
  # TracerProvider[ io.opentelemetry.sdk.trace.SdkTracerProvider@... ]

A Java agent prints the banner at startup; Node and Python print the instrumentation patches; Go is silent unless log-level is set.

Confirm the self-observability counters

  curl -s http://checkout-svc:9464/metrics | \
    grep -E '^otel_sdk_span_started'
  # otel_sdk_span_started_count 12482

A non-zero counter confirms the SDK is loaded, initialised, and creating spans. A zero counter with no banner is “SDK not loaded”. A zero counter with a banner is “SDK loaded but no spans created”.

Confirm the batch processor flushes

  curl -s http://checkout-svc:9464/metrics | \
    grep -E '^otel_sdk_span_(ended|exported)'
  # otel_sdk_span_ended_count 12481
  # otel.sdk.span.exported_count{exporter="otlp"} 12480

A gap between ended and exported of more than a few spans means the batch processor has not flushed. A flat exported counter with a rising ended counter is link A phase 4 broken (no exporter wired).

Confirm the agent is attached to the JVM (Java only)

  kubectl exec deploy/checkout-svc -- \
    jcmd 1 VM.command_line | grep -E 'agent|javaagent'
  # -javaagent:/opt/otel/opentelemetry-javaagent.jar

A missing -javaagent argument in the JVM command line is the Java-specific link-A signal.

How it can fail

Six failure shapes, ordered by frequency:

1. Container base image swap dropped the SDK. A migration to a slimmer base image (e.g. python:3.12-slim instead of python:3.12) removed the auto-instrumentation binary or the agent jar. The application builds and runs; the SDK is absent. Symptom: no banner, no env vars, no otel_sdk_span_started metric endpoint, all three diagnostics fail at once.

2. OTEL_SDK_DISABLED=true left in env. A defensive ops change during an earlier incident set the SDK disabled. The env var was never unset. Symptom: banner absent even though the agent jar is on disk; env var visible in printenv; the SDK metric endpoint is bound but the counters are zero.

3. Auto-instrumentation bootstrap wrapper removed. A refactor of the entrypoint removed opentelemetry-instrument from the command line. Symptom: Python application runs without instrumentation; the opentelemetry-instrument binary is in $PATH but not in the actual command.

4. Manual bootstrap code removed in a refactor. A refactor deleted the SdkTracerProvider setup; the rest of the codebase still calls otel.GetTracerProvider(). The calls resolve to the no-op default. Symptom: SDK self-observability metrics are absent; application code has tracer.start() calls but no spans are produced.

5. BatchSpanProcessor not attached. The TracerProvider is built but the WithBatcher() option is missing. Spans are created and ended; they accumulate in the in-memory queue. Symptom: otel_sdk_span_started rises; otel.sdk.span.exported is flat at zero; the collector sees no spans.

6. Shutdown hook not wired. The application does not register a SIGTERM handler that calls tp.Shutdown(). On pod termination, the batch processor is not given a chance to flush. Symptom: trace IDs appear in application logs near the end of a pod’s life, but the corresponding trace is not in Tempo because the spans were dropped on exit.

How to troubleshoot it

The diagnostic order, link A first, cheapest signal first:

1. Env vars. printenv | grep ^OTEL_ inside the pod. If the canonical four are not present, the SDK was not configured at start. Re-check the deployment manifest and the base image.
2. Banner. Search the application logs for the SDK banner. If absent, the auto-instrumentation is not attached. Re-check the entrypoint command.
3. Self-observability counters. Scrape the SDK’s /metrics endpoint. A connection refused means the SDK is not loaded at all. A flat zero counter means the SDK is loaded but no spans are being created.
4. Banner present, counters flat. The SDK is loaded but no Tracer.start() is being called. Search the codebase for tracer calls; confirm a TracerProvider is set as the global.
5. span.started rises but span.exported is flat. The batch processor is not exporting. Confirm WithBatcher() is attached; confirm the OTLP endpoint env var is correct.
6. Span exporter works in steady state but not on pod shutdown. The shutdown hook is missing. Add tp.Shutdown(ctx) to the SIGTERM handler.

Security implications

Link A is rarely a security concern, but two adjacent hazards exist:

• Auto-instrumentation broadens attack surface. The Java agent, when attached, can read class files and intercept method calls. An attacker who controls the agent jar can inject arbitrary code at startup. Verify the agent jar checksum against the upstream release at build time.
• Env vars carry OTLP endpoint URLs. A misconfigured endpoint can leak spans to an attacker-controlled collector. Restrict the endpoint env var to internal DNS names; validate the endpoint in the deployment manifest.

Performance implications

The OpenTelemetry SDK’s runtime cost is small but non-zero. The auto-instrumentation agent adds 2-5% CPU overhead on typical Java services; the manual Go SDK adds under 1%. The batch processor buffers spans in memory; a buffer that is too large increases memory pressure. The default BatchSpanProcessor settings (max_queue_size=2048, schedule_delay=5s, max_export_batch=512) are tuned for moderate-volume services; high-volume services raise max_queue_size and shorten schedule_delay.

The bigger performance trap is link A phase 4: a BatchSpanProcessor that never flushes keeps the buffer in memory indefinitely. The buffer grows; the process OOMs. The fix is the shutdown hook.

Production guidance

• Always set OTEL_SERVICE_NAME and the OTLP endpoint env vars in the deployment manifest, not in the Dockerfile. Env vars in the manifest are visible in kubectl describe; env vars in the image are not.
• Always wire a SIGTERM shutdown hook to the TracerProvider. The hook is the difference between “buffered spans flushed” and “buffered spans dropped”.
• Always enable the SDK’s self-observability exporter on port 9464 and scrape it from Grafana Alloy. The otel.sdk.span.started counter is the canary for link A.
• Always pin the SDK version in the dependency manifest. An SDK upgrade that changes default behaviour is a silent change unless pinned.

Verification

You should now be able to answer:

• What is the cheapest diagnostic that confirms link A of the missing-trace chain?
• How do you distinguish “SDK not loaded” from “SDK loaded but no spans created” from “SDK loaded and spans buffered but not exported”?
• Which self-observability counter is the canonical “spans are being created” signal?
• Why is a SIGTERM shutdown hook attached to the TracerProvider?
• What is the difference between auto-instrumentation and manual bootstrap, and how does each fail differently?

Quiz