Auto-Instrumentation

A Python service is three years old. It has manual spans on the five domain operations that matter. A new dependency on urllib3 is added at 16:00. At 16:05 the service starts producing trace data with full HTTP request and response details — including the bearer token in the Authorization header. The manual spans did not catch this. The auto-instrumentation did, because it patched urllib3 at import time.

This is the second lesson of the instrumentation series, and it is the lesson that explains why the production default is “auto-instrument the boundary, manual-instrument the domain.” The boundary is what the team usually forgets to scrub. The domain is what the team usually needs to search.

What it is

Auto-instrumentation in OpenTelemetry is the package that hooks into the runtime of a language and starts emitting spans without any change to the application source. The implementation is different in every language because the runtime hook is different in every language:

• Java — a Java agent (-javaagent:otel.jar) that rewrites class bytecode at load time using the JVM instrumentation API.
• Python — a launcher script (opentelemetry-instrument) that imports the agent before the application and patches the standard library and supported third-party libraries via sitecustomize and import hooks.
• Node.js — a loader hook (@opentelemetry/instrumentation/auto-instrumentations-node) that patches modules on require via the Node loader API.
• Go — an eBPF-only path. OpenTelemetry eBPF Instrumentation (OBI) attaches uprobes to the running binary and reconstructs HTTP / gRPC spans from the kernel and the program’s own socket calls.

The PHP and .NET runtimes have analogous packages, but the shape of the lesson is the same: a runtime hook, a list of libraries it understands, and a configuration surface that turns the patches on or off.

Why a sysadmin cares

The justification for auto-instrumentation is operational, not aesthetic. The time-to-first-span for a new service is the difference between a 20-minute ticket (“attach the agent”) and a multi-day story (“coordinate the PRs, the SDK upgrade, the dependency matrix”). The blast radius is the difference between “roll the agent forward” and “coordinate the rollout across thirty services”.

The trade-off is that the agent knows only the libraries it ships support for. When an application uses a library the agent does not patch, the agent misses it. When the agent upgrades, the library version the application uses has to be in the agent’s support matrix. The failure shape is “the service was fully instrumented yesterday and partially instrumented today”, which is exactly the kind of regression that goes unnoticed until the next incident.

How it works

The boundary between the SDK and the application is the same in every language. What differs is the hook.

+------------------+    +------------------+    +------------------+
| Java             |    | Python           |    | Node.js          |
|                  |    |                  |    |                  |
| -javaagent       |    | opentelemetry-   |    | --loader         |
|  otel.jar        |    | instrument       |    |  @opentelemetry/ |
|                  |    | python app.py    |    |  instrumentation |
| JVM instrumentation | sitecustomize  |    |  -auto-          |
|  API rewrites    |    |  patches imports |    |  instrumentation |
|  bytecode on     |    |  of stdlib +     |    |  -node/Sdk       |
|  class load      |    |  third-party libs|    |  patches modules |
|                  |    |                  |    |  on require      |
+------------------+    +------------------+    +------------------+

+------------------+
| Go (OBI)         |
|                  |
| eBPF uprobes     |
| attached to the  |
| running binary;  |
| spans recon-     |
| structed from    |
| socket calls and |
| kernel events    |
+------------------+

The OTel project publishes the support matrix per language under “Instrumentation libraries” in each language’s docs. The Java agent supports the largest set; the OBI path supports the smallest.

How to configure it

The configuration is identical in shape — environment variables — and different in the surface that reads them. The four examples below are minimum-viable configurations.

Java. The agent jar is downloaded alongside the JVM and is the first -javaagent on the command line. The configuration lives entirely in environment variables.

# Dockerfile snippet
ADD --chown=app:app https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.10.0/opentelemetry-javaagent.jar /opt/otel/agent.jar

ENV OTEL_SERVICE_NAME=checkout-svc
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
ENV OTEL_EXPORTER_OTLP_PROTOCOL=grpc
ENV OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.42.0

ENV JAVA_TOOL_OPTIONS="-javaagent:/opt/otel/agent.jar"
ENTRYPOINT ["java", "-jar", "/opt/checkout-svc/checkout-svc.jar"]

JAVA_TOOL_OPTIONS is the resilient delivery mechanism: it is honoured by the JVM no matter how the container is started, and the agent appears in jcmd <pid> VM.command_line for the audit.

Python. The launcher is opentelemetry-instrument, which ships in the opentelemetry-distro package. The application is invoked through it.

# Dockerfile snippet
RUN pip install \
    opentelemetry-distro \
    opentelemetry-exporter-otlp
RUN opentelemetry-bootstrap -a install

ENV OTEL_SERVICE_NAME=oauth-batcher
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

ENTRYPOINT ["opentelemetry-instrument", "python", "/opt/batcher/batch.py"]

opentelemetry-bootstrap -a install installs the supported instrumentation libraries so the launcher can find them. The application is no longer the entry point; the launcher is.

Node.js. The loader is registered via --require before the application module is loaded.

# Dockerfile snippet
RUN npm install --save \
    @opentelemetry/api \
    @opentelemetry/sdk-node \
    @opentelemetry/auto-instrumentations-node \
    @opentelemetry/exporter-trace-otlp-http

ENV OTEL_SERVICE_NAME=web-spa
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

ENTRYPOINT ["node", "--require", "./tracing.js", "server.js"]

The tracing.js file is the SDK bootstrap: it registers the exporter, the resource, and the propagator.

Go (OBI). OBI is a sidecar, not a build-time thing. The deployment is a DaemonSet that runs one OBI instance per node and attaches uprobes to the service’s binary.

# Kubernetes DaemonSet snippet -- not the full manifest
spec:
  template:
    spec:
      containers:
        - name: obi
          image: otel/obi:0.10.0
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: otel-collector:4317
            - name: OTEL_SERVICE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['app.kubernetes.io/name']

The constraint is the kernel: OBI requires Linux 4.18+ with BTF, and the host’s seccomp / AppArmor policy has to permit the bpf(2) syscall. Without those, the deployment fails fast and the agent logs permission denied on the probe load.

How to validate it

The validation is the same triplet as the previous lessons: confirm the agent is loaded, confirm the resource attributes, confirm the export reached the backend.

jcmd $(pgrep -f checkout-svc) VM.command_line | grep -i javaagent | head

ps -o command= -p $(pgrep -f batch.py) | grep -i opentelemetry-instrument
OTEL_PYTHON_LOG_LEVEL=debug python -c 'import requests; requests.get("http://localhost:8080/")' 2>&1 | head -20

ps -eo args | grep '[n]ode --require'
OTEL_LOG_LEVEL=debug node --require ./tracing.js server.js 2>&1 | head -20

curl -s http://$(kubectl get pod -l app=obi -o jsonpath='{.items[0].status.podIP}'):9464/metrics | grep obi

$ tempo-cli query '{ resource.service.name = "checkout-svc" }'

{
"traces": [
  {
    "traceID": "0af7651916cd43dd8448eb211c80319c",
    "root": "POST /api/v1/orders",
    "spans": [
      "POST /api/v1/orders (root)",
      "  http.method GET (HttpURLConnection)",
      "    orders.JDBC SELECT",
      "  kafka.send orders.created"
    ]
  }
]
}

How it can fail

Six failure shapes. The first three are misconfigurations; the last three are operational.

1. The agent is missing on the JVM. The Dockerfile builds the container without the agent jar and the JAVA_TOOL_OPTIONS env var. The class loader never sees the agent. Symptom: jcmd <pid> VM.command_line does not include -javaagent. Tempo is empty for the service.
2. The Python launcher is bypassed. The application is run as python batch.py instead of opentelemetry-instrument python batch.py. The launcher never runs; the patches never apply. Symptom: the application’s outbound HTTP calls appear without spans; the inbound HTTP entry-span is present (because the web framework was patched earlier, or the auto-instrumentation library is imported separately).
3. The Node loader is not the first require. The tracing.js is loaded after the Express server has already required the HTTP module. The HTTP module is unpatched. Symptom: HTTP entry-spans are missing on the first request after restart; subsequent requests are fine.
4. The library version is outside the agent support matrix. The application upgrades to a new Kafka client that the agent does not know about. The agent logs INSTRUMENTATION ERROR for the unsupported version and skips the patching. Symptom: the Kafka client calls are missing from the trace; the agent logs are the only sign.
5. The agent upgrade is not coordinated with the SDK upgrade. The agent and the SDK are two different versions. The exporter protocol is incompatible. The SDK rejects the spans from the agent at runtime. Symptom: half the spans arrive at Tempo with one set of attributes and half arrive at Tempo with another, depending on which library emitted them.
6. The OBI deployment is missing the kernel capability. The cluster runs a hardened kernel with seccomp blocking bpf(2). OBI cannot load the uprobes. Symptom: OBI logs failed to load BPF probe: permission denied on startup; the Go service has no spans.

How to troubleshoot it

The diagnostic order is “is the agent loaded?”, “is the agent version compatible with the libraries?”, “is the path to the exporter alive?”, “are the spans arriving?”.

# Java
jcmd $(pgrep -f checkout-svc) VM.command_line | grep -i javaagent

# Python
ps -o command= -p $(pgrep -f batch.py) | grep -i opentelemetry-instrument

# Node
ps -eo args | grep '[n]ode --require'

# Go
curl -s http://$(kubectl get pod -l app=obi -o jsonpath='{.items[0].status.podIP}'):9464/metrics | grep -E 'obi_build_info|obi_exporter'

curl -s -H "traceparent: 00-0af7651916cd43dd8448eb211c80319c-aaaaaaaaaaaaaaaa-01"    http://checkout-svc:8080/api/v1/orders
tempo-cli query '{ resource.service.name = "checkout-svc" } && trace = "0af7651916cd43dd8448eb211c80319c"'

docker logs --tail 1000 checkout-svc 2>&1 | grep -i 'INSTRUMENTATION ERROR' | head

Security implications

The agent patches the standard library and the third-party HTTP / SQL / messaging libraries. The patches run in the same process as the application and have the same access to environment variables, file descriptors, and credentials.

The canonical incident is the JDBC instrumentation capturing the SQL statement, which includes the password field of a production connection string. The HTTP instrumentation captures the URL, which includes query-string parameters and sometimes bearer tokens. The trace context propagation includes the trace ID and span ID, which are not sensitive by themselves but can become a correlation handle if leaked into logs.

The mitigation is to scrub at the SDK, not at the Collector. The auto-instrumented attribute on the SQL statement is db.statement. A SpanProcessor wrapping the BatchSpanProcessor can strip values that match a known sensitive pattern before the batch queue receives the span.

Performance implications

The cost is small but real. The Java agent adds 1-5 percent CPU on the request path and a class-load overhead on the first request after start-up. The Python and Node launchers add 50-200 ms of import time and a few percent CPU on the patched paths. OBI adds a small per-socket overhead.

The cost is dominated by the exporter and the serialisation in the SDK. The agent does not change the application’s request path; it adds metadata to the libraries the application already calls. The cost is bounded by the library coverage the agent enables.

Production guidance

Verification

You should now be able to answer:

• What are the four zero-code mechanisms the OTel project ships, and which language does each suit?
• Why is the Java agent’s mechanism not viable for Go?
• What does opentelemetry-bootstrap -a install do, and why is it required before the launcher runs?
• What is the failure shape you would expect from a Python service that is run as python app.py instead of opentelemetry-instrument python app.py?
• Why does the production discipline pin the agent version in the lockfile?

Quiz