Manual vs Automatic Instrumentation

A four-service system goes live on a Friday. The Java service is already covered by the OpenTelemetry Java agent and produces spans for every HTTP, JDBC and Kafka call. The Go service has nothing. The Python batch script has nothing. The Node front-end has nothing. At 02:00 on Saturday the front-end starts returning 503s for one percent of requests, the batch job runs twice as long as usual, and the only signal in Tempo is from the Java service.

The team now has a question that is independent of the incident: for each of these services, which kind of instrumentation should have been in place before the incident? The answer is not the same for all four. The Java service is already done. The Node front-end needs the @opentelemetry/auto-instrumentations-node loader. The Go service needs otelhttp middleware applied at the framework boundary. The Python batch needs the autoloader plus a couple of spans of its own. The lesson is not “auto is better” or “manual is better”; it is “pick the right layer for each service, and know which gaps you accepted when you picked.”

What it is

OpenTelemetry’s documentation splits instrumentation into two categories that are explicit about the implementation boundary:

• Code-based instrumentation (the term the spec prefers over “manual”). The developer writes API calls into the application: tracer.startAsCurrentSpan('checkout'), otelhttp.NewHandler, getTracer().spanBuilder().startSpan(). The application’s own code is the source of spans.
• Zero-code instrumentation (the term the spec prefers over “auto”). The SDK is attached without editing the source: bytecode rewriting in the JVM, a launcher script for Python, monkey-patch on require for Node.js, or eBPF probes for Go. The application’s own code is unchanged; spans come from the libraries it uses.

The two are not layers of the same thing; they are layers of different things. Manual instrumentation talks about your domain (“order validated”, “fraud score computed”). Auto instrumentation talks about the libraries you call (“HTTP GET /api/v1/orders”, “SELECT FROM orders”). Both end up in the same trace. The lesson’s first question is which of those two stories each service needs, and the second is which the team can actually maintain.

Why a sysadmin cares

A sysadmin who inherits an instrumentation estate is going to spend the first week mapping it: which services have a -javaagent flag, which have a OTEL_SERVICE_NAME env var, which have nothing. The map drives the runbook. If the map is wrong, the runbook is wrong, and the first incident is the one that proves it.

Auto-instrumentation has the lowest time-to-first-span because no code change is required. It also has the smallest blast radius per service because the rollout is a flag, not a PR. The trade-off is control: an instrumentation library can only describe what its target library exposes, and the auto agent cannot describe the business meaning of a method. Manual instrumentation has the opposite trade-off: it costs more per service to install, but it captures the names that the engineering team actually searches for in Tempo.

How it works

The mental model has two layers and one shared convergence point.

+--------------------------------+   +-------------------------------+
| Manual (code-based)            |   | Zero-code (auto)              |
|                                |   |                               |
|  tracer.startAsCurrentSpan(    |   |  -javaagent:otel.jar         |
|    "order.validate")           |   |  opentelemetry-instrument     |
|  span.set_attribute("order.id",|   |  otelhttp.NewHandler          |
|                   "ord-123")   |   |  @opentelemetry/instrument    |
|                                |   |  ion-auto-instrumentations-   |
|                                |   |  node /Sdk                   |
|                                |   |  OBI (eBPF user-space probe) |
+---------------+----------------+   +-----------------+-------------+
                |                                       |
                |   both layers emit OTLP spans         |
                +---------------------+-----------------+
                                      |
                              OTLP to Collector
                                      |
                              Tempo / backend

The convergence point is the OTLP exporter. Manual and auto emit the same protocol, so both appear in the same trace. That is the property that makes the layering useful rather than a compromise.

How to configure it

The two layers are configured at different points in the lifecycle. Auto is configured at the process boundary (env vars or JVM flag). Manual is configured in the application source.

Auto: Java. The agent is attached with -javaagent and reads its configuration from environment variables. The first job is naming the service; the rest is optional.

# service.name comes from the OTEL_SERVICE_NAME env var
export OTEL_SERVICE_NAME=checkout-svc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.42.0

java -javaagent:/opt/otel/opentelemetry-javaagent.jar \
     -jar /opt/checkout-svc/checkout-svc.jar

Auto: Python. The opentelemetry-instrument script is the entry point. The same environment variables apply.

export OTEL_SERVICE_NAME=oauth-batcher
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
opentelemetry-instrument python /opt/batcher/batch.py

Manual: Go. The HTTP middleware wraps the standard library handler stack. The choice here is to instrument the request / response as one span, which is what an auto agent would do in Java, and to leave the application’s own functions for the developer to instrument by name.

package main

import (
    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/healthz", healthz)
    mux.Handle("/api/v1/orders", otelhttp.NewHandler(
        ordersHandler{}, "orders.create",
    ))
    http.ListenAndServe(":8080", mux)
}

Manual: span on a domain operation. The compensating move is to add a span that captures the business meaning, which is what auto cannot do.

from opentelemetry import trace

tracer = trace.get_tracer("checkout-svc")

def validate_order(order_id: str) -> None:
    with tracer.start_as_current_span("order.validate") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.channel", "web")
        # ... call the validation pipeline

The two layers coexist inside the same process: the otelhttp middleware opens the parent span (“POST /api/v1/orders”), and the start_as_current_span("order.validate") opens the child.

How to validate it

The validation matrix is the same regardless of which layer produced the span. What you are checking is that the trace contains a service-named span, that the parent is the trace root, and that the export reached Tempo.

ps -ef | grep '[o]pentelemetry-javaagent.jar'
docker logs --tail 200 checkout-svc | grep -i 'opentelemetry' | head

OTEL_PYTHON_LOG_LEVEL=debug opentelemetry-instrument python my_app.py 2>&1 | head -30

curl -s -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' http://checkout-svc:8080/api/v1/orders/1234
tempo-cli query --service checkout-svc --limit 1

$ tempo-cli query '{ resource.service.name = "checkout-svc" }'

{
"traces": [
  {
    "traceID": "0af7651916cd43dd8448eb211c80319c",
    "root": "POST /api/v1/orders",
    "spans": [
      "POST /api/v1/orders (root)",
      "  order.validate",
      "    orders.POSTGRES query",
      "    payments-svc.POST /authorise"
    ]
  }
]
}

How it can fail

Six failure shapes appear repeatedly when manual and auto instrumentation are managed by the same team.

1. The auto agent is missing on the service that needed it. The team’s checklist is “auto for all Java” but the new service is Python. The team forgets the launcher. Tempo has no spans for the service. Symptom: service-mesh metrics show latency but Tempo is empty for the service name.
2. Manual spans are present but the service name is wrong. The developer wrote tracer = trace.get_tracer("orders") and forgot OTEL_SERVICE_NAME. Spans land in Tempo under unknown_service:python. Symptom: Tempo returns nothing for { resource.service.name = "checkout-svc" }.
3. Auto and manual name the same span differently. The otelhttp middleware opens a parent named POST /api/v1/orders and the developer opens a child named PostOrder. A debugging dashboard that filters by name = "POST ..." misses the children. Symptom: the trace looks “incomplete” — roots without children with the expected name.
4. The patch happens too late. For Python and Node.js, the autoloader has to run before the target library is imported. If the application does import requests at module top before the agent runs, the requests calls are unpatched. Symptom: the application’s outbound HTTP calls appear without spans, while the inbound HTTP entry-span is present.
5. The agent pulls in a transitive dependency that the application does not test. The Python opentelemetry-instrument script patches urllib3 to a version OpenTelemetry tested. The application uses a method added in a newer urllib3. The call fails at runtime. Symptom: cryptic import errors following the instrument flag; the same code works fine without it.
6. The agent is upgraded without the application’s support matrix being re-checked. The OTel Java agent supports a matrix of library versions. Upgrading one out of the matrix produces a span error at runtime, not a startup failure. Symptom: agent logs are full of “INSTRUMENTATION ERROR” lines, and the affected spans are absent.

How to troubleshoot it

The diagnostic order is the same for every language: confirm the SDK is loaded, confirm the resource attributes are correct, confirm the exporter is reachable, then confirm the spans are arriving.

# Java
jcmd $(pgrep -f checkout-svc) VM.command_line | grep -i javaagent

# Python
ps -o command= -p $(pgrep -f batch.py) | grep -i opentelemetry-instrument

# Go
docker logs checkout-svc 2>&1 | grep -i 'otel' | head

curl -s http://checkout-svc:9464/metrics | grep -E 'otel_scope|service_name'

nc -zv otel-collector 4317
nc -zv otel-collector 4318

Security implications

The Java agent has full access to the JVM’s instrumentation API. A malicious or compromised agent can read environment variables, intercept method calls, and inspect sensitive argument values that pass through instrumented libraries. Pin the agent version the same way you pin a dependency, and review upgrades the same way. The Python and Node launchers perform sitecustomize and loader work, respectively, which executes arbitrary Python / JavaScript at start-up. The trust model is the same as the runtime’s.

Zero-code instrumentation also produces traces that may contain sensitive data through the auto-instrumented libraries. The canonical example is the SQL statement captured by the JDBC instrumentation, which can include credentials in a misconfigured connection string. Treat auto-instrumentation as a new data exfiltration path and apply the same span-attribute scrubbing rules you would apply to manual spans.

Performance implications

The performance cost is real and varies by language. Java agents typically add 1-5 percent CPU overhead and a noticeable startup penalty (the agent does its bytecode rewriting as classes load). Python and Node launchers add 50-200 ms of import time and a few percent of CPU on the patched paths. Go’s OBI is roughly proportional to the number of connected sockets.

This is the trade-off that decides the gradient. The headroom calculation is “is the latency cost of the agent smaller than the latency cost of not having the trace next incident?” The answer is almost always yes for the request-serving path and sometimes no for the steady-state batch worker that already saturates the database. The decision is per service, not per platform.

Production guidance

Verification

You should now be able to answer:

• What is the difference between code-based and zero-code instrumentation in the OpenTelemetry model?
• What are the four zero-code mechanisms that the OTel project ships, and which language does each suit?
• What does the auto agent see that the manual span does not, and vice versa?
• What is the failure shape you would expect from a Java service that is missing the -javaagent flag, and how would you confirm it?
• Why is the “auto for all services” answer wrong?

Quiz