OpenTelemetry SDK

The OTel SDK is the boundary between your code and your telemetry. Everything you write about in the lesson series — manual spans, auto instrumentation, propagation, attributes — calls into one of the SDK’s interfaces. If the SDK is configured wrong, none of those things work. If the SDK is configured partially, the platform looks healthy and a specific kind of investigation breaks in a specific way.

The lesson is not a tour of every SDK method. It is the seven classes and three configuration files you have to understand to keep a fleet of services producing traces that the on-call engineer can actually search.

What it is

The OpenTelemetry SDK is the language-specific implementation of the OTel API plus the configuration surface that turns spans into wire traffic. The boundary is the same in every language:

• API — the packages the application imports. Stable, thin, no I/O. The compiler does not allow you to couple your code to anything vendor-specific.
• SDK — the packages the application initialises once at start-up. Owns the exporter, the sampler, the resource, the span processors, the batch queue.

Application code
       |
       | imports (API only)
       v
opentelemetry.api.trace.Tracer      (stable interface)
       |
       | resolves through (SDK)
       v
TracerProvider  -->  Tracer  -->  Span  -->  SpanContext
       |                  |
       | configured by    |
       v                  v
Resource           Sampler
Exporter          SpanProcessor (Simple / Batch)

The relationship is one Provider per process, many Tracers per Provider (one per instrumentation library), many Spans per Tracer, and exactly one SpanContext per active Span. The SpanContext is the 16-byte Trace ID plus 8-byte Span ID that gets serialised into the traceparent header the next lesson is going to spend its time on.

Why a sysadmin cares

The SDK is the only place where a service’s telemetry is named, sampled, and exported. Three production decisions live there:

1. What is the service called? service.name is an attribute on the Resource, not a property of the Tracer. Without it, Tempo shows the service as unknown_service:python and the engineer cannot pivot to it.
2. What is the exporter? OTLP/gRPC, OTLP/HTTP, or stdout for debugging. The exporter is the only path data leaves the process.
3. What is the sampler? A service that always samples 100 percent produces a burn rate that the backend cannot sustain. A service that samples 0 percent produces no traces at all. The production default is parent-based with a per-service rate.

If the SDK is wrong, the production cost is either “we have no data” or “we have too much data”. Both happen quietly. A failed OTLP export does not crash the application by design — the SDK is supposed to be invisible to the application’s correctness.

How it works

The span lifecycle is the core of the SDK. Every other concept exists to support or shape it.

initialised                started                   ended
    |                          |                        |
    v                          v                        v
TracerProvider --> Tracer --> Span.begin() --> span.end() --> exporter
                          |       |              ^
                          |       |              |
                          |       +-> set_attribute()
                          |       +-> add_event()
                          |       +-> record_exception()
                          |       +-> set_status()
                          +-> SpanContext (traceID, spanID, flags)

A span is started with a parent context (or no parent if it is a root). The span accumulates attributes, events, and exceptions over its lifetime. The span is ended — either manually or by leaving a with / using context manager. The exporter serialises the span to OTLP and ships it.

Two properties of this model drive everything else:

• A span’s start-clock and end-clock are captured in the SDK, not by the application. The application calls span.end(); the SDK records the wall time at that moment. If the application crashes without calling end(), the SDK drops the span. This is why every manual span has to be inside a try/finally or a context manager.
• The SpanContext is the only thing that crosses the process boundary. Attributes, events, exceptions are not propagated; the receiving side creates its own span with the parent’s SpanContext as its parent. This is what the next lesson covers.

How to configure it

The SDK is configured once at process start. The example below is Python; the Go equivalent is in the same shape with a slightly different API.

# observability.py -- imported at process start
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, OTLPSpanExporter
from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatio

resource = Resource.create({
    SERVICE_NAME: "checkout-svc",
    SERVICE_VERSION: "1.42.0",
    "deployment.environment": "prod",
})

sampler = ParentBasedTraceIdRatio(0.10)  # 10 percent of root spans
provider = TracerProvider(resource=resource, sampler=sampler)

exporter = OTLPSpanExporter(
    endpoint="http://otel-collector:4317",
    insecure=True,
)
provider.add_span_processor(BatchSpanProcessor(exporter))

trace.set_tracer_provider(provider)

The four configuration decisions are visible in the code:

• Resource. service.name, service.version, deployment.environment. The semantic-convention constants come from opentelemetry-semantic-conventions. The Resource.create factory merges in the environment-detected attributes (process, host, container) from the resource detector.
• Sampler. ParentBased(TraceIdRatio(0.10)) means “sample 10 percent of root spans, and respect the parent’s decision for child spans”. This is the production default.
• Exporter. OTLP/gRPC against the local Collector. The Collector is the right place to apply the sampling and redaction policy that the SDK should not be implementing.
• SpanProcessor. BatchSpanProcessor is the production choice; SimpleSpanProcessor is the diagnostic choice. Simple exports synchronously on every span end, which is what you want for a one-off test and never what you want in production.

The Tracer is acquired by name from the global provider:

tracer = trace.get_tracer("checkout-svc")

The string is the instrumentation library name, which becomes the otel.scope.name attribute on every span. It is searchable in Tempo and is the way to identify “which library produced this span” inside a trace.

How to validate it

The check sequence is the same as for the auto-instrumentation lesson — confirm the SDK is loaded, then confirm the resource attributes, then confirm the export.

curl -s http://checkout-svc:9464/metrics | grep -E 'otelcol_exporter_sent_spans|otelcol_exporter_queue_size'

TRACEID=$(uuidgen | tr -d '-')
curl -s -H "traceparent: 00-$TRACEID-aaaaaaaaaaaaaaaaaa-01"    http://checkout-svc:8080/api/v1/orders/1234
tempo-cli query "{ resource.service.name = "checkout-svc" && trace_id = "$TRACEID" } | limit 1"

$ docker logs --tail 200 checkout-svc 2>&1 | grep -i 'opentelemetry.sdk.trace'

[opentelemetry.sdk.trace] Resource initialized: service.name=checkout-svc, service.version=1.42.0, deployment.environment=prod
[opentelemetry.sdk.trace] Sampler: ParentBasedTraceIdRatio(0.10)
[opentelemetry.sdk.trace] SpanProcessor: BatchSpanProcessor(OTLPSpanExporter(endpoint=http://otel-collector:4317, insecure=True))

python - <<'PY'
from opentelemetry import trace
tracer = trace.get_tracer("probe")
with tracer.start_as_current_span("probe") as span:
  try:
      raise RuntimeError("boom")
  except RuntimeError as exc:
      span.record_exception(exc)
      span.set_status(trace.Status(trace.StatusCode.ERROR))
PY

How it can fail

Five failure shapes recur. The first three are misconfigurations; the last two are operational.

1. Service name is the wrong default. The Python SDK defaults to unknown_service:python and the Go SDK defaults to unknown_service:go. If the application code does not call trace.set_tracer_provider with a Resource that includes service.name, every span lands in unknown_service:*. Symptom: Tempo is empty for the service name; the SRE dashboard shows the service as “missing”.
2. The exporter is the wrong endpoint. The application reports OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 inside the container, which is the loopback of the application container, not the Collector. The SDK reaches the wrong place and the export quietly fails. Symptom: otelcol_exporter_send_failed_spans is the only counter that increments.
3. The sampler is 100 percent. A developer copies the debug example into production. The burst of traffic on Monday morning doubles the Collector’s memory and trips the local-disk limit. Symptom: Collector logs ingester.max-spool-size errors.
4. The Resource detector is wrong. The k8s resource detector is imported but the service account does not have read access to the pod metadata. The resource attributes fall back to bare host-only attributes. Symptom: the k8s.pod.name attribute is absent from spans; the on-call engineer cannot correlate the trace to a pod.
5. The SDK is not initialised. The developer adds tracer.start_as_current_span("domain") somewhere in the code, but the application imports the API package and never calls set_tracer_provider. The global provider is a no-op tracer. Symptom: the application’s spans are dropped silently; only the auto-instrumented spans appear.
6. The BatchSpanProcessor queue overflows. The Collector is down for ten minutes. The queue hits 2048. The SDK drops new spans with a log line that appears at WARN severity. The application does not see the drops. Symptom: otelcol_exporter_dropped_spans is non-zero and the trace coverage drops; the on-call engineer sees a partial trace.

How to troubleshoot it

The diagnostic order is “is the SDK doing what I think it is doing?”, which is the same question you ask of any daemon.

# Python (debug logging)
OTEL_PYTHON_LOG_LEVEL=debug python -c 'import my_app' 2>&1 | head -30

# Go
docker logs checkout-svc | grep -i 'tracerprovider'

# Java
ps -ef | grep '[o]pentelemetry-javaagent.jar'

curl -s http://checkout-svc:9464/metrics | grep -E 'service_name|service_version|deployment_environment' | head

watch -d 'curl -s http://checkout-svc:9464/metrics | grep otelcol_exporter'

Security implications

The SDK ships spans with attributes that may contain sensitive data. The Resource detector that reads /proc/self/environ will capture all environment variables, which includes database URLs and API tokens for many deployments. The JDBC instrumentation captures the SQL statement, which can include credentials in a misconfigured production connection string. The HTTP instrumentation captures the URL, which can include query-string parameters.

The production discipline is to scrub at the SDK boundary, not at the Collector. The SDK’s SpanProcessor can be wrapped in a redaction layer that strips known-sensitive patterns before the spans reach the batch queue. The Collector is too late: the application spent the wall-clock time to build the attributes, and the data is in the process heap.

Performance implications

The default BatchSpanProcessor overhead is roughly 1-3 percent of CPU on the application process, dominated by the serialisation step. The bounded queue is 2048 spans by default, which is enough for a few seconds of burst traffic at 1000 spans/s. The background worker drains the queue every 5 seconds by default; on a slow exporter this is the dominant cost.

The SimpleSpanProcessor is the right choice for unit tests and the wrong choice for production. It serialises on every span end, which means an HTTP request handler now does JSon-over-OTLP conversion synchronously before returning. That is a 5-30 percent latency hit on the request path.

Production guidance

Verification

You should now be able to answer:

• What is the difference between the OTel API and the OTel SDK?
• What does the SDK do on span.end() that the application does not do?
• Why is service.name a Resource attribute and not a property of the Tracer?
• What is the production default for the sampler, and why?
• What is the bounded queue in the BatchSpanProcessor for, and what happens when it fills?

Quiz