SDK Deployment

A team is rolling out a new Python service. The Docker image bakes the SDK package at build time. The Helm chart sets OTEL_SERVICE_NAME=checkout-svc at runtime. The Kubernetes downward API sets OTEL_RESOURCE_ATTRIBUTES=k8s.pod.name=... at runtime. The OTel Operator injects the SDK at runtime through an admission webhook. The service crashes because two of the four mechanisms are configured at the same time.

The lesson is about the four deployment patterns and the order of precedence. The four patterns are not rivals; they are layers, and the wrong combination silently does the wrong thing.

What it is

OpenTelemetry SDK deployment is the discipline of getting the SDK into the application process, with the right configuration, in a way that survives the deployment. The deployment is one of four patterns, and the choice between them is the platform’s decision.

• Build-time in-process. The application’s requirements.txt / package.json / go.mod / pom.xml includes the SDK packages. The application code calls TracerProvider(...) at start-up. The configuration is shipped with the image.
• Build-time zero-code. The application’s image includes the auto-instrumentation agent (Java jar, Node loader, Python launcher). The configuration is environment variables that the agent reads at start-up.
• Runtime injection. The OpenTelemetry Operator for Kubernetes injects the SDK and the agent into the application’s pod via an admission webhook. The application is unaware; the agent is injected at pod-creation time.
• Runtime sidecar. Grafana Alloy or the OpenTelemetry Collector is deployed as a sidecar that scrapes the application’s metrics and logs and forwards traces. The application’s process is untouched.

The four patterns coexist. The lesson is the boundary between them and the place where each one is the right answer.

+-----------------------+   +-----------------------+
| Build-time in-process |   | Build-time zero-code  |
| (Python import, Go    |   | (Java agent, Node     |
|  init, Java SDK)      |   |  loader, Python       |
|                       |   |  launcher)            |
+-----------------------+   +-----------------------+
              |                            |
              +-------------+--------------+
                            |
                            v
+-----------------------+   +-----------------------+
| Runtime injection     |   | Runtime sidecar       |
| (OTel Operator)       |   | (Alloy, OTel          |
|                       |   |  Collector)           |
+-----------------------+   +-----------------------+

The right pattern is per platform. The lesson is the selection criteria.

Why a sysadmin cares

The deployment is the moment the SDK’s defaults are overridden. Everything the team relies on in production — the service name in the index, the deployment environment on the alert, the pod name on the trace — is set by the deployment pattern and the resource detector that runs inside it.

The failure shape of a misconfigured deployment is silent. The application runs, the spans arrive, the index grows. The on-call engineer cannot find the service by name, or the dashboard does not filter by environment, or the pod-level attribute is missing. The platform is “healthy” according to the metrics; the operator’s experience is “the platform is broken”.

How it works

The four deployment patterns share a common boundary: the OTel SDK’s resource detector runs at start-up, populates the Resource attributes, and the TracerProvider reads the environment variables to fill the rest. The interaction between the four is the lesson.

Build-time code              Build-time package
  (manual spans)             (SDK packages)
       |                            |
       +-------------+--------------+
                     |
                     v
                 Application start
                     |
                     v
       Resource detector runs
       (env vars, OTEL_RESOURCE_ATTRIBUTES,
        process info, host info, k8s info)
                     |
                     v
              TracerProvider
              (Resource, Sampler,
               SpanProcessor, Exporter)
                     |
                     v
              OTLP to Collector

The resource detector is the bridge. The detector runs at SDK initialisation, reads the environment, and emits the attributes that the backend will search by. The detector that comes with the SDK emits a small set of attributes (service, process, host). The detector that the OTel Operator can wrap adds the Kubernetes attributes (k8s.pod.name, k8s.namespace.name).

How to configure it

The configuration is per pattern. The four example configurations below are minimum-viable.

Build-time in-process (Python).

# app.py / observability.py
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.semconv.resource import (
    ResourceAttributes,
)

resource = Resource.create({
    ResourceAttributes.SERVICE_NAME: "checkout-svc",
    ResourceAttributes.SERVICE_VERSION: "1.42.0",
})

provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)

The SDK ships with the Resource.create factory. The factory reads OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME and merges them with the explicit dictionary. The explicit dictionary wins on conflict.

Build-time zero-code (Java).

# Dockerfile
ENV OTEL_SERVICE_NAME=checkout-svc
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
ENV JAVA_TOOL_OPTIONS="-javaagent:/opt/otel/agent.jar"

The Java agent reads the environment at start-up. The agent detects the JVM runtime environment and the container environment; the resulting resource includes the process and host attributes.

Runtime injection (OTel Operator).

# Instrumentation resource
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
spec:
  exporter:
    endpoint: http://otel-collector:4317
  resource:
    service.name: checkout-svc
  propagators:
    - tracecontext
    - baggage
  samplers:
    - arg: "0.1"
      type: parentbased_traceidratio

The Operator injects the SDK and the agent into the pod via an admission webhook. The application is unaware; the resource attributes are set by the Operator and override the application’s own configuration.

Runtime sidecar (Grafana Alloy).

// alloy.river
otelcol.exporter.otlp "default" {
  client {
    endpoint = "tempo:4317"
  }
}

otelcol.receiver.otlp "default" {
  output {
    traces = [otelcol.exporter.otlp.default.input]
  }
}

The sidecar pattern is the right answer for services that cannot be modified. The application’s traces are forwarded to the Collector, which sets the resource attributes for the backend.

How to validate it

The validation is the same triplet as the previous lessons, plus the resource attribute audit.

tempo-cli trace <trace_id> | jq '.resourceAttributes'

opentelemetry-instrument --help 2>&1 | grep -i detector
ps -eo args | grep '[o]pentelemetry-instrument'

$ tempo-cli trace 0af7651916cd43dd8448eb211c80319c | jq '.resourceAttributes'

{
"service.name": "checkout-svc",
"service.version": "1.42.0",
"deployment.environment": "prod",
"k8s.pod.name": "checkout-svc-7d9f8c-xkz7v",
"k8s.namespace.name": "shop",
"process.pid": 17,
"host.name": "checkout-svc-7d9f8c-xkz7v",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.language": "python",
"telemetry.sdk.version": "1.27.0"
}

env | grep -E '^OTEL_'
kubectl exec deploy/checkout-svc -- env | grep -E '^OTEL_'

kubectl get pod -l app=checkout-svc -o jsonpath='{.items[0].spec.initContainers[*].name}'
kubectl get pod -l app=checkout-svc -o jsonpath='{.items[0].metadata.annotations}' | jq

How it can fail

Six failure shapes recur. The first three are misconfigurations; the last three are operational.

1. The service name is the language default. The application does not set service.name and the OTEL_SERVICE_NAME env var is unset. The trace lands in unknown_service:python. Symptom: Tempo is empty for the service name; the SRE dashboard shows the service as missing.
2. The deployment environment is hard-coded. The application sets deployment.environment = prod in the code. The staging cluster runs the same image. The staging trace lands in deployment.environment=prod. Symptom: the staging alerts include prod traffic.
3. Two resource sources conflict. The OTel Operator sets service.name=checkout-svc. The application also sets service.name=orders-api. The Operator wins by precedence. The application’s name is lost. Symptom: the dashboard for orders-api is empty.
4. The Operator webhook is bypassed. The team uses kubectl apply on a pod that has not been labelled. The Operator does not inject the SDK. The pod runs without instrumentation. Symptom: the service is missing from Tempo. The pod’s logs are clean.
5. The runtime sidecar is misconfigured. The Alloy sidecar does not have the OTLP receiver enabled. The application’s traces are sent into the void. Symptom: the Collector has no incoming traces; the application logs no errors.
6. The version attribute is wrong. The application reads service.version from a build-time constant that is updated by CI. The deployed image is rolled out with a five-minute-old version. The trace is correctly tagged but the version is wrong. Symptom: the dashboard filters by the wrong version; the on-call engineer does not see the right code.

How to troubleshoot it

The diagnostic order is “is the SDK in the process?”, “is the resource detector running?”, “is the env var set?”, “is the trace in Tempo with the right attributes?”.

# Python
python -c 'import opentelemetry; print(opentelemetry.__version__)'

# Go
docker exec checkout-svc ls /opt/checkout-svc/

# Java
jcmd $(pgrep -f checkout-svc) VM.command_line | grep -i javaagent

tempo-cli query '{ resource.service.name = "checkout-svc" } | limit 1' | jq '.resourceAttributes'

kubectl exec deploy/checkout-svc -- env | grep -E '^OTEL_|SERVICE_NAME'

Security implications

The resource detector reads the environment and the process metadata. The k8s detector reads the pod’s metadata; the detector that runs as a privileged sidecar can read the host’s metadata. The credentials the detector reads are not exposed in the trace, but the attributes the detector sets are.

The deployment environment attribute is the one that matters. A misconfigured deployment that sets deployment.environment=prod for the staging cluster causes the staging alerts to include prod traffic. The mitigation is to set the deployment environment from the cluster’s metadata, not from the application’s code.

The runtime injection pattern is the right answer for a Kubernetes platform because the admission webhook has the cluster’s metadata. The build-time in-process pattern is the right answer for a VM or bare-metal deployment because there is no cluster to ask.

Performance implications

The cost of the SDK is one-time: the resource detector runs at start-up and the rest is the regular SDK cost. The runtime injection pattern adds a small admission webhook latency to pod creation. The runtime sidecar pattern adds a sidecar process to every pod, which is a memory cost.

The build-time zero-code pattern is the cheapest at the pod level. The cost is paid at the build pipeline: the agent has to be downloaded and tested against the matrix of target library versions.

Production guidance

Verification

You should now be able to answer:

• What are the four SDK deployment patterns and which platforms is each one suited for?
• What is the resource detector, and what does it contribute to the trace?
• What is the order of precedence for the resource attributes, and why does it matter?
• Name three resource attributes that should be set by the resource detector, not by the application, and why.
• What is the failure shape of a service that is missing the service.name attribute?

Quiz