Trace instrumentation — manual and auto-instrumentation

Trace instrumentation is the discipline of adding tracing to the application. The manual instrumentation is the explicit code; the auto-instrumentation is the zero-code integration. This lesson walks the manual, the auto, the choice, and the production patterns.

The manual instrumentation

The manual instrumentation:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

func MyHandler(w http.ResponseWriter, r *http.Request) {
    tracer := otel.Tracer("my-app")
    ctx, span := tracer.Start(r.Context(), "MyHandler")
    defer span.End()
    
    // Custom logic
    result := doSomething(ctx)
    
    // Add attributes
    span.SetAttributes(
        attribute.String("result.type", fmt.Sprintf("%T", result)),
        attribute.Int("result.size", len(result)),
    )
}

The manual instrumentation is the explicit code.

The auto-instrumentation

The auto-instrumentation:

# Java
java -javaagent:opentelemetry-javaagent.jar -jar my-app.jar

# Python
opentelemetry-instrument python my-app.py

# Node.js
node --require @opentelemetry/auto-instrumentations-node my-app.js

# Go
import _ "go.opentelemetry.io/otel/contrib/instrumentation/net/http/otelhttp"

The auto-instrumentation is the zero-code integration.

The auto-instrumentation libraries

The auto-instrumentation libraries:

The libraries cover the standard libraries.

The propagation

The propagation:

import (
    "go.opentelemetry.io/otel/propagation"
)

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
))

The propagation is the link across services.

The resource attributes

The resource attributes:

import (
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/semconv/v1.26.0"
)

resource.New(ctx,
    resource.WithAttributes(
        semconv.ServiceName("my-app"),
        semconv.ServiceVersion("1.0.0"),
        semconv.DeploymentEnvironment("production"),
    ),
)

The resource attributes identify the service.

The custom attributes

The custom attributes:

import (
    "go.opentelemetry.io/otel/attribute"
)

span.SetAttributes(
    attribute.String("http.method", "GET"),
    attribute.Int("http.status_code", 200),
    attribute.String("user.id", "abc123"),
)

The custom attributes enrich the spans.

The events

The events:

span.AddEvent("cache_miss", trace.WithAttributes(
    attribute.String("cache.key", "abc123"),
))

The events are the time-stamped annotations.

The errors

The errors:

if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
}

The errors are recorded and the status is set.

The production patterns

The production patterns:

flowchart LR
    A[Application] --> B[Auto-instrumentation]
    B --> C[HTTP, gRPC, database]
    A --> D[Manual instrumentation]
    D --> E[Custom logic]
    C --> F[OTel SDK]
    E --> F
    F --> G[OTel Collector]

The hybrid pattern is the production.

The instrumentation challenges

The instrumentation challenges:

flowchart LR
    A[Challenge 1: Library coverage] --> B[Solution: manual instrumentation]
    C[Challenge 2: Performance] --> D[Solution: sampling, batching]
    E[Challenge 3: Privacy] --> F[Solution: attribute filtering]

The challenges are the production concerns.

The cross-course references

• The OpenTelemetry course covers the SDK.
• The Jaeger course covers the trace backend.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Use the auto-instrumentation. For standard libraries.
• Use the manual instrumentation. For custom logic.
• Configure the resource attributes. The service identity.
• Configure the propagation. The traceparent header.
• Configure the errors. RecordError, SetStatus.
• Monitor the instrumentation overhead. The performance.

The trace instrumentation is the production pattern. Operating it well is via the auto-instrumentation for standard libraries, and the manual instrumentation for custom logic.