Service Metadata in Logs

A Grafana panel shows the error rate for {service="checkout"} spiked at 03:14. The on-call engineer opens Loki, queries {service="checkout"}, and the panel returns zero lines. Tempo shows no spans tagged service.name=checkout. The service is running; the metrics are flowing; the join key is wrong. Half the team’s services have service.name=checkout, half have service.name=checkout-api, and one containerised deployment has service.name=checkout-7f8c9d4d5-x7kq2 — the Kubernetes pod name has leaked into the field. The investigation cannot proceed because the platform cannot tell the services apart from their environments.

The fix is not a new dashboard. The fix is a service-naming policy enforced by a single, shared place where the resource attributes are stamped. The four attributes that matter — every service, every signal — are service.name, service.version, deployment.environment, and service.namespace. Everything else is optional, bounded, or application-specific.

What it is

A resource is the OpenTelemetry object that describes the entity producing telemetry: the service, its version, its environment, the host or pod it runs on, the process that started it. Resource attributes are set once on the TracerProvider and LoggerProvider and attached to every span and every log record the provider produces. They are the metadata that travels with the telemetry to the backend; they are the join key across signals.

The canonical attributes for this lesson:

• service.name — the human-facing name of the service. Not the hostname. Not the pod. Not the binary. The name the team uses in conversation.
• service.version — the build version. git_sha is fine; a v1.4.2-style tag is fine; a date stamp is fine. Anything that changes between builds.
• deployment.environment.name — prod-eu, staging, dev, lab. Optional but expected on every production-bound deployment.
• service.namespace — the top-level grouping the service belongs to. payments, checkout, platform. The value teams reach for when the dashboard needs to slice a hundred services into a dozen products.

Why a sysadmin cares

The same fleet produces three signals. The signals live in three data sources. Three data sources do not know they are talking about the same fleet unless they share a join key. The join key is resource attributes:

• Metrics. Prometheus identifies series by label pairs. The label most relevant for joining is job, which in Prometheus-scrape conventions is the same value as service.name (Prometheus relabelling rewrites service.name into job for the metrics series).
• Logs. Loki identifies streams by label pairs. The team convention is to put service_name (label) and deployment_environment (label) on every stream.
• Traces. Tempo groups by service.name and deployment.environment.name resource attributes by default. The “Inspect” panel in the trace UI assumes the values are the same ones Grafana sees elsewhere.

If the three signals carry different values for what should be the same identity — checkout vs checkout-api vs checkout-7f8c9d4d5-x7kq2 — the platform degenerates into three orphan worlds. The on-call engineer has to know that “checkout-api” in metrics, “checkout” in logs, and the pod name in traces are all the same service. The automation does not.

The cost of enforcing resource attributes is bounded: four strings, set once, validated on the way out. The cost of not enforcing them scales with the size of the fleet.

How it works

The OpenTelemetry SDK attaches the resource to every record at creation time:

+-------------------+
| TracerProvider    |
|  Resource         |
|    service.name     = "checkout-api"
|    service.version  = "v1.4.2"
|    deployment.environment.name = "prod-eu"
|    service.namespace = "checkout"
|    host.name         = "checkout-api-7f8c9d4d-x7kq2"
+-------------------+
        |
        |  provider -> exporter
        v
+-----------------------------------+
| Span (trace)                       |
|   resource = (above)               |
|   name     = "POST /checkout"      |
|   trace_id = "4bf92f..."           |
|   span_id  = "..."                 |
|   attrs    = {...}                 |
+-----------------------------------+
        |
        v
+-----------------------------------+
| LogRecord (logs)                   |
|   resource = (above)               |
|   body     = "checkout started"    |
|   trace_id = "4bf92f..."           |
|   severity = INFO                  |
+-----------------------------------+

Three operational observations:

1. The Resource object is built once at process start, from OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, and any detectors the SDK is configured to run. Every span and log record emitted by that process carries the same Resource.
2. The backend receives the attributes as part of the OTLP payload. Tempo groups by service.name; Prometheus (via the OTel collector) converts the attributes into Prometheus labels; Loki ingests them as structured metadata (and as a stream label, if the Alloy pipeline promotes them).
3. Changing the resource is a process restart. A running service cannot change its service.name mid-flight; attempts to mutate the Resource object after TracerProvider construction are silently ignored.

How to configure it

The application side — Go, with the OpenTelemetry SDK and a single source of truth:

import (
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func newResource(ctx context.Context) (*resource.Resource, error) {
    return resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("checkout-api"),
            semconv.ServiceVersion(os.Getenv("GIT_SHA")),
            semconv.DeploymentEnvironmentName(os.Getenv("DEPLOY_ENV")),
            semconv.ServiceNamespace("checkout"),
        ),
        // Resource detectors — host.name, process.pid, etc.
        resource.NewSchemaless(),
    )
}

Python, via environment variables that the SDK reads on start:

# /etc/checkout-api.env
OTEL_SERVICE_NAME=checkout-api
OTEL_SERVICE_VERSION=1.4.2
DEPLOY_ENV=prod-eu
SERVICE_NAMESPACE=checkout

The SDK reads these on process start and builds the Resource. A container orchestrator injects them; a configuration management tool enforces them. The application code never has to think about it.

The Alloy / OTel collector pipeline — promote the four attributes into the right places and drop anything that fails the policy:

// /etc/alloy/config.alloy
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  output { traces = [otelcol.processor.resourcedetection.default.input] }
  output { metrics = [otelcol.processor.resourcedetection.default.input] }
  output { logs    = [otelcol.processor.resourcedetection.default.input] }
}

otelcol.processor.resourcedetection "default" {
  detectors = ["env", "system", "kubernetes"]
  timeout   = "5s"
  output { traces  = [otelcol.processor.transform.allowed.input] }
  output { metrics = [otelcol.processor.transform.allowed.input] }
  output { logs    = [otelcol.processor.transform.allowed.input] }
}

// Allow-list for service.name. Anything outside drops here.
otelcol.processor.transform "allowed" {
  error_mode = "ignore"
  trace_statements {
    context = "resource"
    statements = [
      `set(attributes["service.name"], "drop") where attributes["service.name"] != "checkout-api" and attributes["service.name"] != "payments-api"`,
    ]
  }
  output { traces  = [otelcol.exporter.otlp.tempo.input] }
  output { metrics = [otelcol.exporter.otlp.prom.input] }
  output { logs    = [otelcol.exporter.loki.produu.input] }
}

otelcol.exporter.loki "produu" {
  forward_to = [loki.write.produu.receiver]
}

loki.write "produu" {
  endpoint { url = "http://loki-prod-eu.internal:3100/loki/api/v1/push" }
}

The Prometheus side — the OpenTelemetry collector’s service.name becomes a Prometheus label, conventionally relabelled to job at scrape time. The collector’s prometheus exporter handles this in the resource_attributes_as_labels block.

How to validate it

# READ-ONLY: confirm the environment variables are set on the
# process that the OpenTelemetry SDK reads.
docker exec checkout-api-7f8c printenv | grep -E '^(OTEL_SERVICE|DEPLOY|SERVICE_)'
# OTEL_SERVICE_NAME=checkout-api
# OTEL_SERVICE_VERSION=1.4.2
# DEPLOY_ENV=prod-eu
# SERVICE_NAMESPACE=checkout

# READ-ONLY: confirm a span and a log line carry the same
# service.name, deployment.environment.name, and service.namespace.
TRACE=4bf92f3577b34da6a3ce929d0e0e4736
curl -fsS -u "$TEMPO_USER:$TEMPO_PASS" \
  "http://tempo.internal:3200/api/traces/$TRACE" \
  | jq '.batches[].resource.attributes[] | select(.key | test("service|deploy"))'
# { "key": "service.name",                    "value": { "stringValue": "checkout-api" } }
# { "key": "service.version",                 "value": { "stringValue": "1.4.2" } }
# { "key": "deployment.environment.name",     "value": { "stringValue": "prod-eu" } }
# { "key": "service.namespace",               "value": { "stringValue": "checkout" } }

# READ-ONLY: confirm the same values appear on Loki lines.
logcli query --since 1h --no-labels \
  '{service_name="checkout-api",deployment_environment="prod-eu"}' \
  | head -3
# 2026-08-13T14:22:11.000Z {} checkout-api service.name=checkout-api ...
# (Loki has demoted to structured metadata after parsing.)

# READ-ONLY: confirm Prometheus has a job label that matches.
curl -fsS -G "http://prometheus.internal:9090/api/v1/label/job/values"
# ["checkout-api","payments-api","orders-api",...]

# READ-ONLY: confirm Grafana variable queries return the same
# values across data sources.
curl -fsS -u "$GRAFANA_ADMIN" \
  'http://grafana.internal:3000/api/datasources/uid/prom-prod-eu' \
  | jq '.jsonData // .'
# ... (compare $__metric label_values against the Loki $__logs_label stream label)

If step 2 returns resources with service.name=checkout-api but step 3 returns Loki lines with a different value, the Alloy pipeline is not promoting the attributes (or is mapping the wrong key). If step 4 returns checkout-api-7f8c9d4d-x7kq2 in the Prometheus job label, the application is the source of the wrong name and the collector’s resourcedetection detector is being shadowed by the application’s setting.

How it can fail

1. Pod name leaks into service.name. The application reads HOSTNAME and uses it as the service name. Symptom: Prometheus has one series per pod, series cardinality explosion, every “Inspect” in Grafana shows a different service for what should be one.
2. Environment is encoded in the name. A service is called checkout-prod-eu because the DEPLOY_ENV got concatenated into OTEL_SERVICE_NAME at deploy time. Symptom: every environment has its own service.name, every dashboard needs four queries to cover staging / prod-eu / prod-us / lab.
3. Empty or missing service.version. The application ships without OTEL_SERVICE_VERSION. Symptom: a deployment rolls out, every metric series doubles (the old version’s series and the new version’s are distinct), the dashboard shows both at once and the user cannot tell which row to read.
4. No deployment.environment.name on the resource. Symptom: staging data and production data look identical on the dashboards. A regression in staging is visible as a production alert. A spike in production is attributed to a staging deploy.
5. Different teams use different naming conventions. The platform team calls itself platform-api; the checkout team calls itself checkout. Both teams then write a service that handles both responsibilities and choose payments-processor. Symptom: the service.namespace is empty on the latter, the cross-team dashboard shows it alone, the second on-call engineer cannot tell what product it belongs to.
6. Allow-list drift. The collector’s transform processor falls behind the new service onboarding. Symptom: new services’ telemetry reaches Prometheus but does not reach Tempo, or vice versa. The team “fixes” the symptom by removing the allow-list and the contractor notebook joins the production fleet.

How to troubleshoot it

The diagnostic order for “the same service shows up under different names across our signals”:

1. Where is OTEL_SERVICE_NAME set? Inspect the systemd unit, the Kubernetes manifest, the Docker Compose file. A value that is the same in every environment is correct; a value that includes a hostname, an environment, or a build ID is wrong.
2. What does the OTel collector see? Use otelcol-contrib with --feature-gates=otelcol.print.to.log or its debug exporters to dump a single span and inspect the Resource. The values that exit the collector are the source of truth for what Tempo and Loki see.
3. What does Loki parse? Query Loki for | json | __error__="" to inspect what fields were extracted and where errors happened during parsing.
4. What does Prometheus have as a label? The resourcedetection and attributes processors rewrite attributes; the prometheus exporter’s resource_attributes_as_labels knob is the final mapping. A typo in the knob name is the second most common cause of “metrics are missing the job label”.
5. Compare across signals. Run a query for service_name="checkout-api" on Loki, a query for job="checkout-api" on Prometheus, and a search for service.name="checkout-api" in Tempo. All three should return overlapping data for the same incident window.

Security implications

Resource attributes describe the entity that produces telemetry. They are not, in general, sensitive. Two exceptions:

• service.namespace can encode business groupings that an external attacker would not otherwise know. A leaked service.namespace="acquisitions" reveals the company is actively acquiring. Promote with the same care as any business metadata.
• deployment.environment.name can confirm to an attacker that a host is production — a useful target. Match the alerting policy: dashboards that leak environment names to external viewers should be auth-gated.

Neither risk is large; neither should be ignored.

The collector-side allow-list is an enforcement concern. A misconfigured allow-list that drops legitimate services is an availability bug, not a security bug. A misconfigured allow-list that lets through unknown services is a security bug, because the unknown service’s traffic is then indistinguishable from production in the platform. Review the allow-list in the same change window as any new service onboarding.

Performance implications

Resource attributes are stamped once on the provider. The cost is negligible on the span and log path: a few dozen bytes per record, attached by reference, no allocation. The on-the-wire cost in OTLP is the attribute key/value pairs at the head of every payload; this is bounded by the cardinality of the Resource, which is small.

The performance cost is in the labels. Choosing service.name as a Prometheus label means every metric series for the service gets the value; that is one extra label dimension. Adding service.namespace is another dimension. Adding deployment.environment.name is another. Each is bounded and stable, so the cost is bounded; the total label count across all series should stay under 10 to leave headroom. Adding service.version as a label quadruples the cardinality of every metric — it changes on every deploy and is almost always the wrong choice, even though it is useful. The version belongs in a Grafana variable or the value of an annotation, not in the label set.

Loki streams whose labels include the four resource attributes have the same shape: bounded cardinality, but multiply by the number of services (100) by the number of environments (3) and the stream count is 300. Add pod_name and the stream count is the pod count, which is unbounded. The bounds are not free; review them with the team that owns the alerting policy.

Production guidance

• Enforce the four attributes at the SDK boot path. Read from environment variables or from the orchestrator’s downward API; never from code defaults.
• Promote the resource attributes to Loki stream labels with a fixed cardinality budget: service_name and deployment_environment (yes, the names Loki uses are the snake_case form the agent prepends). service_namespace is optional; promote it if your dashboards slice by namespace.
• Use the OTel collector’s resourcedetection processor for environment-level attributes (cluster, region, host) that the application should not have to know about.
• Add an allow-list at the collector’s edge. Drop unknown services from the production pipeline; bill them to the test environment instead.
• Make the four attributes part of every service’s onboarding checklist. The test in lesson 06 fails the deployment if any of the four is empty.

Verification

You should now be able to answer:

• What four resource attributes should every service stamp on every signal?
• Why is putting the pod name into service.name the wrong choice?
• Where in the OpenTelemetry pipeline does the Resource object get attached to a span or a log record?
• What is the cardinality cost of adding service.version as a Loki stream label?
• What is the diagnostic step you take when service.name is correct on spans but empty on logs?

Quiz