The Workload Layer

An SRE opens a Grafana dashboard at 03:20. The checkout service panel is red. They open the trace view and Tempo returns zero results for the time window. They open the logs and the service.name="checkout" filter is empty. The metrics are present, but both other signals are missing for the same service. The on-call engineer has to debug the application without the application telling them what it is doing.

The workload layer is the source. Everything downstream is a view onto what the workload emits. When the workload is silent, the rest of the stack is silent. The discipline of the workload layer is the discipline of producing the right signals in the right shape.

What it is

The workload layer is the set of processes that produce telemetry. In this course the workloads are grouped into four categories:

• Linux hosts — the kernel, the scheduler, the filesystem, the network stack. Telemetry: node_exporter metrics.
• Containers — the cgroups, the namespaces, the OCI runtime. Telemetry: cAdvisor metrics, container logs.
• Applications — the user-written services. Telemetry: client libraries for the language (Go, Python, Java, Node, Rust), OpenTelemetry SDKs, structured logs.
• Databases and queues — the third-party services that applications depend on. Telemetry: specialised exporters (postgres_exporter, redis_exporter, mongodb_exporter, kafka_exporter), and synthetic probes.

The workload is the only layer that requires the application engineer to change code. The collector, backend, and presentation layers are operated by the platform team. The workload is wherever the application team lives.

Why a sysadmin cares

The workload layer is the source of truth. A misconfigured collector that drops a label is recoverable. A workload that emits a label with user IDs in it corrupts the backend for the lifetime of the retention. The cardinality decisions made at the source cannot be unmade cheaply.

The four cardinal sins of workload telemetry, in order of operational cost:

1. High-cardinality labels. A label that has one value per user, per request, per UUID. The backend OOMs. The fix is never at the backend.
2. Unstructured logs. Free-text logs without trace IDs, service names, or error levels. The log store can index them but cannot correlate them.
3. Missing trace context. A service that emits metrics and logs but does not propagate the W3C traceparent header. Spans become orphans. The trace view is a forest of singletons.
4. Swallowed errors. A try { ... } catch { ... } block that returns success to the caller. The application is failing but the metric says it is healthy. The on-call engineer hears from the user first.

The workload layer is the only layer where the engineering discipline of the application team shapes the operational quality of the platform.

How it works

The mental model: each workload has a small, fixed set of telemetry channels, and the right instrumentation produces the right channel on every event.

  +-------- Linux host ---------+    +------ container --------+
  |                             |    |                          |
  |  kernel  -- node_exporter   |    |  cgroup -- cAdvisor      |
  |  systemd -- node_exporter   |    |  OCI runtime -- cadvisor |
  |  /proc, /sys -- node_export |    |  stdout/stderr -- log    |
  |                             |    |  (driver = json-file)    |
  +-----------------------------+    +--------------------------+
            |                                      |
            | :9100/metrics                        | kubelet /metrics/cadvisor
            |                                      |
            v                                      v
  +-------- application ---------+    +------- database ----------+
  |                              |    |                          |
  |  OTel SDK  -- spans          |    |  postgres_exporter       |
  |  prom client -- metrics      |    |  redis_exporter          |
  |  structured logger -- logs   |    |  mongodb_exporter        |
  |  (stdout JSON, with trace_id)|    |  (vendor exporter format)|
  +------------------------------+    +--------------------------+
            |                                      |
            | OTLP gRPC /metrics :9100             | :9187/metrics
            |                                      |
            v                                      v
                            collector

The propagation rule: every request that crosses a service boundary must carry the same trace_id. Every span must carry the same service.name, service.version, and deployment.environment. Every log must carry the same fields. The labels are not optional; they are the contract that turns many signals into one investigation.

Under the hood

The canonical instrumentation per workload type:

Linux host (node_exporter). A single Go binary that reads /proc, /sys, and the systemd D-Bus, then exposes a /metrics endpoint in Prometheus text format. The --collector.filesystem.mount-points flag controls which mounts are reported. The default set of collectors covers everything except the NFS, WiFi, and systemd-timer collectors, which are enabled on demand.

Container (cAdvisor). Runs as a kubelet-managed process on every node. Reads cgroup accounting files and exposes /metrics/cadvisor. The metric container_cpu_usage_seconds_total is the canonical CPU signal for a container. The metric container_memory_working_set_bytes is the canonical memory signal.

Application (OpenTelemetry SDK). The OTel SDK is the modern default for new code. It exposes a tracer, a meter, and a logger through a single provider. The provider is configured at startup to export OTLP to the collector. The exporter endpoint is the only environment-specific change.

Database (vendor exporter). Each storage system has a small exporter that translates the vendor’s internal counters into Prometheus format. The exporter is a sidecar or a side-process. The connection string is the only application- specific configuration.

The four categories share a discipline: each one emits a fixed set of standard labels, and the standard labels are the investigation surface.

How to configure it

Application instrumentation in Go, with OpenTelemetry. The provider is configured once at startup and the resulting exporter is shared across the trace, metric, and log pipelines.

// internal/obs/provider.go
package obs

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func Provider(ctx context.Context, service, version string) (*trace.TracerProvider, *metric.MeterProvider, error) {
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName(service),
            semconv.ServiceVersion(version),
            semconv.DeploymentEnvironmentName("prod-eu-west-1"),
        ),
    )
    if err != nil {
        return nil, nil, err
    }

    traceExporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("localhost:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, nil, err
    }

    tp := trace.NewTracerProvider(trace.WithBatcher(traceExporter), trace.WithResource(res))

    metricExporter, err := otlpmetricgrpc.New(ctx,
        otlpmetricgrpc.WithEndpoint("localhost:4317"),
        otlpmetricgrpc.WithInsecure(),
    )
    if err != nil {
        return nil, nil, err
    }

    mp := metric.NewMeterProvider(metric.WithReader(metric.NewPeriodicReader(metricExporter)), metric.WithResource(res))

    otel.SetTracerProvider(tp)
    otel.SetMeterProvider(mp)
    return tp, mp, nil
}

The semantic-convention attributes (ServiceName, ServiceVersion, DeploymentEnvironmentName) are the attribute names that the rest of the stack relies on. The deployment environment is the most operationally useful — it turns a single trace query into “checkout in prod-eu-west-1” or “checkout in staging”.

Prometheus client in Go (alternative for metrics-only). When the workload has no tracer, the prometheus/client_golang library exposes a /metrics endpoint that the collector scrapes. The standard labels are attached at registration:

prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: "checkout_inflight_requests",
        Help: "Number of currently in-flight checkout requests.",
    },
    []string{"region", "payment_provider"},
)

The region and payment_provider labels are finite (low-cardinality). The label set is registered once and never extended with a request-specific value.

Structured application logs. Logs are written to stdout as JSON. The collector tail reads the file and forwards to Loki. The trace ID is included in the line so the on-call engineer can pivot from a log entry to the trace:

log.Info("payment authorised",
    "trace_id", span.SpanContext().TraceID().String(),
    "span_id",  span.SpanContext().SpanID().String(),
    "amount",   amount,
    "currency", currency,
)

node_exporter service unit. Standard systemd unit, with flags for the collectors that are off by default:

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --collector.filesystem.mount-points-exclude='^/(sys|proc|dev|host|etc)($$|/)' \
  --collector.systemd \
  --web.listen-address=127.0.0.1:9100
Restart=on-failure

[Install]
WantedBy=multi-user.target

The 127.0.0.1:9100 bind is the production default — the collector is the only consumer, and the network policy is host-local.

How to validate it

READ-ONLY — confirm the workload is exposing instrumentation.

curl -s http://localhost:9100/metrics | head -20
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
# go_gc_duration_seconds{quantile="0"} 4.3e-05
# ...

curl -s http://localhost:9100/metrics | grep -c '^node_'
# 1200

READ-ONLY — confirm an end-to-end trace is being produced by the application.

curl -sf "http://tempo.monitoring.svc:3200/api/search?tags=service.name%3Dcheckout&limit=1" | jq .
# { "traces": [{ "traceID": "8f4ab1e3c2d9...", "rootServiceName": "checkout" }] }

READ-ONLY — confirm the trace carries the right attributes.

curl -sf "http://tempo.monitoring.svc:3200/api/traces/8f4ab1e3c2d9" | jq '.resourceSpans[0].resource.attributes[] | select(.key | IN("service.name","service.version","deployment.environment"))'
# { "key": "service.name", "value": { "stringValue": "checkout" } }
# { "key": "service.version", "value": { "stringValue": "1.42.0" } }
# { "key": "deployment.environment", "value": { "stringValue": "prod-eu-west-1" } }

READ-ONLY — confirm the workload logs are queryable.

logcli --addr=http://loki.monitoring.svc:3100 \
  query '{service_name="checkout", deployment_environment="prod-eu-west-1"} | json | trace_id=~"8f4ab1e3c2d9.*"'
# {"service_name":"checkout","level":"info","msg":"payment authorised","trace_id":"8f4ab1e3c2d9..."}

READ-ONLY — confirm the application metric is being scraped.

curl -sf 'http://prometheus.monitoring.svc:9090/api/v1/query?query=checkout_inflight_requests' | jq .
# { "status": "success",
#   "data": { "resultType": "vector",
#             "result": [
#               { "metric": { "region": "eu-west-1", "payment_provider": "stripe" },
#                 "value": [1723651200, "12"] }
#             ] } }

How it can fail

1. High-cardinality labels at the source. A gauge is registered with URL as a label. Prometheus prometheus_tsdb_head_series climbs past 50 million. The workload is the cause; the collector is the only place to mitigate.
2. Trace context not propagated. A service uses an HTTP client that strips the traceparent header. The downstream spans are orphans. Tempo shows a single span per request instead of a tree.
3. Logs are unstructured. The application writes to os.Stderr with fmt.Println. The collector tails the file but the Loki index has nothing to pivot on. The log store contains text but cannot answer questions.
4. Old client library. A Go service uses a vendored prometheus/client_golang from 2018. It does not emit exemplars. The metric and the trace cannot be linked.
5. Service exposes metrics without authentication in a semi-trusted network. A different tenant on the same network can read the metrics. The fix is a listener-only bind or a reverse proxy with authentication.
6. Swallowed error. A try { ... } catch (...) { return nil } pattern around the database call. The metric says success; the data is missing. The user reports the data is missing before the metric says anything is wrong.

How to troubleshoot it

The diagnostic order for “the workload should be emitting, but nothing is in the dashboard”:

1. Can the workload reach the collector? End-to-end connectivity. nc -vz localhost 4317 for OTLP, curl http://localhost:9100/metrics for the scrape endpoint.
2. Are the right env vars in the workload? The OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME variables are the canonical configuration knobs.
3. Is the collector receiving? Each collector has a self-telemetry endpoint that exposes the totals of accepted and rejected datapoints.
4. Are the labels populated? A trace that says service.name=unknown_service is a workload that did not call the resource builder.
5. Is the trace context propagated? Open a single request in Tempo and confirm the span tree has more than one span.

Security implications

The workload exposes a /metrics endpoint and a log stream. Both are sensitive.

• The /metrics endpoint exposes the workload’s internal counters. Bind to loopback or to a network that has a policy. Bind to 0.0.0.0 on a host with a public address is a leak.
• The logs may contain request bodies, user IDs, or secrets. The collector that tails the logs must be on the same trust boundary. The log storage must enforce redaction at ingest.
• The OTLP exporter is a credential. The workload must authenticate to the collector with mTLS or a token. The environment variable that holds the token is a secret; rotate it like any other.
• The semantic-convention attributes are not sensitive by default. The application-specific attributes (user IDs, order IDs) are. The official semantic conventions are the safe baseline; the custom attributes are the audit target.

Performance implications

The workload is the source of every byte the rest of the stack processes. The four knobs:

• Sample rate. A 100% trace sample rate is the default for development. Production should drop to 1–10% for high-volume endpoints and 100% for low-volume ones (errors, payment).
• Cardinality. One label per series. The total number of series is the cardinality of the workload. The wrong answer is to log every dimension; the right answer is to log only the dimensions that are bounded.
• Log volume. A noisy log line at 10 kHz fills the log pipeline. The fix is to drop the line at the source, not in the collector.
• Synchronous vs asynchronous exporter. The OTel SDK defaults to a batch exporter. The trace is recorded in memory and shipped in a background goroutine. A synchronous exporter would block the request path and is the wrong choice.

Production guidance

• Standard semantic conventions. Use the OpenTelemetry semantic conventions for service.name, service.version, deployment.environment. They are the contract with the rest of the stack.
• Limit label cardinality at the source. Every label has a documented set of values. The CI pipeline rejects labels whose cardinality crosses a threshold.
• Trace context on every request. The OTel SDK does this for the first-party HTTP client. For third-party clients that strip headers, the platform team maintains the patch list.
• Structured logs only. JSON logs on stdout. The collector tails the file. Free-text logs are debug-grade and should not reach production.
• Workload-owned exporters. The postgres_exporter and redis_exporter are deployed by the platform team but owned by the application team. The exporter is part of the workload’s contract.

Verification

You should now be able to answer:

• What are the four cardinal sins of workload telemetry, and which one is the most operationally costly?
• Why is the collector the right place to enforce cardinality caps even though the workload is the source?
• What is the difference between pushing OTLP and exposing /metrics for the collector to scrape?
• Why do structured logs need a trace_id field, and what does it enable?
• What is the failure mode when a service strips the traceparent header?

Quiz