Lab: Build an OpenTelemetry Collector Pipeline

Objective

By the end of this lab one OpenTelemetry Collector will be accepting all three signals on OTLP and delivering them to three different backends, and you will be able to account for a single log record, a single span and a single metric point from the receiver counter to the exporter counter to the backend’s own query API.

Then you will break it three times. Two of the three refuse to start and tell you exactly which line is wrong — that is the factory model doing its job. The third passes validation, starts cleanly, runs for weeks and then kills the process during the first downstream outage. Knowing which category a mistake falls into is most of what operating a collector is.

Architecture

One collector process, three pipelines, three backends. The collector is the choke point: every signal the platform sees passes through it, so its own counters are the first evidence in any telemetry investigation.

                        push by hand (curl, OTLP/JSON)
                                    |
                        +-----------v-----------+
                        | otelcol-contrib       |
                        |  receivers: otlp      |
                        |    :4317 grpc         |
                        |    :4318 http         |
                        |                       |
                        |  processors (in order)|
                        |    memory_limiter     |
                        |    resource           |
                        |    batch              |
                        |                       |
                        |  :13133 health        |
                        |  :8888  self-metrics  |
                        +--+--------+--------+--+
                           |        |        |
              otlphttp     |        | otlp   | prometheusremotewrite
                           v        v        v
                    +------+--+ +---+----+ +-+-------------+
                    | loki 3.3| |tempo   | | prometheus    |
                    | :3100   | |2.6     | | 2.55  :9090   |
                    | /otlp   | |:4317   | | /api/v1/write |
                    +---------+ +--------+ +---------------+

Note that the collector’s OTLP receiver and Tempo’s OTLP receiver both want port 4317. Only the collector publishes it to the host; Tempo’s stays inside the compose network. That is the normal production shape too — the collector is the only thing applications talk to.

Requirements

• A Linux host with Docker Engine 28.x and Docker Compose v2.
• curl and jq on the host. Every push in this lab is JSON built with jq -n, which keeps the payloads readable and the quoting survivable.
• Free TCP ports 4317, 4318, 8888, 13133, 3100, 3200 and 9090, all bound to loopback.
• About 1.5 GiB of free memory and 1 GiB of free disk.
• No out-of-band access requirement. Nothing outside the lab directory and the compose project is modified.

Scenario

A platform team is replacing three vendor agents with one collector. The migration plan is one page long and the configuration is forty lines, so it is scheduled as a low-risk change.

It is not. Two of the three configurations the team writes during the migration will not start, which is fine — the collector says why, in one line, before any data moves. The third starts perfectly, passes review, and OOM-kills itself eight days later during a twenty-minute Loki outage, taking every buffered span on the host with it. Nobody connects the two events, because the change was to a processor list and the outage was in a different system.

You are going to write all three, in order, on a bench.

Tasks

Task 1: Write the collector configuration

LABDIR="$HOME/rb-obs-otelpipeline"
mkdir -p "$LABDIR"
cd "$LABDIR"

otelcol.yaml. Read the service block first — it is the only part that decides what actually runs, and everything above it is inert until something in service names it:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # First in every pipeline. The gate that refuses data before the exporter
  # queues fill; without it the queue is the only brake and the OOM killer is
  # the only limit.
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64

  # Enrich late: one static attribute stamped on every resource, so a query in
  # any of the three backends can tell lab data from anything else.
  resource:
    attributes:
      - key: deployment.environment
        value: lab
        action: upsert

  # Last in every pipeline. Coalescing before the exporter is what makes the
  # per-request overhead of three backends affordable.
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  # Loki 3.x ingests OTLP natively at /otlp; the otlphttp exporter appends
  # /v1/logs to the endpoint below. No Loki-specific exporter is involved.
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
    tls:
      insecure: true

  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
    tls:
      insecure: true

  # Prints what passes through it. Invaluable while wiring; expensive and
  # noisy in production, which is why it is on its own here rather than added
  # to every pipeline.
  debug:
    verbosity: normal

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
    path: /health

service:
  extensions: [health_check]
  pipelines:
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [otlphttp/loki, debug]
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [otlp/tempo]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [prometheusremotewrite]
  telemetry:
    metrics:
      address: 0.0.0.0:8888
    logs:
      level: info

Task 2: Write the backends and the compose file

loki.yaml — single-binary Loki with the schema OTLP ingestion needs:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: warn

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  # OTLP ingestion puts non-indexed resource and record attributes into
  # structured metadata, which requires this and schema v13 together.
  allow_structured_metadata: true
  reject_old_samples: false

analytics:
  reporting_enabled: false

tempo.yaml:

server:
  http_listen_port: 3200
  log_level: warn

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317

ingester:
  # Short blocks so a span pushed during the lab is flushed inside its lifetime.
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 1h

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

prometheus.yml — Prometheus scrapes the collector’s own telemetry, and separately receives the application metrics by remote write:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: otelcol
    static_configs:
      - targets: ['otelcol:8888']

compose.yaml:

name: rb-obs-otelpipeline

services:
  otelcol:
    image: otel/opentelemetry-collector-contrib:0.110.0
    command: ['--config=/etc/otelcol/otelcol.yaml']
    volumes:
      - ./otelcol.yaml:/etc/otelcol/otelcol.yaml:ro
    ports:
      - '127.0.0.1:4317:4317'
      - '127.0.0.1:4318:4318'
      - '127.0.0.1:8888:8888'
      - '127.0.0.1:13133:13133'
    depends_on:
      - loki
      - tempo
      - prometheus

  loki:
    image: grafana/loki:3.3.0
    command: -config.file=/etc/loki/loki.yaml
    volumes:
      - ./loki.yaml:/etc/loki/loki.yaml:ro
      - loki-data:/loki
    ports:
      - '127.0.0.1:3100:3100'

  # Tempo runs as uid 10001 and its storage paths do not exist in the image,
  # so the named volume arrives owned by root and Tempo cannot write to it.
  tempo-init:
    image: grafana/tempo:2.6.0
    user: root
    entrypoint: ['/bin/sh', '-c']
    command: ['chown -R 10001:10001 /var/tempo']
    volumes:
      - tempo-data:/var/tempo

  tempo:
    image: grafana/tempo:2.6.0
    command: -config.file=/etc/tempo/tempo.yaml
    depends_on:
      tempo-init:
        condition: service_completed_successfully
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - '127.0.0.1:3200:3200'

  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=3h'
      - '--web.enable-remote-write-receiver'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - '127.0.0.1:9090:9090'

volumes:
  loki-data:
  tempo-data:
  prom-data:

Task 3: Validate before you start, then start

The collector has a parse-and-build check that constructs every component and resolves every pipeline reference without opening a socket. Run it first, every time. It is faster than a restart loop and its errors are the same ones the runtime would produce:

cd "$LABDIR"
docker compose run --rm --no-deps otelcol validate --config=/etc/otelcol/otelcol.yaml \
  && echo "CONFIG OK"

$ docker compose up -d

sleep 30
docker compose ps
curl -sf http://127.0.0.1:13133/health && echo
curl -sf http://127.0.0.1:3100/ready && echo LOKI-READY
curl -sf http://127.0.0.1:3200/ready && echo TEMPO-READY
curl -sf http://127.0.0.1:9090/-/ready && echo PROM-READY

Confirm the receiver is actually listening on both protocols. A receiver bound to the wrong address is the failure that looks like an application problem:

ss -ltn | grep -E '4317|4318'

Take a baseline of the counters. Everything in Task 4 is a difference against these numbers, so they have to be read before anything is pushed:

curl -s http://127.0.0.1:8888/metrics \
| grep -E '^otelcol_(receiver_accepted|exporter_sent)' \
| sort

On a collector that has received nothing, the families may be entirely absent rather than zero. That is normal: a Prometheus counter is created when it is first incremented, so “no line” and “zero” are the same statement here.

Task 4: Push one of each, and account for it

This is the task that turns the diagram into evidence. Three pushes, three counter checks, and the arithmetic must close each time.

One log record. OTLP/HTTP takes JSON; severityNumber 17 is ERROR in the specification’s severity scale:

NOW_NS=$(( $(date +%s) * 1000000000 ))

jq -nc --arg ts "$NOW_NS" '{resourceLogs:[{
  resource:{attributes:[{key:"service.name",value:{stringValue:"checkout"}}]},
  scopeLogs:[{
    scope:{name:"runbook-academy-lab"},
    logRecords:[{
      timeUnixNano:$ts, observedTimeUnixNano:$ts,
      severityNumber:17, severityText:"ERROR",
      body:{stringValue:"payment gateway rejected the authorisation"},
      attributes:[{key:"http.response.status_code",value:{intValue:"502"}}]
    }]}]}]}' \
| curl -sf -X POST http://127.0.0.1:4318/v1/logs \
    -H 'Content-Type: application/json' --data-binary @- \
    -w '\nlogs push: HTTP %{http_code}\n'

One span. Trace and span ids are hex strings and timestamps are nanoseconds-as-strings; that is the JSON mapping for OTLP’s bytes and uint64 fields, not a stylistic choice:

TRACE_ID=$(head -c16 /dev/urandom | od -An -tx1 | tr -d ' \n')
SPAN_ID=$(head -c8 /dev/urandom | od -An -tx1 | tr -d ' \n')
END_NS=$(( $(date +%s) * 1000000000 ))
START_NS=$(( END_NS - 4200000000 ))
echo "trace id $TRACE_ID"

jq -nc --arg tid "$TRACE_ID" --arg sid "$SPAN_ID" \
       --arg start "$START_NS" --arg end "$END_NS" \
  '{resourceSpans:[{
      resource:{attributes:[{key:"service.name",value:{stringValue:"checkout"}}]},
      scopeSpans:[{
        scope:{name:"runbook-academy-lab"},
        spans:[{
          traceId:$tid, spanId:$sid,
          name:"POST /api/v1/checkout", kind:2,
          startTimeUnixNano:$start, endTimeUnixNano:$end,
          status:{code:2, message:"payment declined"}
        }]}]}]}' \
| curl -sf -X POST http://127.0.0.1:4318/v1/traces \
    -H 'Content-Type: application/json' --data-binary @- \
    -w '\ntraces push: HTTP %{http_code}\n'

One metric point. aggregationTemporality: 2 is cumulative, which is what prometheusremotewrite requires — a delta sum is rejected rather than converted:

NOW_NS=$(( $(date +%s) * 1000000000 ))
START_NS=$(( NOW_NS - 60000000000 ))

jq -nc --arg start "$START_NS" --arg now "$NOW_NS" \
  '{resourceMetrics:[{
      resource:{attributes:[{key:"service.name",value:{stringValue:"checkout"}}]},
      scopeMetrics:[{
        scope:{name:"runbook-academy-lab"},
        metrics:[{
          name:"checkout_orders_total", unit:"1",
          sum:{aggregationTemporality:2, isMonotonic:true,
               dataPoints:[{asInt:"7",
                            startTimeUnixNano:$start,
                            timeUnixNano:$now}]}
        }]}]}]}' \
| curl -sf -X POST http://127.0.0.1:4318/v1/metrics \
    -H 'Content-Type: application/json' --data-binary @- \
    -w '\nmetrics push: HTTP %{http_code}\n'

Each push answers HTTP 200 with {"partialSuccess":{}}. A 400 is a malformed payload, not a rejected signal — read the response body, it names the field.

Now wait out one batch timeout and read the counters:

sleep 10
curl -s http://127.0.0.1:8888/metrics \
| grep -E '^otelcol_(receiver_accepted|exporter_sent|exporter_send_failed)' \
| sort

Three accepted, three sent, nothing failed — one log record, one span, one metric point, each named by its own counter family with the receiver and exporter as labels. This pairing is the single most useful diagnostic the collector produces, and the runbook for a collector failure is built entirely on reading these two numbers against each other.

Task 5: Confirm each backend actually has it

Counters say the collector believes it sent the data. The backends are the only authority on whether they received it.

Loki. OTLP resource attributes map into stream labels; service.name becomes service_name:

curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode 'since=15m' \
| jq -r '.data.result[0].values[0][1] // "NOT FOUND"'

Look at the labels the ingestion actually produced rather than assuming them — this mapping is the single most common surprise when a team moves from a log shipper to OTLP:

curl -sfG http://127.0.0.1:3100/loki/api/v1/series \
  --data-urlencode 'match[]={service_name="checkout"}' \
  --data-urlencode 'since=15m' | jq -c '.data[]'

Tempo. Ask for the id you pushed:

curl -sf -H 'Accept: application/json' \
  "http://127.0.0.1:3200/api/traces/$TRACE_ID" \
| jq '[.. | .spans? // empty | .[]] | length'

One span. A 404 here after a 200 on the push means the id you are querying is not the id you sent — compare them character by character before suspecting Tempo.

Prometheus. The remote-write exporter normalises metric names, so find out what landed rather than guessing at the suffix:

curl -sf http://127.0.0.1:9090/api/v1/label/__name__/values \
| jq -r '.data[] | select(startswith("checkout"))'

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query={__name__=~"checkout.*"}' \
| jq -r '.data.result[] | "\(.metric.__name__) = \(.value[1])  env=\(.metric.deployment_environment // "unset")"'

The deployment_environment label is the resource processor’s static attribute arriving at the far end. Its presence in all three backends is the proof that the processor chain ran, and it is worth checking in Loki and Tempo too — in Loki it appears in the stream labels or structured metadata, and in Tempo among the resource attributes on the span.

Task 6: Two configurations that refuse to start

Both of these fail at build time, before a byte moves. That is the factory model working exactly as designed, and the reason a collector misconfiguration is usually a five-minute problem.

The undeclared component. Change one character in the logs pipeline:

cp otelcol.yaml otelcol.yaml.good
sed -i 's|exporters:  \[otlphttp/loki, debug\]|exporters:  [otlphttp/lokii, debug]|' otelcol.yaml

docker compose run --rm --no-deps otelcol validate --config=/etc/otelcol/otelcol.yaml \
  || echo "validate refused the config, as it should"

Read the message and note two things: it names the component id that does not exist, and it names the pipeline that referenced it. There is no plugin loader and no fallback — a pipeline may only name components declared in their block.

The signal mismatch. Restore, then wire a metrics exporter into the logs pipeline, which is the classic copy-paste result:

cp otelcol.yaml.good otelcol.yaml
sed -i 's|exporters:  \[otlphttp/loki, debug\]|exporters:  [prometheusremotewrite]|' otelcol.yaml

docker compose run --rm --no-deps otelcol validate --config=/etc/otelcol/otelcol.yaml \
  || echo "validate refused the config, as it should"

The error names the signal. prometheusremotewrite is a metrics exporter and a logs pipeline cannot carry metrics; there is no coercion and no partial start.

Restore and confirm you are back to a good configuration before continuing:

cp otelcol.yaml.good otelcol.yaml
docker compose run --rm --no-deps otelcol validate --config=/etc/otelcol/otelcol.yaml \
  && echo "CONFIG OK"

Task 7: The configuration that starts, and should not

Now the third one. Swap the order of two processors — put batch before memory_limiter in every pipeline, which is what happens when somebody groups the list “logically” by what each processor does:

sed -i 's|processors: \[memory_limiter, resource, batch\]|processors: [batch, resource, memory_limiter]|g' otelcol.yaml

docker compose run --rm --no-deps otelcol validate --config=/etc/otelcol/otelcol.yaml \
  && echo "VALIDATE PASSED — read that again"

docker compose up -d --force-recreate otelcol
sleep 20
curl -sf http://127.0.0.1:13133/health && echo " <- healthy"
docker compose logs --tail=20 otelcol | grep -i 'warn\|error' || echo "no warnings logged"

It validates. It starts. It reports healthy. It will pass every review that consists of reading the configuration, because there is nothing in the configuration that is wrong in isolation — both processors are declared, both are legal in a pipeline, and both are in every pipeline exactly once.

Put it back, and keep the discipline as a written rule rather than as a memory: memory_limiter first, batch last, everything else in between, dropping early and enriching late.

cp otelcol.yaml.good otelcol.yaml
docker compose up -d --force-recreate otelcol
sleep 15
curl -sf http://127.0.0.1:13133/health && echo " <- healthy"

Task 8: Take a backend away, twice

First, with the default in-memory queue. Stop Tempo and push spans into a collector that cannot deliver them:

$ docker compose stop tempo

for i in $(seq 1 20); do
  TRACE_ID=$(head -c16 /dev/urandom | od -An -tx1 | tr -d ' \n')
  SPAN_ID=$(head -c8 /dev/urandom | od -An -tx1 | tr -d ' \n')
  END_NS=$(( $(date +%s) * 1000000000 ))
  jq -nc --arg tid "$TRACE_ID" --arg sid "$SPAN_ID" --arg end "$END_NS" \
    '{resourceSpans:[{resource:{attributes:[{key:"service.name",
        value:{stringValue:"checkout"}}]},
      scopeSpans:[{scope:{name:"outage-test"},
        spans:[{traceId:$tid, spanId:$sid, name:"POST /api/v1/checkout", kind:2,
                startTimeUnixNano:$end, endTimeUnixNano:$end}]}]}]}' \
  | curl -sf -X POST http://127.0.0.1:4318/v1/traces \
      -H 'Content-Type: application/json' --data-binary @- -o /dev/null
done
echo "pushed 20 spans into an outage"

sleep 30
curl -s http://127.0.0.1:8888/metrics \
| grep -E '^otelcol_exporter_(send_failed_spans|queue_size|queue_capacity)'

The receiver still accepts — from the sender’s point of view nothing is wrong, which is exactly why an application-side error rate does not detect this. send_failed climbs, the queue holds what it can, and retry_on_failure keeps trying with backoff.

Now restart the collector while Tempo is still down, which is what a deploy or a node drain does:

docker compose restart otelcol
sleep 20
docker compose start tempo
sleep 45

curl -sf -H 'Accept: application/json' \
  'http://127.0.0.1:3200/api/search?tags=service.name%3Dcheckout&limit=50' \
| jq -r '.traces | length // 0'

Count what arrived. The spans queued in memory at the moment of the restart are gone — not delayed, gone — and no counter in the collector will ever mention them again, because the process that held the counters is the process that died.

Second, with the queue on disk. Add the file_storage extension and point the Tempo exporter’s queue at it:

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000
      # Names the extension below. With it the queue is a directory; without
      # it the queue is heap, and a restart is a data-loss event.
      storage: file_storage

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
    path: /health
  file_storage:
    directory: /var/lib/otelcol/storage
    timeout: 10s

service:
  extensions: [health_check, file_storage]

Apply those three edits to otelcol.yaml, then give the collector a writable directory. The contrib image runs as a non-root user, so a named volume needs its ownership fixed the same way Tempo’s did — add this service and this volume to compose.yaml, and add the volume mount plus the dependency to otelcol:

  otelcol-init:
    image: otel/opentelemetry-collector-contrib:0.110.0
    user: root
    entrypoint: ['/bin/sh', '-c']
    command: ['mkdir -p /var/lib/otelcol/storage && chown -R 10001:10001 /var/lib/otelcol']
    volumes:
      - otelcol-storage:/var/lib/otelcol

The otelcol service gains a mount and a dependency, and the file gains one more named volume:

  otelcol:
    # ... everything already there stays ...
    volumes:
      - ./otelcol.yaml:/etc/otelcol/otelcol.yaml:ro
      - otelcol-storage:/var/lib/otelcol
    depends_on:
      otelcol-init:
        condition: service_completed_successfully
      loki:
        condition: service_started
      tempo:
        condition: service_started
      prometheus:
        condition: service_started

volumes:
  loki-data:
  tempo-data:
  prom-data:
  otelcol-storage:

Mixing the short and long depends_on forms is not allowed, so once one entry needs condition: all of them do — which is why the three backends are spelled out again above.

docker compose run --rm --no-deps otelcol validate --config=/etc/otelcol/otelcol.yaml \
  && echo "CONFIG OK"
docker compose up -d
sleep 25
curl -sf http://127.0.0.1:13133/health && echo " <- healthy with a disk-backed queue"

Repeat the outage, exactly as before:

docker compose stop tempo

for i in $(seq 1 20); do
  TRACE_ID=$(head -c16 /dev/urandom | od -An -tx1 | tr -d ' \n')
  SPAN_ID=$(head -c8 /dev/urandom | od -An -tx1 | tr -d ' \n')
  END_NS=$(( $(date +%s) * 1000000000 ))
  jq -nc --arg tid "$TRACE_ID" --arg sid "$SPAN_ID" --arg end "$END_NS" \
    '{resourceSpans:[{resource:{attributes:[{key:"service.name",
        value:{stringValue:"checkout-durable"}}]},
      scopeSpans:[{scope:{name:"outage-test"},
        spans:[{traceId:$tid, spanId:$sid, name:"POST /api/v1/checkout", kind:2,
                startTimeUnixNano:$end, endTimeUnixNano:$end}]}]}]}' \
  | curl -sf -X POST http://127.0.0.1:4318/v1/traces \
      -H 'Content-Type: application/json' --data-binary @- -o /dev/null
done

sleep 20
docker compose restart otelcol
sleep 20
docker compose start tempo
sleep 60

curl -sf -H 'Accept: application/json' \
  'http://127.0.0.1:3200/api/search?tags=service.name%3Dcheckout-durable&limit=50' \
| jq -r '.traces | length'

Compare against the first run and write both numbers down. The service name differs deliberately, so the two outages are countable separately.

Validation

1. The configuration builds. From a clean state, validation passes and the health endpoint answers:

cd "$HOME/rb-obs-otelpipeline"
docker compose run --rm --no-deps otelcol validate --config=/etc/otelcol/otelcol.yaml \
  && curl -sf http://127.0.0.1:13133/health && echo " OK"

2. Accepted and sent agree for all three signals, with no failures:

curl -s http://127.0.0.1:8888/metrics \
| grep -E '^otelcol_(receiver_accepted|exporter_sent|exporter_send_failed|processor_refused)' \
| sort

Every send_failed and refused family should be absent or flat. A send_failed counter that is non-zero and no longer climbing is the residue of Task 8 and is fine; one that is still climbing is a live fault.

3. All three backends hold data carrying the resource attribute. One query per backend, each returning a non-empty result and each showing deployment.environment = lab (or deployment_environment, depending on how that backend normalises attribute names).

4. The two refused configurations are recorded. Your notes contain the exact error text for the undeclared component and for the signal mismatch, and for each one the pipeline the error named.

5. The silent one is recorded as silent. Your notes state that [batch, resource, memory_limiter] passed validate, started, and reported healthy — because that sentence is the reason the ordering rule has to live in a review checklist rather than in the tool.

6. The queue comparison has two numbers. Spans recovered after a restart-during-outage with the memory queue, and with the file-backed queue.

Expected Outcome

• Six containers running (five services plus a completed init), a collector reporting healthy, and three backends holding one hand-pushed record each.
• A otelcol.yaml in which memory_limiter is first and batch is last in every pipeline, and every component named in service is declared above it.
• Counter readings showing one accepted and one sent per signal for the Task 4 pushes.
• Two recorded startup errors and one recorded non-error.
• Two span-recovery counts from the same outage, differing by whether the queue was in memory or on disk.

Troubleshooting

validate reports an unknown component such as otlphttp/loki. You are running the core collector image rather than contrib. prometheusremotewrite, resource, health_check and file_storage are all contrib components; the image tag must be otel/opentelemetry-collector-contrib.

The collector exits immediately and docker compose ps shows it not running. A build failure is a fatal start failure, by design. docker compose logs otelcol holds the reason, and the last line before the exit names the component or the field.

A push returns HTTP 415. The Content-Type: application/json header is missing, so the OTLP/HTTP receiver is expecting protobuf.

A push returns HTTP 400 with a message about a field. The JSON does not match the OTLP schema. The usual causes are a numeric timeUnixNano where the mapping requires a string, and a trace id that is not exactly 32 hex characters.

Counters move but Loki has nothing. Check the stream labels rather than the query: OTLP ingestion derives them from resource attributes, so {job="checkout"} finds nothing while {service_name="checkout"} finds everything. GET /loki/api/v1/labels lists what exists.

prometheusremotewrite logs a rejection about temporality. The metric was sent with aggregationTemporality: 1 (delta). Remote write requires cumulative; either send cumulative or add the cumulativetodelta conversion deliberately.

Prometheus rejects the write with a 404. The --web.enable-remote-write-receiver flag is missing, so /api/v1/write does not exist.

Tempo restarts in a loop with a permission error on /var/tempo. The tempo-init container did not complete. docker compose ps -a should show it Exited (0); if not, docker compose up -d --force-recreate tempo-init.

The collector fails to start after file_storage is added, with a permission error. otelcol-init has not run, or ran after the collector. The same condition-based dependency Tempo uses applies here.

Cleanup

$ cd ~/rb-obs-otelpipeline && docker compose down -v

docker volume ls | grep rb-obs-otelpipeline || echo "volumes gone"

Keep the configuration and your notes, then remove the directory:

mkdir -p "$HOME/rb-obs-deliverables"
cp -a "$HOME/rb-obs-otelpipeline/otelcol.yaml" \
      "$HOME/rb-obs-deliverables/otelcol-pipeline.yaml"

rm -rf "$HOME/rb-obs-otelpipeline"

The images stay in the local cache. Remove them if you want the disk back:

docker image rm otel/opentelemetry-collector-contrib:0.110.0 \
  grafana/loki:3.3.0 grafana/tempo:2.6.0 prom/prometheus:v2.55.1

Production notes

Put validate in CI and the counter pair in the runbook. Every configuration change should fail in a pipeline rather than on a host, and validate catches both of the errors from Task 6 in under a second with no backend involved. What it does not catch is Task 7, so the review checklist needs one line the tool cannot enforce: memory_limiter first, batch last.

The collector is a single point of failure with a queue, and the queue is a capacity decision. queue_size multiplied by the average item size is the memory the collector will hold when the backend goes away, and it is the amount of data you lose if the process restarts with the queue in heap. Decide how long an outage you intend to survive, size the queue for it, and put it on disk — the change is three lines and it converts a data-loss event into an availability event.

Deploy collector configuration the way you deploy application configuration. A start failure means no telemetry from that host until somebody notices, and the thing that would normally tell you a host went quiet is the telemetry. Roll to one host, confirm otelcol_receiver_accepted_* is climbing there, then roll wider. A canary is cheap; a fleet that stopped reporting at 02:00 is not.

Monitor the collector with something that is not the collector. Its self-telemetry on :8888 is scraped by Prometheus here, which works until the collector is also the path Prometheus depends on. In a gateway topology, have the agents scrape their own local collector and the gateway scraped from the monitoring cluster directly, so no single failure removes both the signal and the evidence of its absence.