Processors

A gateway collector at a 500-host fleet OOMs twice in one week. The team investigates. The memory_limiter is configured with limit_percentage: 80 against a 2 GiB heap limit; the process_runtime_total_alloc_bytes metric climbs to 2.4 GiB before the kernel acts. The order of the processors in the chain is batch, memory_limiter, resource, transform. The batch grows unbounded; the limiter never sees the queue.

The fix is a one-line reorder. The lesson is that processors are not a set; they are a chain. The chain order is the discipline.

What it is

A processor transforms, batches, filters, or enriches telemetry between a receiver and an exporter. A processor takes pdata in and emits pdata out; it does not accept input from the wire and it does not write to a backend. Processors are declared in their own block and wired into pipelines by name.

   receiver
      |
      v
   processor_1      (memory_limiter - gate)
      |
      v
   processor_2      (filter        - drop noise)
      |
      v
   processor_3      (resource      - stamp labels)
      |
      v
   processor_4      (attributes    - mutate)
      |
      v
   processor_5      (batch         - coalesce)
      |
      v
   exporter ...

The processors run sequentially in the order declared in service.pipelines. Each processor sees every pdata record that the previous processor emitted. A processor that drops a record passes nothing forward; a processor that emits multiple records amplifies the chain.

Why a sysadmin cares

The processor chain is where the operational shape of the collector is decided. Three properties make it so.

1. Cost control. The memory_limiter is the gate that prevents the chain from filling the heap. Without it, a slow exporter grows the queue until the kernel OOM-kills the process. The position of the gate matters: a memory_limiter placed after batch never sees the queue.
2. Cost shaping. Processors have per-record CPU and memory cost. A transform processor that runs a regex on every log body can dominate CPU. A tail_sampling processor that evaluates every trace can dominate memory. The chain placement shapes the cost: drop with filter before batching; enrich with resource before batching; batch last.
3. Signal shape. The processors decide what the pipeline emits. A resource processor stamps job and env. A resourcedetection processor adds cloud.region, host.name, and k8s.pod.name. A transform processor parses fields out of a log body. The shape of the signal that reaches the exporter is the shape of the work the processors did.

How it works

Memory limiter

The memory_limiter processor refuses data when the process approaches a memory limit. It must be the first processor in the chain. The processor checks the runtime allocator metrics on a configurable interval (check_interval); when the heap exceeds limit_percentage of the configured limit, the processor refuses new data until the heap falls below limit_percentage minus spike_limit_percentage.

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

The default limit_percentage is 80; the default spike_limit_percentage is 20. The defaults are tuned for a collector with no other processors; with batching and transformation in the chain, lower the limit and accept more refusals. Refusals back-pressure the receivers; the upstream source sees the back-off and retries.

Batch

The batch processor coalesces entries to reduce per-call cost. A larger send_batch_size with a longer timeout trades latency for throughput. The default timeout: 200ms is too aggressive for a gateway; five seconds is a more realistic starting point.

processors:
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 10000

send_batch_size is the target; send_batch_max_size is the hard limit. The processor emits when either the timeout fires or the batch reaches the size. The default send_batch_max_size is 0 (unlimited); set it to a value above send_batch_size to bound the per-call payload.

Attributes and resource

The attributes processor mutates the per-entry attributes; the resource processor mutates the resource attributes that apply to every entry in the same pdata batch.

processors:
  attributes:
    actions:
      - key: http.request.method
        from_attribute: http.method
        action: insert
      - key: sensitive.header
        action: delete
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: service.namespace
        value: checkout
        action: upsert

The difference matters for Loki labels. The Loki exporter maps resource attributes to stream labels; the per-entry attributes become the log line’s structured fields. A label change is a new stream; a field change is a new entry in the same stream. The cost of a label change is the cost of a new stream in Loki.

Resource detection

The resourcedetection processor adds attributes discovered from the environment: cloud provider, region, instance, host, Kubernetes pod.

processors:
  resourcedetection:
    detectors: [env, system, ec2, gcp, azure]
    timeout: 2s
    override: false

The processor runs once at start; it does not refresh on a schedule. The right discipline is to combine resourcedetection with a resource processor that stamps static labels (env, team, tier). The static labels are always present; the detected labels are added once.

Filter and transform

The filter processor drops entries that match an expression. The transform processor applies a small DSL (the OpenTelemetry Transformation Language) to entries.

processors:
  filter:
    logs:
      exclude:
        match_type: regexp
        bodies:
          - '.*DEBUG.*'
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["level"], "info") where body == "noise"

The filter processor is the cheapest way to drop noise; the transform processor is the most flexible. Use filter for high-volume drop; use transform for parsing, normalisation, and conditional logic.

Tail sampling

The tail_sampling processor buffers traces until a decision policy can be evaluated. The processor runs only on the traces pipeline; it is the most expensive processor in the collector.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500

The processor holds num_traces traces in memory until the decision_wait window closes; the memory cost is real. A gateway that tail-samples 50,000 traces at 10 seconds of wait holds roughly 2-5 GiB depending on span count.

How to configure it

A production chain for a logs pipeline on a per-host agent. The order is the discipline.

# /etc/otelcol/config.yaml

processors:
  # 1. memory_limiter first - the gate that prevents OOM.
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

  # 2. filter - drop noise before any other work.
  filter:
    logs:
      exclude:
        match_type: regexp
        bodies:
          - '.*healthcheck.*'

  # 3. resource - stamp static labels.
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: host.name
        from_attribute: host.name
        action: insert

  # 4. resourcedetection - add cloud/host metadata once.
  resourcedetection:
    detectors: [system, ec2]
    timeout: 2s
    override: false

  # 5. attributes - mutate per-entry fields.
  attributes:
    actions:
      - key: parsed.level
        from_attribute: level
        action: insert

  # 6. batch last - coalesce for exporter efficiency.
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 10000

The chain order is the contract. memory_limiter first; batch last; the rest in between in the order that minimises wasted work. The principle is to drop early, enrich before batching, and batch for exporter efficiency.

How to validate it

Validation is a parse-check plus a runtime check of the processor counters.

# CONFIGURATION: parse-check against the schema.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: confirm the batch processor is emitting.
curl -s http://localhost:8888/metrics | grep otelcol_processor_batch

otelcol_processor_batch_batch_send_size_count{bucket="8.192e+03"} 12
otelcol_processor_batch_batch_send_size_sum 98432
otelcol_processor_batch_metadata_block_cardinality 4218

# READ-ONLY: confirm the memory limiter is gating.
curl -s http://localhost:8888/metrics | grep otelcol_processor_memory_limiter

otelcol_processor_memory_limiter_accepted 42183
otelcol_processor_memory_limiter_refused 0

If batch_send_size_count is flat, the batch is not emitting; the timeout is too long or the input is too sparse. If memory_limiter_refused is non-zero, the heap is exceeding the limit and the chain is refusing data; either lower the limit, raise the limit, or reduce the per-record work.

How it can fail

Six failure modes specific to processors.

1. The memory_limiter placed after batch. The chain grows the batch unbounded; the limiter never sees the queue. Symptom: process_runtime_total_alloc_bytes climbs to 2-3x the configured limit before the kernel OOM-kills the process.
2. The filter placed after batch. The batch consumes capacity for entries that the filter would have dropped. Symptom: per-call payload stays high; the exporter throughput drops; the receivers back-pressure.
3. The resourcedetection detector that timed out. The ec2 detector requires the IMDS endpoint; in a network without IMDS the detector hangs for the full timeout. Symptom: the collector Start sequence takes 2+ seconds; the timeout does not bound the actual wait.
4. The transform processor with a backtracking regex. A regex like (a+)+$ runs pathological backtracking on a non-matching input. Symptom: the CPU profile shows the transform processor dominating; the batch_send_size drops; the chain back-pressures.
5. The tail_sampling processor that ran out of traces. The num_traces is set to 5,000 but the trace volume is 50,000 per second. Symptom: the processor logs “dropping trace because policy decision wait exceeded”; the sampler decisions are biased toward the most recent traces.
6. The resource processor that mutated a required label. The processor upserts service.name to a static value; the application set it per-instance. Symptom: every entry in Loki carries the same service_name; dashboards return nothing.

How to troubleshoot it

When the chain is not behaving, the diagnostic order matters.

1. Confirm the chain order. Read service.pipelines.*.processors from top to bottom. memory_limiter must be first; batch must be last. A reorder is the most common fix.
2. Confirm the per-processor counters. The otelcol_processor_*_accepted and otelcol_processor_*_refused metrics tell you whether the processor is doing its job.
3. Confirm the heap is bounded. process_runtime_total_alloc_bytes should oscillate around the configured limit; a monotonic climb means the limiter is not in the chain or is in the wrong place.
4. Profile the per-processor cost. pprof is an extension; enable it, capture a CPU profile, and identify the processor that dominates.
5. Test the chain in isolation. Disable every processor, re-enable one at a time, and measure the throughput. The processor that halves the throughput is the one that is misconfigured.

Security implications

Processors are the right place to enforce data hygiene.

• Redaction. The attributes and resource processors support action: delete and action: hash. Use them to strip credentials and PII before the data leaves the host.
• Filter. The filter processor can drop entries that match a sensitive body pattern (for example, a request that carries a credit-card number in the URL).
• Sampling. The tail_sampling and probabilistic_sampler processors reduce the data volume. A probabilistic sampler that drops 99% of traces preserves privacy at the cost of coverage.

Performance implications

Each processor has a per-record cost. The right chain is the chain that minimises the cost on entries that will be dropped or summarised.

• memory_limiter. The cost is a periodic heap check (every check_interval). The default 1 second is fine; 100 milliseconds is overhead with no benefit.
• filter. The cost is the cost of evaluating the expression. A regexp match on the body is cheap; a backtracking pattern is catastrophic.
• resource. The cost is the cost of mutating the resource attributes. Negligible at modest rates; measurable at millions of records per second.
• attributes. The cost is similar to resource. The per-entry mutation is the hot path.
• resourcedetection. The cost is a one-shot detection at Start. Subsequent runs of the chain do not re-detect.
• transform. The cost is the cost of evaluating the Transformation Language. A regex on every body is expensive; a set() is cheap.
• batch. The cost is the cost of buffering. A larger batch is a larger memory footprint; the timeout is the latency the entry waits in the buffer.
• tail_sampling. The cost is the cost of holding the trace in memory for the decision window. A 10-second window at 10,000 spans per second holds roughly 100,000 spans.

Production guidance

• memory_limiter first. The chain order is the discipline. A batch placed before memory_limiter grows unbounded; the process OOMs.
• batch last. The processor coalesces for exporter efficiency; placing it earlier wastes capacity on entries that subsequent processors will drop.
• Drop early, enrich late. The filter processor should run before any expensive work. The resource processor should run before batch so the labels are present when the batch is exported.
• Smoke test the chain. Ship a known log line with a unique marker and confirm it arrives with the expected resource attributes and the expected fields within ten seconds.

Verification

You should now be able to answer:

• What is the difference between attributes and resource, and why does it matter for Loki labels?
• Why must memory_limiter be the first processor in the chain, and what happens if it is not?
• Where should batch sit in the chain, and what is the trade-off between send_batch_size and timeout?
• What does tail_sampling cost in memory, and when is the cost worth paying?
• How does the per-processor counter (otelcol_processor_*_accepted and otelcol_processor_*_refused) help diagnose a misbehaving chain?

Quiz