OTLP

A payment service emits 50,000 spans per second. The OTel SDK ships them over OTLP to the local Collector. The Collector receives them on port 4317 (gRPC). The gateway Collector downstream receives them on port 4317 from the local Collector. The application is healthy. The traces are arriving. Then a single host starts emitting 5 million spans per second during a retry storm. The local Collector’s batch span processor queues the records; the gRPC stream back-pressures; the SDK retries with exponential back-off; the application thread pool stalls. The fix is not a code change — it is a wire-protocol discipline.

OTLP is the OpenTelemetry Line Protocol, the wire format the OTel SDKs and Collectors speak to each other. It is defined as a set of protobuf messages over either gRPC or HTTP/protobuf. Every signal — metrics, logs, traces, profiles — ships in the same envelope, on the same port, over either transport. The lesson that follows names the transports, names the request boundaries, and walks the configuration and failure modes of OTLP at the wire boundary.

What it is

OTLP is a wire protocol that carries telemetry records from a producer (an SDK or a Collector) to a consumer (a Collector or a backend). The protocol is defined as protobuf messages (opentelemetry-proto) over two transports:

• gRPC — a bidirectional HTTP/2 stream on port 4317 by default. The producer opens one stream per signal type and sends ExportRequest messages; the consumer replies with ExportResponse messages. The stream is persistent; the producer batches records into the request.
• HTTP/protobuf — a unidirectional HTTP/1.1 or HTTP/2 POST on port 4318 by default. The producer POSTs a binary protobuf body to /v1/traces, /v1/metrics, or /v1/logs. The consumer replies with a protobuf response. Each POST is one request; each response is one reply.

Both transports carry the same protobuf messages. The difference is the framing: gRPC multiplexes many requests over a single HTTP/2 connection; HTTP/protobuf is one request per HTTP round-trip.

The default ports are the IANA Service Name and Transport Protocol Port Number Registry assignments for opentelemetry- grpc (4317) and opentelemetry-http (4318). Both are unprivileged ports (above 1024) so an unprivileged process can bind them without root.

Why two transports

gRPC and HTTP/protobuf serve different environments.

gRPC is the production default for SDK-to-Collector and Collector-to-Collector. HTTP/protobuf is the right choice when the traffic crosses an HTTP-only proxy, a serverless function boundary, or a browser. The protocol is the same; the transport choice is operational.

Why a sysadmin cares

Three failure shapes appear when the wire protocol is treated as “magic happens between SDK and Collector”.

1. The payload that hit the size limit. The gRPC default maximum message size is 4 MiB. A service that emits high-cardinality attributes can produce an ExportLogServiceRequest larger than 4 MiB. The receiver rejects the message with RESOURCE_EXHAUSTED; the SDK retries; the queue grows. The fix is to raise the receiver’s max_recv_msg_size or to reduce the per-record payload.
2. The retry storm that overwhelmed the gateway. A transient backend outage caused every Collector to retry every pending batch. The retries coincided; the gateway’s gRPC server hit its connection limit; new connections were refused. The fix is the sending_queue and a back-off discipline on the exporter, not a code change.
3. The TLS bundle that expired silently. The Collector’s CA bundle has not been refreshed since deploy. The receiver rejects the TLS handshake with x509: certificate signed by unknown authority. The SDK sees a connection failure and starts retrying. The error is loud in the SDK log and invisible in the receiver log. The fix is the CA bundle rotation discipline.

OTLP is the wire boundary. The configuration lives at the receiver and the exporter; the discipline is to understand both.

How it works

The mental model. The SDK produces records, the batch processor coalesces them into an ExportRequest, and the transport sends the request to the receiver.

   SDK (Go, Python, Java, ...)
       |
       | batch timeout or batch size reached
       v
   ExportRequest (protobuf)
   {
     resource_metrics: [
       { resource: { service.name: "checkout" },
         scope_metrics: [
           { scope: { name: "otelpython" },
             metrics: [ ... ]
           }
         ]
       }
     ]
   }
       |
       v
   gRPC stream (HTTP/2) OR HTTP POST
       |
       v
   Collector OTLP receiver
       |
       v
   pipelined: receivers -> processors -> exporters

The ExportRequest envelope is the same shape for metrics, logs, traces, and profiles. The only difference is the message type name (ExportMetricsServiceRequest, ExportTraceServiceRequest, ExportLogsServiceRequest, ExportProfilesServiceRequest) and the HTTP path (/v1/metrics, /v1/traces, /v1/logs, /v1/profiles).

Payload size

The gRPC default is 4 MiB per message. The HTTP/protobuf default depends on the receiver; the OTel Collector accepts up to its configured max_request_size (no default; unbounded). The OTel Collector lets the operator set both:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size: 16777216  # 16 MiB
        max_concurrent_streams: 100
      http:
        endpoint: 0.0.0.0:4318
        max_request_size: 16777216   # 16 MiB

A larger max_recv_msg_size trades memory pressure on the receiver against fewer retries. A reasonable production value is 16 MiB. Beyond that, the typical cause is a per-record payload problem (high-cardinality attributes, oversized log bodies), not a wire-protocol problem.

Compression

gRPC supports per-message compression. The OTel SDKs send gzip-compressed messages by default; the Collector accepts gzip and uncompressed.

HTTP/protobuf does not specify compression at the wire layer; the SDK and the Collector negotiate compression via the Content-Encoding header. gzip is the production default.

A compressed payload is typically 5-10x smaller than the uncompressed equivalent for typical telemetry. The trade-off is CPU on the sender and the receiver against network bandwidth. On a fast LAN, compression is wasted CPU. On a WAN or a metered link, compression is the difference between a working pipeline and a backed-up one.

TLS

OTLP over gRPC supports TLS via standard HTTP/2 over TLS. The OTel Collector accepts a tls block on the receiver:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/otelcol/tls/server.crt
          key_file: /etc/otelcol/tls/server.key
          client_ca_file: /etc/otelcol/tls/ca.crt
          # Enforce client certificate authentication.
          # Without this, mTLS is server-only.

The client_ca_file enables mutual TLS — every client must present a certificate signed by the CA. In agent mode this is the right answer when the SDK is on a different host. In gateway mode, mTLS is the right answer when the receiver is exposed on a network that is not already protected.

HTTP/protobuf supports TLS the same way; the configuration is the same tls block.

Retry semantics

gRPC has its own retry semantics. The OTel SDK does not retry OTLP exports at the gRPC layer; it retries at the BatchSpanProcessor (and equivalents for metrics and logs) level with exponential back-off.

The Collector exporter retries at the exporter’s sending_queue level: when the consumer is unreachable, the exporter queues the batch and retries with back-off. The sending_queue is bounded by num_consumers and the queue size. When the queue fills, the exporter drops with resource_exhausted rather than blocking.

The discipline is to set sending_queue with a disk-backed file_storage extension on the gateway. Agent-mode collectors can run with the default in-memory queue because the host reboot is the recovery path.

How to configure it

A Collector that receives OTLP on both transports with TLS, proper message size limits, and a downstream OTLP exporter configured with the sending queue and back-off.

# /etc/otelcol/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        # Raise the message size for high-cardinality payloads.
        max_recv_msg_size: 16777216
        # Limit concurrent streams to protect against fan-in.
        max_concurrent_streams: 100
        # Read and write timeouts; default is 10s.
        read_timeout: 30s
        write_timeout: 30s
        # gRPC keepalive to detect half-open connections.
        keepalive:
          server_parameters:
            min_time_between_pings: 10s
            permit_without_stream: true
        tls:
          cert_file: /etc/otelcol/tls/server.crt
          key_file: /etc/otelcol/tls/server.key
          client_ca_file: /etc/otelcol/tls/ca.crt
      http:
        endpoint: 0.0.0.0:4318
        max_request_size: 16777216
        tls:
          cert_file: /etc/otelcol/tls/server.crt
          key_file: /etc/otelcol/tls/server.key

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/gateway:
    endpoint: gateway.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/ca.pem
    # The sending queue survives a gateway outage.
    sending_queue:
      enabled: true
      num_consumers: 10
      # 50 MB memory queue. Disk queue via file_storage is also possible.
      queue_size: 5000
      # Initial retry delay; doubles up to max_delay.
      retry_initial_interval: 5s
      retry_max_interval: 30s
      retry_max_elapsed_time: 300s

extensions:
  # Disk-backed queue for the sending_queue.
  - file_storage/queue:
      directory: /var/lib/otelcol/queue
      timeout: 10s

service:
  extensions: [file_storage/queue]
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlp/gateway]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlp/gateway]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlp/gateway]

The receiver raises max_recv_msg_size to 16 MiB to accept high-cardinality batches. The exporter uses a sending_queue with back-off to survive a gateway outage without retry storms. The file_storage extension persists the queue to disk.

How to validate it

Validate the wire protocol with grpcurl against the gRPC receiver and curl against the HTTP receiver.

# READ-ONLY: gRPC health check (the OTel Collector supports
# the grpc.health.v1.Health service on the OTLP port).
grpcurl -plaintext otelcol:4317 grpc.health.v1.Health/Check

{
  "status": "SERVING"
}

# READ-ONLY: gRPC reflection (the OTel Collector exposes the
# OTLP services via reflection by default in dev mode).
grpcurl -plaintext otelcol:4317 list

grpc.health.v1.Health
opentelemetry.proto.collector.logs.v1.LogsService
opentelemetry.proto.collector.metrics.v1.MetricsService
opentelemetry.proto.collector.trace.v1.TraceService

# READ-ONLY: HTTP receiver health.
curl -s http://otelcol:4318/v1/traces -X POST \
  -H 'Content-Type: application/x-protobuf' \
  --data-binary '' -o /dev/null -w '%{http_code}\n'

415

A 415 from the HTTP receiver means “I received a request but the content type is acceptable; the body was empty so I could not parse it”. The 415 confirms the receiver is alive and parsing the wire protocol.

# READ-ONLY: confirm the exporter is shipping.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_spans{exporter="otlp/gateway"} 1024

How it can fail

Five failure modes that arise at the OTLP wire boundary.

1. The payload that exceeded the message size limit. A service emits a batch with a 12 MiB single record (a log body with an oversized stack trace). Symptom: the receiver rejects the message with RESOURCE_EXHAUSTED; the SDK retries; the queue grows. The fix is to raise max_recv_msg_size on the receiver and to reduce the per-record payload at the source.
2. The TLS bundle that expired. The Collector’s CA bundle has not been refreshed since deploy. Symptom: the TLS handshake fails with x509: certificate signed by unknown authority; the SDK retries; the receiver log is silent. The fix is the CA bundle rotation discipline — see lesson 06-otel-collector-security.
3. The gRPC connection that hung. A half-open HTTP/2 connection (a NAT timeout, a load-balancer idle timeout) causes the receiver to wait indefinitely. Symptom: the SDK sees back-pressure; the receiver log is silent; no error is surfaced. The fix is keepalive.server_parameters so the receiver pings idle connections.
4. The retry storm that overwhelmed the gateway. A transient backend outage caused every collector to retry every pending batch. The retries coincided. Symptom: the gateway’s gRPC server hit its connection limit; new connections refused; the receivers logged connection refused. The fix is sending_queue with a back-off policy and a per-consumer limit.
5. The decompression bomb. A malicious or buggy producer sends a 1 MB gzipped message that decompresses to 8 GiB. Symptom: the receiver exhausts memory and crashes. The fix is the memory_limiter processor and a max_request_size cap.

How to troubleshoot it

When the wire is silent, the diagnostic order is from the network up.

1. Is the port open? ss -tlnp | grep 4317. If the process is not listening, the receiver is not running or the wrong port is bound.
2. Is the TLS handshake succeeding? openssl s_client -connect otelcol:4317 -servername otelcol. A successful handshake with a valid certificate; a failure with unknown authority means a CA bundle problem.
3. Is the gRPC stream healthy? grpcurl against the health service; a non-SERVING status means the receiver is overloaded.
4. Are the records arriving? otelcol_receiver_accepted_* on the metrics endpoint. Zero means the SDK is not exporting or the network is blocked.
5. Are the records leaving? otelcol_exporter_sent_* and otelcol_exporter_failed_*. A growing failed counter with a non-zero dropped is a queue-overflow problem; a growing failed with a non-zero connection_refused is a network problem.

Security implications

OTLP is the wire boundary; the security surface is at the listener and the TLS configuration.

• The default ports. 4317 (gRPC) and 4318 (HTTP). Both bind to 0.0.0.0 by default in the OTel Collector. In agent mode, bind to localhost or to a private interface. In gateway mode, restrict the listener with a NetworkPolicy.
• mTLS. The client_ca_file on the receiver enables mTLS — every client must present a certificate signed by the CA. Without it, the receiver accepts any TLS connection.
• The CA bundle. Stale CA bundles are the most common cause of silent shipping failure. The bundle must be rotated before the receiver’s certificate expires; see lesson 06-otel-collector-security.
• Header leakage. The OTLP receiver passes HTTP headers through to the processor pipeline. A header that carries a bearer token is a leak unless the processor explicitly drops it.

Performance implications

The wire is the network and the CPU on both ends.

• Compression. Default is gzip. Compression is approximately 5-10x for typical telemetry. The cost is CPU on the sender and the receiver. On a fast LAN, turn it off (compression: none) to save CPU. On a WAN or metered link, keep it on.
• Batching. The OTel SDK batch span processor coalesces records. A larger send_batch_size with a longer timeout trades latency for throughput. The default timeout: 200ms is too aggressive for a gateway; 5 seconds is a more realistic starting point.
• Connection reuse. gRPC multiplexes many requests over a single HTTP/2 connection. The Collector should configure max_concurrent_streams to bound the number of in-flight requests per connection.
• Disk queue. The file_storage extension backs the exporter’s sending_queue on disk. The trade-off is durability against disk I/O. SSD-backed host disk is appropriate.

Production guidance

• Use gRPC on the LAN. HTTP/protobuf across the LAN trades throughput for transport simplicity. gRPC over HTTP/2 multiplexes many requests over one connection.
• Raise max_recv_msg_size to at least 16 MiB. The default of 4 MiB rejects high-cardinality batches.
• Enable mTLS on the gateway. The collector’s TLS block accepts a client_ca_file; set it.
• Configure sending_queue with back-off. The exporter retries exponentially. The gateway’s sending queue with file_storage survives a downstream outage.
• Rotate the CA bundle before expiry. A stale CA bundle is silent shipping failure.
• Monitor the wire. otelcol_receiver_refused_*, otelcol_exporter_failed_*, and the sending_queue size are the production signals.

Verification

You should now be able to answer:

• What are the two transports OTLP supports, and what port does each use?
• What is the protobuf envelope for an OTLP request, and how does it differ across signal types?
• How does OTLP retry behave at the SDK level and at the Collector exporter level?
• What is the right discipline for max_recv_msg_size and sending_queue in production?
• Which failure mode is most likely when the receiver rejects with RESOURCE_EXHAUSTED?

Quiz