Collector Down

A trace lookup returns trace not found. The on-call engineer follows link A and link B; both are healthy. The SDK is initialised; the traceparent header survives every boundary. The next link is the collector. The application log shows an SDK warning: OTLP exporter: export timeout exceeded. The SDK’s otelcol_exporter_sent_spans counter is unavailable because it is on the collector side. The on-call engineer runs kubectl get pods -n observability. The collector pod is in CrashLoopBackOff. The previous crash log shows unknown receiver key: "otlp". A typo in the collector config; the receiver stanza was never parsed; the OTLP ports were never bound. Link C is broken. Every earlier link is irrelevant until the collector is healthy.

What it is

A “collector down” incident is the failure shape where the OpenTelemetry Collector is not accepting OTLP traffic from application hosts. The application SDK is running, the exporters are wired, the propagators are correct, but the OTLP exporter cannot reach the collector. Three sub-shapes exist:

1. Collector process down. The collector pod is not running. The process exited, the pod was evicted, the DaemonSet crashed. The OTLP port is not bound.
2. Collector up but receivers disabled. The collector process is running, but the OTLP receiver stanza was not loaded (a typo in the YAML, a missing receivers: entry on the pipeline). The process is up; the ports are not bound.
3. Collector up, receivers bound, but unreachable from the application host. The collector’s OTLP port is bound, but a network policy, a service mesh, a missing firewall rule, or a wrong DNS name makes the port unreachable from the application’s network namespace.

The three sub-shapes have three different fixes. The wrong fix is to scale the collector when the receiver is misconfigured; that adds replicas of a collector that still binds no ports.

Why a sysadmin cares

Link C is the second most common cause of a missing trace, after link A. A collector that was healthy yesterday and is unhealthy today is a typical incident shape: a config push, a base image change, a network policy change, or a node eviction. The investigation must be cheap because the fix must be fast.

The collector is also a single point of failure in many deployments. A single-replica collector with no replicas>=2 is a deployment smell; a collector with replicas>=2 but a shared network policy that drops traffic to one namespace is a different smell. The diagnostic must distinguish “the collector is down” from “the collector is up but one of its replicas is down”.

How it works

The collector exposes three logical surfaces for OTLP:

  +---------------------------------------+
  |              Application host         |
  |                                      |
  |  SDK                                  |
  |   |-- OTLP gRPC exporter              |
  |   |     -> 4317 (or custom)           |
  |   |                                   |
  |   '-- DNS -> otel-collector.svc:4317  |
  +---------------------------------------+
                   |
                   |  network path
                   |  (DNS, kube-proxy, NetworkPolicy, MTU)
                   v
  +---------------------------------------+
  |          Collector pod                |
  |                                      |
  |  receivers:                           |
  |   otlp:                               |
  |    protocols:                         |
  |     grpc:                             : 4317
  |     http:                             : 4318
  |                                      |
  |  processors: batch, attributes, ...   |
  |  exporters: otlp/tempo, ...           |
  +---------------------------------------+
                   |
                   v
  +---------------------------------------+
  |          Tempo                        |
  +---------------------------------------+

Three things must be true for a span to reach the collector:

• DNS resolution. The application’s OTLP endpoint env var resolves to an IP that exists.
• Network reachability. The IP responds to TCP probes on port 4317 (gRPC) or 4318 (HTTP).
• Receiver binding. The collector process is listening on the port and the receiver stanza is loaded in the pipeline.

The diagnostic confirms each in turn.

The collector’s health surfaces

The collector exposes several HTTP endpoints:

  :8889  /metrics     Prometheus metrics (default)
  :13133 /            Health check extension (default)
  :55679 /v1/trace    zPages trace debug (debug extension)
  :55680 /debug       zPages pipeline state (debug extension)

The /metrics endpoint exposes otelcol_receiver_accepted_spans, otelcol_exporter_sent_spans, and many others. The health check returns 200 OK when the receiver pipelines report healthy; 503 Service Unavailable when a receiver is not ready.

How to configure it

The configuration that makes link C diagnosable is the collector itself, plus the Kubernetes services and network policies around it.

Collector deployment

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: otel-collector
    namespace: observability
  spec:
    replicas: 3
    selector:
      matchLabels: { app: otel-collector }
    template:
      metadata:
        labels: { app: otel-collector }
      spec:
        containers:
          - name: otel-collector
            image: otel/opentelemetry-collector-contrib:0.110.0
            args:
              - --config=/etc/otelcol-contrib/config.yaml
            ports:
              - containerPort: 4317   # OTLP gRPC
              - containerPort: 4318   # OTLP HTTP
              - containerPort: 8889   # Prometheus self-observability
              - containerPort: 13133  # Health check
            readinessProbe:
              httpGet:
                path: /
                port: 13133
              initialDelaySeconds: 5
              periodSeconds: 5
            livenessProbe:
              httpGet:
                path: /
                port: 13133
              initialDelaySeconds: 30
              periodSeconds: 30

Severity: CONFIGURATION. Apply with kubectl apply; the rollout restarts pods (SERVICE-IMPACT).

Collector service

  apiVersion: v1
  kind: Service
  metadata:
    name: otel-collector
    namespace: observability
    labels: { app: otel-collector }
  spec:
    selector: { app: otel-collector }
    ports:
      - name: otlp-grpc
        port: 4317
        targetPort: 4317
      - name: otlp-http
        port: 4318
        targetPort: 4318
      - name: metrics
        port: 8889
        targetPort: 8889

Severity: CONFIGURATION.

Network policy

  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: allow-otlp-from-apps
    namespace: observability
  spec:
    podSelector: { matchLabels: { app: otel-collector } }
    policyTypes: [Ingress]
    ingress:
      - from:
          - namespaceSelector: { matchLabels: { name: apps } }
        ports:
          - port: 4317
            protocol: TCP
          - port: 4318
            protocol: TCP
      - from:
          - namespaceSelector: { matchLabels: { name: monitoring } }
        ports:
          - port: 8889
            protocol: TCP
          - port: 13133
            protocol: TCP

Severity: CONFIGURATION. Apply the policy; missing egress rules will block traffic silently.

How to validate it

Severity: READ-ONLY.

Confirm DNS resolution from the application host

  kubectl exec deploy/checkout-svc -- \
    nslookup otel-collector.observability.svc.cluster.local
  # Server:    10.96.0.10
  # Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
  # Name:      otel-collector.observability.svc.cluster.local
  # Address 1: 10.20.4.18

A missing DNS record or a wrong namespace is a silent link-C failure.

Confirm TCP reachability

  kubectl exec deploy/checkout-svc -- \
    nc -zv otel-collector.observability.svc 4317
  # otel-collector.observability.svc (10.20.4.18:4317) open

  kubectl exec deploy/checkout-svc -- \
    nc -zv otel-collector.observability.svc 4318
  # otel-collector.observability.svc (10.20.4.18:4318) open

A Connection refused is the collector not listening; a timeout is the network blocking the traffic; a No route to host is the kube-proxy routing wrong.

Confirm the collector is healthy

  kubectl exec deploy/otel-collector -n observability -- \
    wget -qO- http://localhost:13133/
  # {"status":"Server available","upSince":"2026-08-12T14:23:18Z"}

  # Or from inside the pod via curl
  kubectl exec deploy/otel-collector -n observability -- \
    curl -s http://localhost:13133/
  # {"status":"Server available"}

A 503 Service Unavailable means a receiver pipeline is not ready. The zpages extension at :55680/debug/pipelines shows which pipeline has failed to start.

Confirm the OTLP receivers are receiving

  curl -s http://otel-collector.observability.svc:8889/metrics | \
    grep -E '^otelcol_receiver_accepted_(spans|metric_points|log_records)'
  # otelcol_receiver_accepted_spans{receiver="otlp",transport="grpc"} 12842
  # otelcol_receiver_accepted_spans{receiver="otlp",transport="http"} 4211

A flat otelcol_receiver_accepted_spans counter with all transports is the link-C “collector up but no traffic” shape. Combined with nc -zv returning open, the cause is upstream of the network path.

Confirm collector pod state

  kubectl get pods -n observability -l app=otel-collector
  # NAME                             READY   STATUS             RESTARTS
  # otel-collector-7c4b8f9d8d-abcde  1/1     Running            0
  # otel-collector-7c4b8f9d8d-fghij  1/1     Running            0
  # otel-collector-7c4b8f9d8d-klmno  0/1     CrashLoopBackOff  5

A CrashLoopBackOff row is the link-C “collector process not running” shape.

How it can fail

Six failure shapes, ordered by frequency in production fleets:

1. Collector pod in CrashLoopBackOff after a config push. A YAML typo in the receiver stanza or a missing field causes the collector to fail to start. Symptom: the pod is in CrashLoopBackOff; the previous container log shows the YAML parse error; no OTLP ports are bound.

2. Network policy blocks OTLP from the application namespace. A new NetworkPolicy was applied to the collector namespace that allows ingress only from the monitoring namespace. The application namespace is blocked. Symptom: nc -zv otel-collector 4317 returns No route to host or times out; the receiver is healthy inside the collector pod.

3. OTLP exporter env var points to the wrong service. The application’s OTEL_EXPORTER_OTLP_ENDPOINT points to a service that does not exist (otel-collector:4317 instead of otel-collector.observability.svc:4317). The connection times out. Symptom: the SDK logs show connection refused repeatedly; the collector is healthy.

4. Collector replica count is zero after a Helm uninstall. A Helm chart removal deleted the collector deployment. The service still resolves to a stale endpoint. Symptom: kubectl get pods returns nothing; nc -zv times out; the service DNS exists but has no endpoints.

5. Service mesh sidecar missing on the collector. A sidecar injection policy was added to the observability namespace, but the collector pod was deployed without the sidecar. The mesh routes traffic to the sidecar address; the sidecar is absent. Symptom: connections time out; the pod itself is healthy.

6. /healthz and /metrics ports in use by another process. Another container on the host is bound to 8889 or 13133. The collector cannot bind them; the process exits. Symptom: CrashLoopBackOff; the log shows bind: address already in use.

How to troubleshoot it

The diagnostic order, link C first, cheapest signal first:

1. Pod state. kubectl get pods -n observability -l app=otel-collector. A non-Running pod is the cause.
2. Container log. kubectl logs deploy/otel-collector -n observability --previous. A YAML parse error, a bind error, or a panic in the previous container log is the exact cause.
3. Health check. curl / on port 13133 from inside the pod or via port-forward. A 503 means a receiver pipeline has failed to register.
4. TCP probe. nc -zv otel-collector.svc 4317 from the application host. A connection refused is the port not bound; a timeout is the network path blocked.
5. DNS check. nslookup from the application host. Wrong IP or missing record is the DNS misconfiguration.
6. Receiver metrics. otelcol_receiver_accepted_spans on the collector’s /metrics. Flat across all transports is upstream of the receiver.

Security implications

The OTLP port is the receiver surface. The receiver accepts unauthenticated traffic by default; the only filter is the network policy and the mTLS configuration.

• Network policy. Restrict ingress to known application namespaces. Block OTLP from the public internet and from untrusted namespaces.
• mTLS. OTLP gRPC supports mTLS natively. Configure tls.cert_file, tls.key_file, and tls.client_ca_file in the receiver stanza for production deployments.
• Authentication context. Use the auth_context extension to extract tenant identifiers from headers; do not rely on the source IP alone.

Performance implications

The collector is CPU-bound on decoding and network-bound on spans/sec. The receivers consume CPU on protobuf deserialization; the exporters consume network bandwidth on gRPC writes. The bottleneck is usually the exporters at the moment Tempo rejects spans.

The biggest performance trap is running a single-replica collector with a 50,000 spans/sec ingress rate. The collector becomes the bottleneck and drops spans at the receiver. The fix is horizontal scaling and a headroom of 2x expected peak.

The memory footprint is bounded by the batch processor’s queue size. A queue of 50,000 spans at 1 KiB per span is 50 MiB; this is well within typical pod memory limits. A queue that grows unbounded points at the exporter being slower than the receiver; the receiver starts dropping with otelcol_receiver_refused_spans.

Production guidance

• Run the collector with replicas>=3 for HA. The replicas=1 deployment is a single point of failure.
• Restrict ingress to the OTLP port with a NetworkPolicy. Block OTLP from any namespace that does not produce telemetry.
• Enable the health check extension and wire it into Kubernetes readiness probes. A collector that reports unhealthy should not be a load-balancer target.
• Scrape the collector’s /metrics from Prometheus. The otelcol_receiver_accepted_spans and otelcol_exporter_send_failed_spans counters are the link-C and link-D signals.

Verification

You should now be able to answer:

• What are the three sub-shapes of “collector down”?
• Which port is the OTLP gRPC receiver bound to by default?
• How do you confirm the collector is reachable from the application host?
• Which collector metric confirms the receivers are receiving?
• What is the cheapest link-C diagnostic?

Quiz