Skip to main content
RunBook Academy

← All checklists in Observability

Before deploymentotel-readiness

OpenTelemetry Collector Readiness

29 items ·24 critical ·5 warn ·0 info

Run this before a Collector carries production telemetry, and again after any change to the processor chain, the exporter set, the sampling configuration or the deployment topology. Those four are where the expensive mistakes live. Adding a receiver is nearly always safe; reordering a chain is not.

The items are ordered as the pipeline runs: what the binary can build, how the chain is wired, what bounds the memory, what enters, what leaves, how much is kept, and finally whether the whole thing would tell you it had stopped.

Where the answers come from

Three sources answer almost everything here. otelcol validate runs the same parse the runtime runs at start, offline and with a non-zero exit. otelcol components prints what the binary can actually build, which is the only honest way to know whether the config in front of you is deployable on the binary you have. Everything else comes from the self-telemetry endpoint, which is the collector reporting on its own pipeline.

Substitute your own paths, component addresses and job names — the ones below are the shapes this course uses, not an estate that exists. Where the collector runs in a container, the same commands run inside it; where several targets run in one process, one endpoint answers for all of them.

The failures that are quiet

A collector fails loudly at start and quietly afterwards. A missing component, an undeclared receiver, a wrong signal: all of these refuse to start, print the reason and the line, and are fixed by reading the error. Nothing on this list protects you from those, because they protect themselves.

What this checklist is actually for is the other set. A missing tenant header ships everything successfully to the wrong place. A limiter behind a batch processor lets the heap grow until the kernel intervenes. A filelog receiver without persistent checkpoints duplicates its files on every restart. An undersized tail sampler drops the rare traces it was installed to keep. In each case the process is running, the config is valid, the health check is green, and the data is wrong or absent.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Largest telemetry producer: ___ Date: ___________

Critical24 items

  1. otelcol components
  2. otelcol validate --config=/etc/otelcol/config.yaml
  3. grep -A2 "processors:" /etc/otelcol/config.yaml
  4. systemctl show otelcol -p MemoryMax -p MemoryCurrent
  5. curl -s http://localhost:8888/metrics | grep -E "otelcol_processor_memory_limiter_refused|otelcol_receiver_refused"
  6. curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted_log_records
  7. ss -tlnp | grep -E "4317|4318|8888|13133"
  8. grep -A6 "^exporters:" /etc/otelcol/config.yaml | grep -i "X-Scope-OrgID"
  9. curl -s http://localhost:8888/metrics | grep otelcol_exporter_send_failed
  10. curl -s http://localhost:8888/metrics | grep -E "otelcol_exporter_queue_size|otelcol_exporter_queue_capacity|otelcol_exporter_dropped"
  11. grep -n "debug" /etc/otelcol/config.yaml
  12. curl -s http://localhost:8888/metrics | grep -E "probabilistic_sampler_count_traces|tail_sampling_count_traces"
  13. curl -s http://localhost:8888/metrics | grep -E "otelcol_processor_tail_sampling_count_traces_(kept|dropped)"
  14. grep -n routing_key /etc/otelcol/config.yaml
  15. grep -nEi "(token|password|authorization):[[:space:]]*[^$]" /etc/otelcol/config.yaml
  16. curl -s -G http://meta-prom:9090/api/v1/query --data-urlencode 'query=up{job="otelcol"}' | jq -r '.data.result[] | "\(.metric.instance) \(.value[1])"'
  17. curl -s http://localhost:8888/metrics | grep -E "otelcol_receiver_accepted|otelcol_exporter_sent"
  18. otelcol --version

Warning5 items

  1. grep -nE "send_batch_size|send_batch_max_size|timeout:" /etc/otelcol/config.yaml
  2. grep -n max_recv_msg_size_mib /etc/otelcol/config.yaml
  3. grep -n -A4 retry_on_failure /etc/otelcol/config.yaml
  4. grep -n -A4 resourcedetection /etc/otelcol/config.yaml