Collector Not Running

A page fires at 03:14 about a 5xx spike on the payment service. The on-call engineer opens Grafana and runs the standard {job="payments"} query. The panel is empty. The engineer opens a terminal on the payment host and runs systemctl status alloy. The unit is inactive (dead). The start time is two hours ago. The exit code is 1. The last log line is level=error msg="Reload failed" err="...". The collector has not been running for two hours. Every log line the payment service emitted in those two hours is gone.

“Collector not running” is the second most common cause of a missing-logs incident. The line was written to the source file. The collector was not running. The distributor received nothing. The diagnosis is one command away; the cost of missing the command is two hours of lost telemetry.

What it is

“Collector not running” is the condition where the log-shipper process on the host (Alloy, OTel Collector, or legacy Promtail) is not running. The agent has stopped. The source file is being written to, but the agent is not reading it. The distributor’s loki_distributor_bytes_received_total for the suspect tenant is flat. The collector’s localhost metrics endpoint is not responding.

The shape of the failure is specific. The application is running. The source file is growing. The Loki distributor is receiving bytes from other tenants but not from the suspect host. The collector process on the suspect host is absent.

  Application         Source file         Collector        Loki
  +-----------+      +-------------+      +-----------+    +--------+
  | running   | ---> | growing     | -X-> | not       |    | no     |
  |           |      |  (mtime OK) |      | running   |    | bytes  |
  +-----------+      +-------------+      +-----------+    +--------+

The arrow from the source file to the collector is broken. The arrow from the collector to Loki is unavailable. The host silently loses every log line between the moment the collector stopped and the moment the operator notices.

Why a sysadmin cares

A stopped collector is the cheapest failure shape to detect and the most expensive failure shape to leave in place. The detection cost is one systemctl status call. The non-detection cost is the cost of every log line that the service emits between the stop and the next deploy that restarts the agent.

The pattern is also operationally common. Config reload failures, OOM kills, liveness probe restart loops, and missing dependencies are the four most common causes. Each appears in production observability stacks at least once per quarter. The discipline is to check the unit state first, the agent log second, and the source file third.

How it works

The collector runs as a long-lived process supervised by systemd or a container runtime. The supervisor restarts the process on exit (subject to Restart= and StartLimit* configuration). The collector accepts telemetry on its listeners, transforms it through the pipeline, and ships it to the backends. The localhost metrics endpoint exposes counters for every stage.

A failure in the process lifecycle shows up in three places: the supervisor’s view, the process’s own log, and the localhost metrics endpoint. The supervisor’s view tells you whether the process is running. The process’s own log tells you why it stopped. The localhost metrics endpoint tells you what the process was doing before it stopped.

Two patterns make the failure easy to miss. First, a config reload that fails can leave the process running with the old config or stopped with the new config, depending on the implementation. Grafana Alloy stops the process on a failed reload; OTel Collector continues with the old config. The operator must read the log to know which. Second, an OOM kill leaves no log in the agent’s own file; the kernel logs the kill to dmesg or the systemd journal. The operator must read the journal to find the cause.

How to configure it

The lesson does not introduce a new collector configuration; it introduces a supervisor configuration that handles the common failure modes. The minimum viable systemd unit for production:

# /etc/systemd/system/alloy.service
[Unit]
Description=Grafana Alloy
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=alloy
Group=alloy
ExecStart=/usr/bin/alloy run /etc/alloy/config.alloy \
  --storage.path=/var/lib/alloy/data
Restart=always
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=10
# Cap memory so the OOM killer acts predictably.
MemoryMax=1G
MemoryHigh=800M
OOMScoreAdjust=-100
# Give the agent a moment to drain on stop.
TimeoutStopSec=30s
# Capture stdout and stderr to the journal.
StandardOutput=journal
StandardError=journal
# Reload on SIGHUP.
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

The unit does four things that matter for the failure modes in this lesson. Restart=always re-runs the process on any exit. StartLimitBurst=10 in a 60-second window prevents a crash loop from eating the supervisor’s restart budget. MemoryMax=1G and MemoryHigh=800M give the kernel a predictable budget for the OOM killer. TimeoutStopSec=30s gives the agent a moment to drain before the supervisor sends SIGKILL.

In Kubernetes, the equivalent is a Deployment with a livenessProbe that hits /status on the collector’s health-check extension:

# snippet from /etc/otelcol/k8s-deployment.yaml
spec:
  template:
    spec:
      containers:
        - name: otelcol
          livenessProbe:
            httpGet:
              path: /status
              port: 13133
            initialDelaySeconds: 10
            periodSeconds: 10
          resources:
            limits:
              memory: 1Gi
              cpu: 500m

The liveness probe is the supervisor’s view in a container orchestrator. The path: /status endpoint is the health_check extension’s response.

How to validate it

The diagnostic order for hop 3. Every command is read-only.

# Step 1: is the unit running?
systemctl status alloy --no-pager | head -15

● alloy.service - Grafana Alloy
     Loaded: loaded (/etc/systemd/system/alloy.service; enabled)
     Active: active (running) since Fri 2026-08-14 03:00:14 UTC
   Main PID: 18421 (alloy)
      Tasks: 11 (limit: 18976)
     Memory: 132.4M
       CPU: 4.211s

An Active: active (running) is healthy. An Active: inactive (dead) or failed is the first smoking gun.

# Step 2: what does the agent log say?
journalctl -u alloy -n 50 --no-pager

Aug 14 03:00:14 host alloy[18421]: ts=2026-08-14T03:00:14Z level=info msg="starting"
Aug 14 03:00:14 host alloy[18421]: ts=2026-08-14T03:00:14Z level=info msg="loaded config"
Aug 14 03:00:14 host alloy[18421]: ts=2026-08-14T03:00:14Z level=info msg="ready"

A clean start and a ready line is healthy. An error between starting and ready is the cause.

# Step 3: is the metrics endpoint bound?
ss -tlnp | grep -E ':12345|:8888'

LISTEN 0  128  0.0.0.0:12345  0.0.0.0:*  users:(("alloy",pid=18421,fd=7))

A bound socket on the metrics port is healthy. An absent socket is the smoking gun for a stopped process.

# Step 4: is the process running?
ps -ef | grep -E 'alloy|otelcol' | grep -v grep

alloy  18421  1  0 03:00 ?  00:00:04 /usr/bin/alloy run /etc/alloy/config.alloy

A running process is healthy. An absent process is the smoking gun.

# Step 5: is the agent pushing?
curl -s http://localhost:12345/metrics | grep -E 'loki_write_sent_entries_total'

loki_write_sent_entries_total{...} 18421

A non-zero counter is healthy. A zero counter over a suspect window is the symptom of a pipeline started but not pushing.

# Step 6: in Kubernetes, is the pod running?
kubectl get pods -n monitoring -l app=otelcol -o wide

NAME          READY   STATUS    RESTARTS   AGE
otelcol-abc   1/1     Running   0          14d

A Running pod with zero restarts is healthy. A CrashLoopBackOff or a high RESTARTS count is the smoking gun for a stopped collector.

How it can fail

Six specific failure shapes appear in production. Each one maps to a recognisable symptom.

1. Config reload failed. An alloy fmt or otelcol validate ran clean but the runtime rejected the new config; the supervisor stopped the process. Symptom: the unit is inactive (dead); the agent log shows level=error msg="failed to reload"; the last good config is in the agent log but not on disk. The fix is to revert the config and reload.
2. OOM killed. The collector’s memory budget was exceeded. The kernel killed the process. Symptom: the unit is inactive (dead); the journal shows kernel: Out of memory: Killed process 18421 (alloy); the agent log is silent after the kill. The fix is to raise MemoryMax or to reduce the agent’s buffer budget.
3. Segfault. A corrupted state or a bug in the agent. Symptom: the unit is inactive (dead); the journal shows segfault at ...; the agent log is silent. The fix is to upgrade the agent and to add a Restart=always if not already present.
4. Missing dependency. The YAML names a component that the binary does not ship (e.g., filelog in a core-only OTel Collector). Symptom: the unit is inactive (dead); the agent log shows component "filelog" not found in the binary. The fix is to switch to the contrib distribution.
5. Liveness probe restart loop. The collector’s health endpoint is not ready in time; the orchestrator kills the pod. Symptom: the pod is in CrashLoopBackOff; the pod log shows Liveness probe failed: HTTP 503; the agent log is silent. The fix is to raise initialDelaySeconds or to check the health endpoint’s dependencies.
6. Disk full or read-only filesystem. The collector cannot write its buffer or its log file. Symptom: the unit is inactive (dead); the agent log shows filesystem error or no space left on device. The fix is to free the disk or to redirect the buffer to a different path.

How to troubleshoot it

The diagnostic order for hop 3. Each step is read-only.

1. Confirm the unit or pod state. systemctl status alloy or kubectl get pods. The supervisor’s view is the first read.
2. Read the agent log. journalctl -u alloy -n 50 or the pod log. The log carries the cause of the stop.
3. Check the metrics endpoint. curl -s http://localhost:12345/metrics. A bound port is healthy; an unbound port is the symptom of a stopped process.
4. Check the supervisor’s restart count. systemctl show alloy -p NRestarts or the pod’s RESTARTS column. A high restart count is the symptom of a crash loop.
5. Check the journal for OOM or segfault. journalctl -k | grep -E 'killed|segfault'. The kernel’s view is the source of the cause for an OOM kill or a segfault.
6. Check the disk and the filesystem. df -h /var/lib /var/log. A full disk or a read-only mount is the symptom of a hardware or orchestration failure.

Security implications

The collector’s process exposes the agent log file, the metrics endpoint, and the supervisor’s view to anyone with shell on the host. Three risks follow:

• The agent log may contain secrets. A logger that emits URLs, tokens, or credentials writes them to the agent log on reload. The log file is readable by the alloy user; the file permissions should be 0640 owned by root:alloy.
• The metrics endpoint exposes counters. Bind the metrics endpoint to localhost or to a private interface. The metrics are not sensitive, but the surface is unnecessarily wide otherwise.
• The supervisor’s view is on the host. The systemd unit file is readable by any user with shell. The ExecStart= line may contain command-line arguments that are sensitive. Use a config file with 0600 permissions and reference it from the unit.

Performance implications

The collector’s runtime cost is roughly 100-150 MiB RAM and 50-100 millicores CPU at modest line rates. The cost grows with batch size, queue depth, and per-record transform work. The OOM kill is the most common performance failure mode: a memory budget that is too low for the line rate. The fix is to size the budget against the worst-case line rate, not the average.

The restart loop is the second most common performance failure mode. A collector that crashes on every reload consumes the supervisor’s restart budget and stops auto-restarting. The fix is to add Restart=always and StartLimitBurst=10 in a 60-second window. The cost of the budget is the cost of handling the rare crash loop gracefully.

Production guidance

• Run the collector as a dedicated user. The alloy user should have read access to the intended log paths and no more. The agent log directory should be 0640 root:alloy.
• Set MemoryMax and MemoryHigh explicitly. The default is unbounded. A buffer that grows under a Loki outage is the symptom of a memory budget that is too high. The unit should declare the budget.
• Mirror the agent log to the journal. StandardOutput=journal ensures the agent log is captured by the system journal and is queryable from a single tool.
• Bind the metrics endpoint to localhost. The metrics are scraped by the host agent; the cluster network should not see the endpoint.
• Alert on the supervisor’s restart count. systemctl show alloy -p NRestarts is the source of the restart counter. A non-zero count is the symptom of a crash loop.
• Test the reload. A SIGHUP that fails is the difference between a quiet reload and a silent outage. The smoke test is a reload followed by a known-line query in Loki.

Verification

You should now be able to answer:

• Which single command on the host is the smoking gun for a stopped collector?
• What does a StartLimitBurst=10 setting in the systemd unit do, and why is it important?
• How does an OOM kill appear in the agent log?
• What is the difference between a config reload failure in Grafana Alloy and a config reload failure in OTel Collector?
• Which Kubernetes field is the equivalent of the systemd unit’s Restart=always?

Quiz