Grafana Alloy vs OpenTelemetry Collector

A team runs Promtail on 800 hosts and Grafana Agent in the Kubernetes cluster. Both ship to Loki. The platform team publishes a single collector policy. The choice is between Grafana Alloy and the OpenTelemetry Collector. Both ship logs, metrics and traces. Both run in a DaemonSet or a sidecar. They differ in the configuration language, the receiver surface, and the migration shape from a Promtail fleet.

This lesson is the comparison. The trade-off is real. The wrong choice for an established OTel fleet is the wrong choice.

What it is

Grafana Alloy is Grafana Labs’ strategic telemetry collector. It is the successor to Grafana Agent and to Promtail. The configuration language is River, a programmable HCL dialect. The component library covers the Grafana stack first (Loki, Mimir, Tempo, Pyroscope) and adds Prometheus remote-write and OpenTelemetry ingestion as compatibility layers.

OpenTelemetry Collector is the vendor-neutral strategic collector from the CNCF OpenTelemetry project. The configuration language is YAML organised into receivers, processors, exporters, and service.pipelines. The component library covers every major observability backend by design.

The two collectors are functionally close. They differ on the following axes:

The strategic choice is not “which is better”. It is “which is the right answer for the ecosystem you have already invested in”.

Why a sysadmin cares

Three failure shapes appear when the choice is treated as a marketing question rather than an operational one:

1. The fleet split by accident. Different teams pick different collectors. The platform ends up running both, with two configuration surfaces, two upgrade cycles, two sets of secrets, two monitoring stacks. The cost of running both is not double the cost of running one; it is more, because the skills and the runbooks do not transfer cleanly.
2. The Promtail stuck in maintenance. A team that does not plan the Promtail migration ends up running an unmaintained collector through a forced upgrade. The migration to Alloy is the path Grafana Labs has documented. The migration to OTel Collector is also possible but requires a translation step from Promtail’s scrape_configs to OTel’s filelog receiver.
3. The configuration language surprise. River is a real programming language; YAML is a structured data format. The difference matters when a config needs to be templated, branched on a per-host value, or composed from several files. River has first-class support for these; YAML needs a templating layer (Helm, Kustomize, or a CI substitution step) outside the collector.

How it works

Both collectors are component graphs. Alloy wires components by naming their receivers; OTel Collector wires components by referencing them from service.pipelines.

Alloy: River

component_a "label" {
  argument = "value"
  forward_to = component_b.label.receiver
}

component_b "label" {
  argument = "value"
}

Each block is a component. forward_to is the wiring. The language supports variables, expressions, and conditional blocks. Two blocks wired together form a graph; many blocks form a pipeline.

OpenTelemetry Collector: YAML

receivers:
  filelog:
    include: [/var/log/app/*.log]

processors:
  batch: {}

exporters:
  loki:
    endpoint: https://loki/loki/api/v1/push

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [loki]

Each section declares a set of components. service.pipelines references them by name. The same components in a different pipeline order form a different graph.

The Promtail migration

Promtail’s clients, scrape_configs, pipeline_stages, and target_config map to Alloy components. The translator exists as the alloy migrate subcommand. A Promtail config that defines:

# promtail config (legacy)
server:
  http_listen_port: 9080

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: https://loki/loki/api/v1/push
    basic_auth:
      username: ingest
      password: secret

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: system
          host: myhost
    pipeline_stages:
      - match:
          selector: '{job="system"}'
          stages:
            - regex:
                expression: '.*(?P<level>INFO|WARN|ERROR).*'
            - labels:
                level:

becomes an Alloy config with loki.source.file, loki.process, and loki.write blocks wired together. The shapes are not identical; the canonical Promtail conventions (job, host, __path__) appear as labels and arguments in the Alloy components. The translator handles ~80% of real-world Promtail configs; the rest need a manual pass.

For an OTel Collector migration, the same Promtail config requires hand-translation. The scrape_configs block becomes a filelog receiver; the pipeline_stages become a chain of processors; the clients become a loki exporter. The translation is mechanical but verbose.

How to configure it

Two minimal real configs that solve the same problem: tail /var/log/app/*.log, parse a level field, ship to Loki.

Alloy config (River)

// /etc/alloy/config.alloy

loki.source.file "app" {
  targets = [{
    __path__ = "/var/log/app/*.log",
    job      = "checkout",
    host     = constants.hostname,
  }]
  forward_to = [loki.process.app.receiver]
}

loki.process "app" {
  stage.regex {
    expression = "^(?P<ts>\\S+) (?P<level>\\S+) (?P<msg>.*)$"
  }

  stage.labels {
    values = { level = "level" }
  }

  forward_to = [loki.write.loki.receiver]
}

loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username     = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
  }
}

OpenTelemetry Collector config (YAML)

# /etc/otelcol/config.yaml

receivers:
  filelog:
    include:
      - /var/log/app/*.log
    operators:
      - type: regex_parser
        regex: '^(?P<ts>\S+) (?P<level>\S+) (?P<msg>.*)$'
      - type: move
        from: body
        to:   attributes
      - type: add
        field: attributes.level
        value: attributes["level"]

processors:
  batch: {}
  resource:
    attributes:
      - key: job
        value: checkout
        action: upsert

exporters:
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    default_labels_enabled: true
    headers:
      X-Scope-OrgID: prod
      Authorization: Basic ${env:LOKI_BASIC_AUTH}

service:
  pipelines:
    logs:
      receivers:  [filelog]
      processors: [resource, batch]
      exporters:  [loki]

The Alloy config is shorter and the wiring is local to each component. The OTel config is more verbose and the wiring lives in the service.pipelines block. Both produce the same labelset on the same lines.

How to validate it

Validation is collector-specific but the steps are the same.

Alloy

# CONFIGURATION: format-check and parse-check.
alloy fmt --check /etc/alloy/config.alloy
alloy validate /etc/alloy/config.alloy

ts=2026-08-13T13:11:04Z level=info msg="config valid"

# SERVICE-IMPACT: reload the running process.
systemctl reload alloy

# READ-ONLY: confirm the new components are live.
curl -s http://localhost:12345/metrics | grep loki_source_file_targets

loki_source_file_targets{job="checkout",path="/var/log/app/app.log"} 1

OpenTelemetry Collector

# CONFIGURATION: parse-check.
otelcol validate --config=/etc/otelcol/config.yaml

# (no output on success; non-zero exit on error)

# SERVICE-IMPACT: reload via SIGHUP.
kill -HUP $(pidof otelcol)

# READ-ONLY: confirm the components are loaded.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_log_records{receiver="filelog"} 1872

# READ-ONLY: confirm the export path is healthy.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_log_records{exporter="loki"} 1872

The numbers match (modulo batching). If receiver_accepted climbs but exporter_sent does not, the failure is in the pipeline. If exporter_sent climbs but the lines do not appear in Loki, the failure is downstream of the collector.

How it can fail

Five failure modes common to both, with the collector-specific symptom called out.

1. The mis-typed receiver name. An Alloy config references loki.source.file.app.receiver but the actual label is apps. Alloy refuses to validate. Symptom: alloy validate returns a non-zero exit with component "loki.source.file.app" not found. The OTel equivalent is service.pipelines.logs.receivers: ["filelog"] referencing a receiver that was renamed; the collector refuses to start.
2. The exporter without headers for the tenant. The Loki exporter ships to the default tenant instead of the production tenant. Symptom: lines appear in Loki, but in the wrong tenant. The query {collector="alloy"} returns nothing; the query for the default tenant returns everything.
3. The pipeline order mistake. A batch processor that drops attributes because they were already moved by an earlier processor. Symptom: labels that should have been present are absent; loki_distributor_ingester_append_failures_total shows rejection for missing labels.
4. The Promtail migration that lost __path__. The translator produced an Alloy config but dropped the __path__ argument. Symptom: loki_source_file_targets shows zero; no files are being tailed.
5. The River expression that returns the wrong type. A coalesce([labels.env, "unknown"]) returns the string "unknown" but a downstream component expected a label, not a value. Symptom: alloy validate succeeds but the agent logs a type mismatch on first received batch.

How to troubleshoot it

When the collector refuses to start, the order matters.

1. Read the error. Both collectors print a parse error with a line number. The first error is usually the only one.
2. Check the wiring. In Alloy, every forward_to must reference a component that exists. In OTel, every entry in service.pipelines must reference a declared component.
3. Validate against the schema. Both collectors have a validate subcommand. Run it on the file before reloading.
4. Tail the agent log on first reload. The first batch after a reload will fail loudly if anything is misconfigured. Watch for parse errors, label collisions, and rejected entries.
5. Confirm the export path is healthy. A configuration that parses cleanly but ships to a dead endpoint looks identical to a configuration that ships to a healthy one until the metrics are inspected.

Security implications

Both collectors expose the same surfaces; the names differ.

• Health and metrics endpoints. Alloy defaults to localhost:12345 for /metrics and /debug. OTel Collector defaults to localhost:8888 for /metrics and localhost:13133 for the health check. Both bind to localhost by default. Expose them on the cluster network only with authentication.
• Secrets in config. River supports env("...") and file("...") lookups for sensitive values. OTel Collector supports ${env:VAR} and ${file:/path}. Both should be used for passwords; neither should appear in the YAML or River file itself.
• CA bundles. The collector must trust the Loki certificate. Mount the CA bundle from the host’s trust store or from a ConfigMap, and reference it by path. A stale bundle is the most common cause of silent shipping failure.

Performance implications

Both collectors have the same hot paths. The on-host cost is ~100-150 MiB RAM and ~50-100 millicores CPU at modest line rates (thousands of lines per second). The differences appear at scale:

• Alloy uses River’s expression evaluation for every batch. Complex expressions in loki.process blocks add CPU cost. Keep stage.regex and stage.json simple; move heavy work to a transform or to Loki’s query-time processing.
• OTel Collector uses the batch processor to coalesce. Batch size and timeout are the dominant knobs. A 1 MiB batch with a 5s timeout is a reasonable starting point; tune from there.

Disk-buffer behaviour is comparable in both; the on-disk format differs but the contract is the same.

Production guidance

• Pick one collector for the platform. Mixed fleets are expensive to maintain. If the existing fleet is Promtail, Alloy is the migration path. If the existing fleet is OTel instrumentation in the application, the OTel Collector is the consistent answer.
• Run the translator on a copy, not on the live config. alloy migrate rewrites the file in place if you let it. Take a backup first, run the translator, diff the result, and commit through review.
• Pin the collector version in your platform manifest. Both Alloy and OTel Collector release monthly. Read the release notes. Breaking changes to config schemas do happen; the migration is usually a one-line edit but the breaking change is the signal to upgrade, not the work itself.
• Smoke test after every config change. Ship a known line with a unique marker and confirm it arrives in the right tenant within ten seconds. The marker should be a UUID, not a timestamp, so a duplicate does not pass for a fresh delivery.

Verification

You should now be able to answer:

• What are the three axes on which Alloy and OTel Collector meaningfully differ, and which one dominates the platform decision?
• How does a Promtail scrape_configs block translate to an Alloy config and to an OTel Collector config?
• What is the first symptom of a missing X-Scope-OrgID header on the Loki exporter?
• Why is “running both” the most expensive answer?

Quiz