Grafana Alloy Overview

A 03:00 alert fires: loki_distributor_ingester_append_failures_total is climbing on the production tenant. The on-call engineer opens Grafana. The dashboards show nothing - they only render metrics from Prometheus, and the metrics from the on-host collectors stopped five hours ago when the platform team rolled out a config change to the old Grafana Agent fleet. The fix is to ship a known line through the new fleet and confirm it arrives at the right tenant. The faster fix is to know that the fleet should have been Grafana Alloy in the first place.

This lesson is the introduction to Alloy: what it is, why Grafana Labs built it, how it represents a pipeline, and what changes when the fleet moves off Promtail and Grafana Agent.

What it is

Grafana Alloy is the strategic telemetry collector published by Grafana Labs. It is the single binary that supersedes both Grafana Agent (the metrics-and-traces collector that predated it) and Promtail (the dedicated log shipping agent). The configuration language is River, a programmable HCL dialect. The component library covers the Grafana stack first - Loki, Mimir, Tempo, Pyroscope - and adds Prometheus remote-write and OpenTelemetry ingestion as compatibility layers.

The position in the stack is the same one Promtail and Grafana Agent occupied. Alloy runs on every host, tails the local log files, scrapes the local metrics endpoints, accepts OTLP from the local applications, and ships everything to the cluster’s backends:

   +----------------------------+
   |  Host: app-007.example.com |
   +----------------------------+
   |  app service     --OTLP----+
   |  nginx           --logs----+
   |  node_exporter   --scrape--+---> Alloy ---> Loki   (HTTP push)
   |  journald        --logs----+         ---> Mimir  (remote write)
   |  kernel ringbuf  --logs----+         ---> Tempo  (OTLP)
   +----------------------------+

The strategic change from Promtail is the configuration language. River is a real programming language with variables, expressions, conditional blocks, and first-class imports. A Promtail config is a YAML document; an Alloy config is a graph of components that can be parameterised, composed, and imported as a module.

Why a sysadmin cares

Three operational pains are specific to running a collector fleet without a programmable pipeline.

1. The fleet that grew by copy-paste. Promtail configurations diverge over time as each operator copies an old version, edits it locally, and forgets to push the change back to the source. The result is a fleet of subtly different configs that all need to ship to the same Loki tenant. A Promtail fleet is not parameterised; a config that should be templated by hostname or by environment must be templated externally (Helm, Kustomize, or a CI substitution), which means a config that needs to be different from the template must be a fork.
2. The agent that could not reach Loki. Grafana Agent shipped metrics and traces but not logs; Promtail shipped logs but not metrics. A host that needed both ran two collectors. Two collectors means two config surfaces, two upgrade cycles, two sets of secrets, and two monitoring stacks.
3. The Promtail stuck in maintenance. Promtail reached end of life as a separate project. New features (the structured metadata pipeline, the new Loki ingestion format) shipped to Alloy first. A team that stayed on Promtail got fewer features and a faster support cliff.

Alloy addresses each of these. The fleet is one binary, one config, one upgrade cycle. The config is parameterised by the language. The migration from Promtail is supported by the alloy migrate subcommand.

How it works

Alloy is a single Go binary that embeds the Grafana Agent Flow runtime. The runtime loads a River configuration, evaluates it as a graph, and starts the components. Each component is a Go module that exposes a typed set of arguments and a set of receivers. The wiring is the list of receivers a component forwards its output to.

The component graph

A River configuration is a list of blocks. Each block declares one component, its arguments, and the receivers it forwards to:

component.kind "label" {
  argument = "value"
  forward_to = [component.kind.other_label.receiver]
}

The label is the instance name within the kind. Two blocks of the same kind can coexist with different labels. A forward_to is a list of receivers on other components. The runtime wires the output of each component to the inputs declared by the receivers in the forward_to list.

The shape of a real logs pipeline:

loki.source.file "app"          loki.process "app"
        |                                 |
        +--- forward_to ---> receiver --->+
                                          |
                                    loki.write "loki"
                                          |
                                          +---> Loki push API

A pipeline is a chain. The components are connected by their receivers. The output of one component flows to every receiver in its forward_to list.

The module library

Alloy’s published module library lives at github.com/grafana/alloy-modules. A module is a River file that declares a set of named arguments and returns a list of components. Other configs import the module and pass the arguments in. The shape resembles a function call:

import.git "log_pipeline" {
  repository = "https://github.com/grafana/alloy-modules.git"
  revision   = "main"
  path       = "modules/log-pipeline/log-pipeline.alloy"
}

log_pipeline "default" {
  forward_to = [loki.write.central.receiver]
  arguments {
    paths    = ["/var/log/app/*.log"]
    job_name = "checkout"
  }
}

The team that owns the platform ships a module; the per-host config imports the module and supplies host-specific arguments. The result is the copy-paste fleet, parameterised.

How to configure it

The minimum viable Alloy configuration for a host that ships application logs to Loki and scrapes node_exporter for metrics, with both streams forwarded to their respective backends:

// /etc/alloy/config.alloy

logging {
  level  = "info"
  format = "logfmt"
}

loki.source.file "system" {
  targets = [{
    __path__ = "/var/log/syslog",
    job      = "system",
    host     = constants.hostname,
  }]
  forward_to = [loki.process.system.receiver]
}

loki.process "system" {
  stage.regex {
    expression = "^(?P<ts>\\S+) (?P<host>\\S+) (?P<process>[^\\[]+)\\[(?P<pid>\\d+)\\]: (?P<msg>.*)$"
  }
  stage.labels {
    values = {
      process = "process",
      host    = "host",
    }
  }
  forward_to = [loki.write.central.receiver]
}

loki.write "central" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username      = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
  }
}

prometheus.scrape "node" {
  targets = [{
    job     = "node",
    address = "localhost:9100",
  }]
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "https://mimir.internal.example.com/api/v1/push"
    basic_auth {
      username      = "ingest"
      password_file = "/etc/alloy/secrets/mimir-pass"
    }
  }
}

The arguments to each block are typed and validated at load time. A block that expects an integer and receives a string fails the parse step; the runtime refuses to start. The constants.hostname expression resolves at load time to the hostname of the host; the equivalent Promtail convention stamps the hostname into a host label via a build-time substitution.

How to validate it

Validation has three stages: format, syntax, and runtime health.

# CONFIGURATION: format-check. The file is left unchanged on
# success; the formatter rewrites it in place if --check is
# omitted and the file is unformatted.
alloy fmt --check /etc/alloy/config.alloy

# (no output on success; non-zero exit if the file is unformatted)

# CONFIGURATION: parse-check against the component schema.
alloy validate /etc/alloy/config.alloy

ts=2026-08-13T13:11:04Z level=info msg="config valid"

# SERVICE-IMPACT: hot-reload the running Alloy process. The
# runtime diffs the new graph against the old and only touches
# the changed components.
systemctl reload alloy

# READ-ONLY: confirm the components are live and producing
# telemetry. The component id is suffixed with the block label.
curl -s http://localhost:12345/metrics | grep loki_source_file

loki_source_file_files_count{component="loki.source.file.system"} 12
loki_source_file_target_lines_total{component="loki.source.file.system",path="/var/log/syslog"} 1872

If the counters advance as expected, the pipeline is live. If they do not, see “How to troubleshoot it” below.

How it can fail

Five failure modes specific to Alloy.

1. The label-collision error. Two loki.write blocks declared with the same label. Symptom: alloy validate exits with component "loki.write.central" already exists. The runtime refuses to load the graph; no traffic flows at all.
2. The forward_to reference that points nowhere. A loki.process block forwards to loki.write.central.receiver but the only loki.write block is labelled loki. Symptom: alloy validate exits with component "loki.process.system.forward_to" references unknown component "loki.write.central". The runtime refuses to start.
3. The mis-typed argument. A targets block inside loki.source.file references host_name instead of host. Symptom: alloy validate exits with unknown argument "host_name"; the file fails to load. The component-level error tells the operator exactly which argument is wrong.
4. The hot-reload that doubled the source. A label rename on loki.source.file from system to sys during a hot reload. Symptom: loki_source_file_target_lines_total shows two counters advancing in parallel for several minutes, and Loki receives duplicated log streams. The old component drains on shutdown; the new one starts in parallel.
5. The River expression that returns the wrong type. A coalesce([labels.env, "unknown"]) that returns the string "unknown" to a downstream component that expected a label map. Symptom: alloy validate succeeds, but the runtime logs a type mismatch on the first received batch and the affected forward_to chain drops its output.

How to troubleshoot it

When the Alloy process is up but no telemetry is arriving, the order matters.

1. Read the agent log. journalctl -u alloy -n 200 shows the last reload. The first error is the one to fix; subsequent errors are usually its consequences.
2. Check the readiness endpoint. curl -s http://localhost:12345/-/ready returns 200 only when every component reports ready. A 503 with a body listing one component is a specific component failure, not a process failure.
3. Inspect the pipeline counters. curl -s http://localhost:12345/metrics | grep _target_lines shows the rate at which each source is producing. A counter that is not advancing means the source has stopped tailing.
4. Confirm the export path is healthy. A configuration that parses cleanly but ships to a dead endpoint looks identical to a working configuration until the metrics are inspected. Compare loki_write_sent_entries_total to loki_source_file_target_lines_total; the second should be less than or equal to the first, modulo the in-flight batch.
5. Smoke test the export. Ship a known marker line with a unique UUID and confirm it arrives in the right Loki tenant within ten seconds.

Security implications

Alloy exposes a small set of network surfaces that must be locked down in production.

• The debug and metrics endpoints. Defaults are localhost:12345 for /metrics and /debug. Both bind to localhost. Expose them on the cluster network only with authentication.
• Secrets in the config. River supports env("...") and file("...") lookups for sensitive values. Use them for passwords, never the literal.
• CA bundles. The collector must trust the Loki and Mimir certificates. Mount the CA bundle from the host’s trust store or from a ConfigMap, and reference it by path. A stale bundle is the most common cause of silent shipping failure.

Performance implications

The on-host cost is roughly 100-150 MiB RAM and 50-100 millicores CPU at modest line rates (thousands of lines per second per host). The differences from Promtail appear at scale.

• River expression evaluation. Every batch is evaluated by the River expression engine. Complex expressions in loki.process blocks add CPU cost. Keep stage.regex and stage.json simple; move heavy work to Loki’s query-time processing.
• Position tracking. Alloy persists file positions to disk on a schedule. A crash between syncs loses up to one sync interval of positions; on restart, the affected files are re-read from the last persisted offset. Tune loki.source.file positions_sync_interval to balance crash-window size against disk churn.
• Disk buffer. Alloy buffers to disk when the destination is slow or unreachable. The buffer is bounded by loki.write buffer_config and is the safety net for downstream outages.

Production guidance

• Pin the Alloy version in your platform manifest. Alloy releases monthly. Read the release notes. Breaking changes to config schemas do happen; the migration is usually a one-line edit but the breaking change is the signal to upgrade, not the work itself.
• Import the module library. The github.com/grafana/alloy-modules repository is the canonical source for log, metrics, and traces pipelines. Pin the revision in production; a module change should be a deliberate commit.
• Run alloy validate before every reload. A parse error caught before the reload is a non-event. The same error caught by the runtime at 03:00 is a page.
• Back up the running config and the positions file. Alloy stores its loaded config in /var/lib/alloy/; the on-disk position file is critical to crash recovery. Back up before any reload that touches a source block.

Verification

You should now be able to answer:

• What three predecessors does Alloy consolidate, and what does each contribute?
• How does a River component graph differ from a YAML pipeline declaration?
• Why does the module library matter for a fleet with more than ten hosts?
• What is the first thing to check when Alloy is up but no telemetry is arriving in Loki?

Quiz