Alloy Configuration

A team rolls a new Loki parsing pipeline. Two of five services ship their log lines with a level label; three do not. The investigation finds that the three services are running an older config that never had a stage.regex for the level field, so the label is missing. The dashboards that filter by level=error return nothing for those services. The team needs a config that parses the field for every service and stamps the label in the right place in the pipeline order.

This lesson is the configuration grammar that makes that fix possible: River blocks, the forward_to wiring, and the stage order inside loki.process.

What it is

River is the configuration language Alloy uses. It is an HCL dialect with three additions over plain HCL: typed components, a receiver-based wiring model, and expressions evaluated against a typed value model. A River file is a list of named blocks. Each block declares one component instance. Components are connected by listing their receivers in a forward_to argument.

The minimum building blocks for a log pipeline are three components.

• loki.source.file - tails one or more files from the host filesystem, applies static labels, and emits log entries.
• loki.process - applies a chain of stages to each entry.
  • stages parse, mutate, filter, and label the entry.
• loki.write - serialises the entry and pushes it to a Loki endpoint.

A pipeline is a chain of these three component kinds. The output of one component is wired into the next via forward_to.

Why a sysadmin cares

The operational cost of a misconfigured pipeline is not visible until it is. Three failure shapes appear when the wiring is treated as an implementation detail rather than a config surface.

1. The pipeline that drops labels. A stage.labels runs before the stage.regex that produces the label value. The labels block has nothing to label. Symptom: the labels are present but empty; Loki receives level="" on every entry; downstream filters return nothing.
2. The pipeline that fans out to two writers. Two loki.write blocks share a loki.process receiver. One writer is healthy, one is misconfigured. Symptom: every entry is sent twice, and the failing writer’s exporter metrics show send failures.
3. The pipeline that is hot-reload invisible. A regex change is rolled out without a systemctl reload. Symptom: the new regex is on disk but the running process is still using the old one; the on-call engineer spends an hour wondering why the change has no effect.

How it works

Block syntax

component.kind "label" {
  argument_one = "value"
  argument_two = 42
  forward_to  = [component.kind.other_label.receiver]
}

Three rules govern the syntax.

1. Kind. The kind is the component type. loki.source.file, loki.process, loki.write. The kind is fixed; the label is the operator’s choice.
2. Label. The label is the instance name within the kind. Two blocks of the same kind with different labels coexist happily. Two blocks of the same kind with the same label collide at parse time.
3. forward_to. The list of receivers that receive the component’s output. Every receiver must exist on a declared component. A forward_to that points to a non-existent component fails alloy validate.

The forward_to wiring

The wiring is a list of references of the form component.kind.label.receiver. A receiver is an exported input on the component. Most components export a single receiver called receiver. Some expose additional receivers for branching.

loki.source.file "app" {
  targets    = [{ __path__ = "/var/log/app/*.log", job = "checkout" }]
  forward_to = [loki.process.app.receiver]
}

loki.process "app" {
  stage.json     { expressions = { level = "level" } }
  stage.labels   { values = { level = "" } }
  forward_to     = [loki.write.central.receiver]
}

loki.write "central" {
  endpoint { url = "https://loki/loki/api/v1/push" }
}

Three components, one pipeline. The runtime wires them in evaluation order.

Variables and expressions

River is a real language. A block can declare local variables; a top-level block can reference constants.hostname, env("VAR"), and file("/path"). The values are typed.

loki.source.file "system" {
  targets = [{
    __path__ = "/var/log/syslog",
    job      = "system",
    host     = constants.hostname,
    env      = env("DEPLOY_ENV"),
  }]
  forward_to = [loki.process.system.receiver]
}

constants.hostname resolves at load time. env("DEPLOY_ENV") resolves at load time from the process environment. The values must be strings; a number passed where a string is expected fails the type check.

How to configure it

A complete annotated logs pipeline for an application that writes JSON lines to /var/log/app/*.log.

// /etc/alloy/config.alloy

logging {
  level  = "info"
  format = "logfmt"
}

// 1. Source: tail the application log files.
loki.source.file "app" {
  targets = [{
    __path__ = "/var/log/app/*.log",
    job      = "checkout",
    host     = constants.hostname,
    env      = env("DEPLOY_ENV"),
  }]
  forward_to = [loki.process.app.receiver]
}

// 2. Process: parse JSON, label level, drop noise.
loki.process "app" {
  // 2a. Parse the line as JSON. The fields become entries in the
  //     value map; the original `entry` field becomes a string.
  stage.json {
    expressions = {
      ts      = "ts",
      level   = "level",
      msg     = "msg",
      traceid = "trace_id",
    }
  }

  // 2b. Promote `level` and `traceid` to labels. Labels are indexed
  //     by Loki; values are not. Choose the smallest set of fields
  //     that the dashboards filter by.
  stage.labels {
    values = {
      level   = "",
      traceid = "",
    }
  }

  // 2c. Drop entries below info. The filter runs against the
  //     entry map; it sees the parsed fields, not the labels.
  stage.match {
    selector = "{job=\"checkout\"}"
    stage.drop {
      expression  = ".*"
      matcher     = "level"
      drop_counter_reason = "below_info"
      older_than = "1h"
    }
  }

  forward_to = [loki.write.central.receiver]
}

// 3. Write: push to Loki over HTTPS.
loki.write "central" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username      = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
  }
}

Stage-by-stage

The four stages used above each have a specific job.

• stage.json parses a JSON line and projects fields into the entry map. If the line is not valid JSON, the entry is passed through with the raw line preserved. The expressions block names the fields to project.
• stage.labels promotes values from the entry map to labels. The value on the right-hand side is the entry-map key; the value on the left-hand side is the label name. An empty string on the right-hand side means “use the same name as the label”.
• stage.match wraps one or more stages in a conditional filter. The selector matches against the current labelset. The nested stages run only when the selector matches.
• stage.drop removes entries from the stream. The expression matches against the entry-map value; the matcher names the field; older_than lets a buffered entry through for a grace period before drop.

A common mistake is to put stage.labels before stage.json. The labels block has no parsed fields to label, so the labels stay empty. The fix is order: parse first, then label.

Other useful stages

• stage.regex - parse a non-JSON line by named capture groups. The captured groups become entry-map values.
• stage.template - rewrite the entry line using Go templating. Useful for normalising formats.
• stage.output - emit the current entry map to a separate receiver. Useful for splitting a stream into two destinations.
• stage.tenant - set the Loki tenant on a per-entry basis. Only effective when the loki.write block does not have its own tenant_id set.

How to validate it

Validation has three stages: format, syntax, and runtime health.

# CONFIGURATION: format-check. The formatter rewrites the file
# in place if it is unformatted.
alloy fmt --check /etc/alloy/config.alloy

# (no output on success; non-zero exit if the file is unformatted)

# CONFIGURATION: parse-check against the component schema.
alloy validate /etc/alloy/config.alloy

ts=2026-08-13T13:11:04Z level=info msg="config valid"

# SERVICE-IMPACT: hot-reload the running Alloy process.
systemctl reload alloy

# READ-ONLY: confirm the source is tailing files.
curl -s http://localhost:12345/metrics | grep loki_source_file_files_count

loki_source_file_files_count{component="loki.source.file.app"} 8

# READ-ONLY: confirm the writer is shipping.
curl -s http://localhost:12345/metrics | grep loki_write_sent

loki_write_sent_entries_total{component="loki.write.central"} 1872

If loki_write_sent_entries_total advances but lines do not appear in Loki, the failure is downstream of the collector (Loki distributor rejecting, network, tenant mismatch). If it does not advance, the failure is in the pipeline above the writer.

How it can fail

Five failure modes specific to Alloy configuration.

1. The label-collision error. Two loki.write blocks declared with the same label. Symptom: alloy validate exits with component "loki.write.central" already exists. The runtime refuses to load the graph; no traffic flows at all.
2. The forward_to reference that points nowhere. A loki.process block forwards to loki.write.central.receiver but the only loki.write block is labelled loki. Symptom: alloy validate exits with component "loki.process.app.forward_to" references unknown component "loki.write.central". The runtime refuses to start.
3. The stage that runs before the parser. A stage.labels block before stage.json, so the labels block has no parsed field to label. Symptom: the labels are present but empty; Loki receives level="" on every entry. The pipeline runs, the labels are silent failures.
4. The regex that matches nothing. A stage.regex whose expression does not match the actual line format. Symptom: loki_source_file_target_lines_total advances, but the captured groups in the value map are empty, and downstream label promotions produce empty labels. The metric counters lie about success.
5. The hot-reload that left the old component running. A label rename on loki.source.file from app to checkout during a hot reload. Symptom: loki_source_file_target_lines_total shows two counters advancing in parallel for several minutes, and Loki receives duplicated log streams.

How to troubleshoot it

When the pipeline is up but the labels are wrong, the order matters.

1. Read the agent log. journalctl -u alloy -n 200 shows the last reload. The first error is the one to fix; subsequent errors are usually its consequences.
2. Inspect the stage outputs. curl -s http://localhost:12345/debug/pprof is not the right tool; instead, add a loki.write block that writes a copy of the stream to a separate tenant, then query that tenant for the labels. The label state at the writer is the label state that arrived at Loki.
3. Compare counters across stages. loki_source_file_target_lines_total is the input rate; loki_process_dropped_lines_total is the drop rate; loki_write_sent_entries_total is the output rate. The output should equal input minus drop modulo the in-flight batch.
4. Run alloy fmt to surface indentation errors. An unclosed block produces a parse error with a line number; alloy fmt will refuse to reformat a file with a syntax error and will print the location.
5. Smoke test the export. Ship a known marker line with a unique UUID and confirm it arrives in the right Loki tenant with the expected labels within ten seconds.

Security implications

The configuration file holds the wiring; it does not hold the secrets.

• Secrets in the config. River supports env("...") and file("...") lookups for sensitive values. Use them for passwords, never the literal. A literal password in a committed config file is a credential leak waiting to happen.
• CA bundles. The collector must trust the Loki certificate. Mount the CA bundle from the host’s trust store or from a ConfigMap, and reference it by path. A stale bundle is the most common cause of silent shipping failure.
• Filesystem access. A loki.source.file block with a wildcard like /var/log/**/*.log will tail whatever the process can read. The process runs as the alloy user by default; ensure the user has read access to the intended paths and no more.

Performance implications

• Stage order. A stage.regex is more expensive than a stage.labels. Put the cheap stages first; put the regex last. The pipeline runs every entry through every stage; a misordered chain spends CPU on regex when a label match would have rejected the entry earlier.
• Buffering. loki.write buffers entries to disk when the destination is slow or unreachable. The buffer is bounded by buffer_config; a buffer that fills up causes the source to stall, which causes the file position to fall behind.
• Hot reload cost. Reloading a config with a small change restarts only the changed components. Reloading a config with a widespread change (a new file pattern, a new label) restarts the affected sources and re-reads the position file. Tune the position-file sync interval to balance crash recovery against hot-reload cost.

Production guidance

• Pin the component versions. Each import.git block pins a revision. Use a tag or a commit SHA, not a branch name. A main reference will silently change after a module owner commits.
• Run alloy validate in CI. The same parse that the runtime runs at reload should run in the pipeline that commits the config. A parse error caught in CI is a comment; the same error caught at 03:00 is a page.
• Smoke test after every config change. Ship a known line with a unique UUID and confirm it arrives in the right Loki tenant with the expected labels. The smoke test should be automatic; manual smoke tests do not run.
• Back up the position file. /var/lib/alloy/positions.yaml is the file that records which lines have been read. A position file lost on restart means re-reading every file from the start. Back it up alongside the config.

Verification

You should now be able to answer:

• What is the relationship between component.kind.label.receiver and the forward_to argument on the previous block?
• Why must stage.json or stage.regex run before stage.labels in a loki.process chain?
• What does alloy fmt --check catch that alloy validate does not?
• What is the difference in symptom between a label collision error and a forward_to reference error?

Quiz