Promtail in Maintenance Mode

An 800-host fleet ships logs to Loki through Promtail. The configurations were authored in 2022. New rules go in. Old rules stay. Grafana Labs announces that Promtail is in maintenance mode: bug fixes only, no new features. The fleet must move. This lesson is the migration, from the announcement to a validated Alloy rollout.

The lesson does not argue for keeping Promtail. The strategic answer is Alloy. The lesson is the path from here to there, with the position store, the labels, and the secrets preserved along the way.

What it is

Promtail is Grafana Labs’ log-shipping agent written in Go and shipped as a single static binary. It reads log files, parses structured fields, attaches labels, and pushes batches to Loki over HTTP. It was the recommended companion to Loki from 2018 until 2024.

Maintenance mode means Grafana Labs commits to shipping bug fixes and security patches for Promtail but no longer accepts feature requests or adds new components. New deployments should use Grafana Alloy. Existing deployments should plan a migration. The Grafana Alloy project publishes a translator (alloy migrate) that converts a Promtail YAML config to an Alloy River config.

The migration is not optional in any sense; the only question is the schedule. A fleet running Promtail through 2027 will face patching pressure (security CVEs that no longer have feature releases behind them) and a smaller pool of operators familiar with the configuration format.

Why a sysadmin cares

Three failure shapes appear when the Promtail fleet is left in place because “it still works”.

1. The deprecation surprise. A Grafana Labs security advisory requires a Promtail version bump that breaks the pipeline_stages syntax used by the fleet. The team upgrades, the configs fail to parse, and the migration that was “for later” is now for today.
2. The drift between two configs. A second fleet (a new Kubernetes cluster, say) is deployed with Alloy. The old fleet is on Promtail. The two configs diverge over months. The platform ends up maintaining two configuration surfaces for the same job.
3. The position store that nobody migrated. Promtail tracks the byte offset into every tailed file in a YAML file (positions.yaml). When the binary is replaced with Alloy, the offset file is no longer understood by Alloy. The new collector starts from the beginning of every file. A 30-day-rotation file on a busy host ships 30 days of duplicate lines to Loki on cutover.

The fix for all three is a planned migration. The translator handles the configuration. The position store needs a different treatment.

How it works

Promtail and Alloy are structurally similar. Both tail files, parse, label, and push. The configuration shapes differ.

Promtail YAML

server:        listens on a HTTP port for /metrics, /ready
positions:     tracks the byte offset per tailed file
clients:       list of Loki push endpoints
scrape_configs: list of jobs, each with targets and pipeline_stages
target_config:  how to sync the targets
limits:        per-stage resource limits

The scrape_configs block is the core. Each job defines the files to tail, the static labels to apply, and the pipeline stages that transform each line.

Alloy River

loki.source.*:       tails a source (file, kubernetes, journal, syslog)
loki.process.*:      applies stages (regex, json, labels, template)
loki.write.*:        ships the assembled batches to Loki

The River equivalents have the same responsibilities. The differences are the wiring (forward_to instead of a centralised clients: block) and the per-component options (River exposes more granular knobs than Promtail’s stage types).

The translation step

alloy migrate reads a Promtail config and emits an Alloy config. The translation covers:

The translator preserves the __path__ glob, the static labels, and the pipeline stages. It does not preserve the position store; that conversion is a separate step.

How to configure it

A realistic Promtail config and the Alloy config the translator emits.

Source Promtail config

# /etc/promtail/config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: https://loki.internal.example.com/loki/api/v1/push
    basic_auth:
      username: ingest
      password_file: /etc/promtail/secrets/loki-pass
    backoff_config:
      min_period: 500ms
      max_period: 5m
      max_retries: 10

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: system
          host: myhost
          __path__: /var/log/*.log
    pipeline_stages:
      - match:
          selector: '{job="system"}'
          stages:
            - regex:
                expression: '^(?P<ts>\S+) (?P<level>\S+) (?P<msg>.*)$'
            - labels:
                level:
            - timestamp:
                source: ts
                format: "2006-01-02T15:04:05Z"

Translated Alloy config

// /etc/alloy/config.alloy (after alloy migrate)

loki.source.file "system" {
  targets = [{
    __path__ = "/var/log/*.log",
    job      = "system",
    host     = "myhost",
  }]
  forward_to = [loki.process.system.receiver]
}

loki.process "system" {
  stage.regex {
    expression = "^(?P<ts>\\S+) (?P<level>\\S+) (?P<msg>.*)$"
  }

  stage.labels {
    values = { level = "level" }
  }

  stage.timestamp {
    source = "ts"
    format = "2006-01-02T15:04:05Z"
  }

  forward_to = [loki.write.loki.receiver]
}

loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    basic_auth {
      username     = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
  }
}

The shapes are recognisably the same. The static labels moved from a static_configs block to a targets argument. The pipeline stages moved from a list under match to a chain of stage.* blocks. The clients list became a loki.write block.

Validating the translation

# CONFIGURATION: run the translator on a copy of the live config.
cp /etc/promtail/config.yml /tmp/promtail.yml
alloy migrate /tmp/promtail.yml > /tmp/alloy.alloy

# (no output on success; the translated config is on stdout)

# CONFIGURATION: format-check and parse-check the result.
alloy fmt --check /tmp/alloy.alloy
alloy validate /tmp/alloy.alloy

ts=2026-08-13T14:01:42Z level=info msg="config valid"

# READ-ONLY: diff the labels between the original and translated
# config to confirm nothing was dropped.
grep -E "job:|host:" /tmp/promtail.yml
grep -E "job|host" /tmp/alloy.alloy

# promtail
          job: system
          host: myhost
# alloy
    job      = "system",
    host     = "myhost",

The labels match. The pipeline stages match. The position store is the only thing not translated, and that is the next step.

How to validate it

The migration has four validation gates. Pass all four before cutting traffic.

1. Translation is complete. Run alloy migrate; diff the labels and stages against the source. Address every drift in review.
2. Alloy validates. alloy validate returns zero. The format is well-formed and the components wire.
3. Side-by-side run. Run Alloy on a canary host while Promtail still ships. Compare the line count and the labelset in Loki over a 24-hour window. The two streams should match within the position-store offset.
4. Cutover with position conversion. Stop Promtail, convert the position store, start Alloy. Confirm the offsets line up and no lines are duplicated beyond the offset gap.

Converting the position store

The position store is a manual conversion. A script reads the Promtail positions.yaml:

# /var/lib/promtail/positions.yaml
positions:
  - /var/log/syslog
  - /var/log/app/app.log
  - /var/log/auth.log

and emits the equivalent SQLite inserts for Alloy’s store. The Grafana Alloy cookbook documents the schema; a one-time script that maps filename to the SQLite instance table and offset to the byte-offset column is sufficient for a fleet cutover.

A simpler approach for small fleets: ship a known marker line into each tailed file before the cutover. Alloy starts from byte zero and re-ships history; the marker line tells you where the re-ship ends and the live stream begins. The cost is a window of duplicate lines in Loki, which Loki’s dedup can absorb if the labels match.

How it can fail

Five failure modes specific to the Promtail-to-Alloy migration.

1. The translator dropped a stage. alloy migrate covers the canonical stage types. A custom stage defined by a user-defined cri: extension is not translated. Symptom: the translated config validates; the Alloy process starts; the lines arrive in Loki without the custom field.
2. The position store offset is wrong. The position store was converted with the wrong byte offset (off by one, say). Symptom: every line is duplicated once or the first line of every file is dropped on cutover.
3. The secret path did not move. The Promtail config references /etc/promtail/secrets/loki-pass. The Alloy config references /etc/alloy/secrets/loki-pass. The latter does not exist yet. Symptom: Alloy starts but logs permission denied on the secret file.
4. The systemd unit was not migrated. Promtail runs as promtail.service; Alloy runs as alloy.service. The cutover host has both running. Symptom: both processes are shipping lines to Loki, doubling the ingest cost.
5. The receiver name changed. The Promtail target label was instance; the Alloy default is instance but the relabel config rewrites it. The downstream LogQL query that filters on instance returns empty. Symptom: dashboards green but data missing; queries by instance return nothing.

How to troubleshoot it

When the canary host is shipping lines to Loki but the labels are wrong, the order matters.

1. Compare the labelsets. On the Promtail side, query Loki for {job="system"}; on the Alloy side, query the same. The labels should match. If they do not, the translation dropped a stage or a label.
2. Check the positions. On the canary host, inspect the Alloy SQLite store with sqlite3 /var/lib/alloy/data/alloy.db "SELECT * FROM instance;". The paths and offsets should match the Promtail positions.yaml within the conversion tolerance.
3. Tail both logs. Stop Promtail cleanly with kill -TERM; the agent writes its position file before exit. Start Alloy; the agent reads its position and continues. The line rate on Loki should drop to zero between the two events and resume from the offset.
4. Confirm the secret path. ls -l /etc/alloy/secrets/. If the file is missing, the agent logs permission denied on the first batch and falls back to no auth. Loki returns 401.

Security implications

The migration does not change the security model. The same considerations apply to the new collector:

• The new collector runs as a dedicated service account.
• The secret file is 0600, owned by the collector user.
• The TLS configuration matches the previous configuration; the CA bundle is mounted from the host trust store.
• The agent’s /metrics endpoint binds to localhost by default; expose it on the cluster network only with authentication.

The rotation story improves: Alloy supports env("...") and file("...") lookups for the Loki password, which means the secret can be mounted from a CSI volume rather than baked into a ConfigMap.

Performance implications

The performance characteristics of Alloy and Promtail at the same workload are within ~10% of each other. The migration does not require a capacity review.

Two operational differences are worth noting:

• Memory model. Promtail allocates a fixed memory budget per stage. Alloy uses Go’s garbage collector; the resident set grows with the queue depth and shrinks with the GC. Plan for ~120 MiB steady state per Alloy instance on a busy host.
• CPU model. Promtail is mostly CPU-bound on parsing. Alloy has the same profile but adds the River expression evaluator. The cost is real but small; expect ~10-15% more CPU per host at the same workload.

Production guidance

• Run the migration in three waves. Wave 1: a single canary host, side-by-side with Promtail, for 48 hours. Wave 2: a canary cluster, 10% of the fleet, for one week. Wave 3: the remainder, in batches of 100 hosts at a time.
• Validate the position store on the canary. Pick a host with large rotated log files. Confirm the cutover does not re-ship the rotated window.
• Roll back is a process restart. Keep the Promtail binary and config on the host for the duration of the migration. When Alloy misbehaves, systemctl stop alloy && systemctl start promtail restores the previous state. The Promtail position store is intact.
• Document the cutover. The migration touches the systemd unit, the secret file path, the config file format, and the position store. A runbook that lists each touch and the order is the difference between a smooth cutover and a paging event.

Verification

You should now be able to answer:

• Why did Grafana Labs move Promtail to maintenance mode, and what does that commitment mean operationally?
• What does alloy migrate translate, and what does it not translate?
• How do you handle the position store on cutover so that no lines are duplicated and none are dropped?
• What is the rollback path if the canary host ships the wrong labels?

Quiz