Pipeline Resilience

The on-call phone rings at 02:00. The application team says logs for the last forty minutes are missing. The on-host agent is running. Loki is running. The network is fine. The investigation finds a five-second network blip that the agent’s retry handled, and a longer forty-minute partition that the agent’s buffer filled during and then drained to /dev/null because the buffer size was set to one minute’s worth of peak traffic.

This lesson is the engineering that turns that forty-minute gap into a forty-minute delay.

What it is

Pipeline resilience is the discipline of keeping a logging pipeline correct under three kinds of stress: bursts of input that exceed the steady-state capacity, transient network failures between the collector and the central store, and sustained outages that last longer than the in-memory queue.

The three forces act on a queue that sits between the on-host agent and the central store. The queue is configured to one of three regimes:

1. In-memory only. Fast, low-cost, lost on process restart. The right answer for development; the wrong answer for production.
2. Bounded disk buffer with drop-oldest. Durable across restarts; bounded disk usage; the failure shape is “logs lost when the buffer overflows”. The right answer for most production workloads.
3. Bounded disk buffer with stall. Durable across restarts; bounded disk usage; the failure shape is “pipeline stalls when the buffer fills”. The right answer when dropping logs is unacceptable but a stalled application is.

The choice between regimes is the operational decision. The configuration of each regime is the engineering.

Why a sysadmin cares

Three failure shapes appear when the buffer is treated as default-sized rather than sized to the workload.

1. The bounded buffer that is too small. The default disk buffer in many agents is sized for steady-state line rate, not for the worst hour of the worst day. When an outage lasts longer than the buffer can absorb, the buffer drops the oldest entries. The agent continues shipping. Loki receives every line written after the cutover. The lines before the cutover are gone.
2. The unbounded buffer that fills the disk. A buffer configured without a max size grows until the disk is full. The host’s other services start failing because the log volume has consumed their disk space. The agent does not alert because the buffer is “still accepting”.
3. The in-memory queue that dies on restart. A pipeline that restarts during an outage loses the in-memory queue. For a systemd-managed agent that restarts every config reload, that is a reload-frequency failure window. For an agent that crashes and restarts, the gap is the duration of the crash plus the time to replay the position store.

The discipline is to choose the regime deliberately, size the buffer for the worst hour, and instrument the buffer so that overflow is loud before it is silent.

How it works

The queue sits between two stages. The producer (source or process) writes into it; the consumer (write or exporter) reads from it. Backpressure is the consumer telling the producer to slow down. The producer’s response is the regime.

  loki.source.file         loki.process
         |                       |
         v                       v
       +-------------------------+
       |  in-memory queue        |    small, fast, lost on crash
       |  ~10 MiB by default     |
       +-------------------------+
                 |
                 v
       +-------------------------+
       |  on-disk buffer         |    durable across crashes
       |  size: configured       |    bounded by max_size
       |  policy: drop-oldest    |
       +-------------------------+
                 |
                 v
          loki.write (HTTP)
                 |
                 v
          Loki distributor

The on-disk buffer is a write-ahead log. The producer appends batches; the consumer reads them in order. Each batch carries the labels and the line body. When the buffer is full, the policy kicks in: drop the oldest batch and continue, or refuse new writes and stall the producer.

Backpressure propagation

When the buffer fills, the agent has two options:

• Drop the oldest entries. The buffer stays bounded; the pipeline keeps running; the oldest data is lost. The agent increments loki_write_dropped_entries_total with the reason buffer_full.
• Stall the producer. The buffer stays bounded; the pipeline stalls; no data is lost but the application on the host may experience backpressure if the source is a push protocol. The agent increments loki_source_files_failed_total with the reason queue_full.

The drop-oldest regime is the right answer when the source is a tail of an application log file (the application cannot back-pressure into its past). The stall regime is the right answer when the source is a push protocol that the application controls (the application can be told to slow down).

How to configure it

A production-ready Alloy loki.write block with a disk buffer that survives a Loki outage:

// /etc/alloy/config.alloy

// Queue and buffer configuration lives on the loki.write
// component. Other sources forward into the same queue via
// the receiver named here.
loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username     = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }

    // Retry knobs. 429 is rate-limited from Loki; honour it.
    retry_on_http_429 = true

    // Per-request timeout for a single push.
    remote_timeout = "30s"

    // Backoff between retries.
    min_backoff_period = "1s"
    max_backoff_period = "5m"
    max_backoff_retries = 12
  }

  // The on-disk buffer lives between the source stage and the
  // HTTP push. Without this block the buffer is in-memory.
  //
  // size: the maximum on-disk size before the policy kicks in.
  // The right value is "one hour of peak traffic".
  // A 50 MiB/s peak line rate for one hour is 180 GiB.
  // For most fleets a 10-20 GiB buffer is sufficient.
  //
  // type: "disk" makes the buffer survive a process restart.
  //
  // When the buffer is full, drop the oldest entries.
  // The alternative is to stall the producer; that is the
  // right choice for push sources, the wrong choice for tail
  // sources (which cannot back-pressure).
  //
  // batching: coalesce into 1 MiB batches with a 1s timeout.
  // The default is fine for most workloads; tune from metrics.
  //
  // max_backoff_perod is intentionally separate from the
  // endpoint block. The buffer handles queue-full; the
  // endpoint handles HTTP 429 and 5xx.
  //
  // The path is mounted from the host; the agent must have
  // write access and the disk must have the size budget.
  //
  // Note: this is the modern Alloy "queue" stanza syntax.
  // Validate the syntax on the running version; field names
  // have changed across releases.
}

A real loki.write block that uses the disk buffer via the batch and queue components:

// The source forwards into a queue and a batcher before
// hitting the disk buffer.
loki.process "app" {
  forward_to = [loki.write.loki.receiver]
}

// loki.write accepts the queue and batcher options directly.
loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username     = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
    retry_on_http_429 = true
    min_backoff_period = "1s"
    max_backoff_period = "5m"
    max_backoff_retries = 12
  }

  // The disk buffer survives a process restart.
  //
  // path must exist and be writable by the agent user.
  // max_size is in bytes. The right size is the worst hour of
  // peak traffic, plus a margin for the next hour.
}

The OpenTelemetry Collector equivalent uses sending_queue on the exporter and file_storage extension for the on-disk buffer.

How to validate it

Validation is empirical. Run a fault, observe the buffer behave, confirm the metrics.

# CONFIGURATION: validate the config.
alloy validate /etc/alloy/config.alloy

ts=2026-08-13T15:11:09Z level=info msg="config valid"

# READ-ONLY: confirm the disk buffer path exists and is writable.
ls -ld /var/lib/alloy/data

drwxr-xr-x 4 alloy alloy 4096 Aug 13 12:00 /var/lib/alloy/data

# CONFIGURATION: simulate a Loki outage by blocking the
# network port. Use iptables or a temporary firewall rule.
iptables -A OUTPUT -p tcp --dport 443 -d loki.internal.example.com -j DROP

# READ-ONLY: watch the buffer fill while the network is blocked.
watch -n 5 'du -sh /var/lib/alloy/data'

12M    /var/lib/alloy/data
380M   /var/lib/alloy/data
2.1G   /var/lib/alloy/data

# READ-ONLY: confirm the agent metrics show entries buffered.
curl -s http://localhost:12345/metrics | grep -E "loki_write_(sent|dropped)"

loki_write_sent_entries_total 0
loki_write_dropped_entries_total 0

# CONFIGURATION: restore the network.
iptables -D OUTPUT -p tcp --dport 443 -d loki.internal.example.com -j DROP

# READ-ONLY: confirm the buffer drains and entries land in Loki.
sleep 30
curl -s http://localhost:12345/metrics | grep -E "loki_write_sent"

loki_write_sent_entries_total 1834521

# READ-ONLY: confirm the lines arrived in Loki.
logcli query --since=5m \
  '{collector="alloy"}' --tail \
  --addr=https://loki.internal.example.com | head -5

The buffer filled during the simulated outage, held the entries, and drained them when the network returned. The metrics tell the story.

How it can fail

Five failure modes specific to pipeline resilience.

1. The buffer that is too small. A 5 GiB buffer is not enough for an hour of 50 MiB/s peak traffic. The buffer fills in 100 seconds and starts dropping entries. Symptom: loki_write_dropped_entries_total rises during the outage, falls back to zero when the buffer drains.
2. The disk that fills. The buffer is unbounded; the on-host disk is 100 GiB; the buffer consumes all of it. Symptom: loki_write_dropped_entries_total does not rise (the buffer is not full), but the host’s other services start failing with ENOSPC on writes.
3. The buffer path that is not mounted. The agent references /var/lib/alloy/data but the directory does not exist or is read-only. Symptom: the agent refuses to start or falls back to the in-memory queue, with a clear error in the logs.
4. The retry that hides the outage. A network blip causes the agent to retry indefinitely. Loki is down for an hour; the agent retries for the entire hour with exponential backoff. Symptom: high CPU on the agent, no entries shipped, loki_write_retries_total rising in lockstep with loki_write_sent_entries_total falling to zero.
5. The bounded buffer that drops the wrong end. A misconfigured policy drops new entries instead of old. Symptom: the agent’s metrics show a healthy drop counter; the application’s recent logs are missing.

How to troubleshoot it

When the agent is shipping nothing to Loki but the buffer is not dropping, the order matters.

1. Is the agent running? systemctl status alloy.
2. Is the buffer path writable? ls -ld /var/lib/alloy/data && touch /var/lib/alloy/data/test && rm /var/lib/alloy/data/test.
3. Is the queue full or stalled? curl -s http://localhost:12345/metrics | grep -E "queue_full|buffer_full".
4. Is the network to Loki reachable? curl -v https://loki/loki/api/v1/push -u ingest:$PASS.
5. Are the retries accumulating? curl -s http://localhost:12345/metrics | grep loki_write_retries.
6. Is Loki healthy? curl -s https://loki/ready.

The diagnosis order matters because each metric tells a different story. The buffer metrics tell you whether the failure is local; the network metrics tell you whether the failure is on the wire; the Loki ready probe tells you whether the failure is at the destination.

Security implications

The on-disk buffer is a security surface.

• Buffer contents are at rest. Lines on the disk buffer may contain credentials, PII, or tokens. The buffer path must have the same access controls as the application log files. Mount it on an encrypted volume if the host’s disk is not already encrypted.
• Buffer rotation is a deletion event. When the buffer drops the oldest segment, the data is unlinked. The actual bytes on disk may persist until overwritten. Use srm or encrypted volumes to ensure the deletion is effective.
• Buffer paths leak information. A path under /var/lib/alloy/data/buffer/ reveals that the host is shipping logs. The path itself is not sensitive, but the contents are.

The right discipline is to treat the buffer path as a production data store. The same controls apply: encryption, access controls, deletion hygiene.

Performance implications

The buffer is the highest-throughput workload on most hosts. Three knobs dominate:

• Buffer size. Bigger means more outage tolerance and more disk usage. The right answer is the worst hour of peak traffic plus a margin.
• Batch size and timeout. Bigger batches mean higher throughput and lower CPU cost per byte. Smaller batches mean lower latency. The default is 1 MiB per 1s; tune from the loki_write_sent_batch_bytes histogram.
• Disk I/O. The buffer writes at the line rate; the consumer reads at the line rate minus what the network accepts. SSD-backed host disk is appropriate; spinning disks become the bottleneck at sustained 10+ MiB/s.

CPU is rarely the bottleneck. RAM scales with the in-memory queue depth (small) plus the agent’s other state. The disk buffer is the limit, not RAM.

Production guidance

• Size the buffer for the worst hour. A rule of thumb: peak line rate in MiB/s multiplied by 3600 seconds, with a 25% margin. For a 10 MiB/s peak line rate, that is 45 GiB.
• Alert on the drop counter. A rise in loki_write_dropped_entries_total is always wrong. Page on it.
• Test the buffer with a fault injection. Block the Loki port for ten minutes. Confirm the buffer fills, the agent does not crash, and the entries land in Loki after the network returns. Do this in staging before relying on it in production.
• Document the rollback. The buffer configuration is part of the agent config. The rollback is the previous config; the verification is the same fault injection running against the old config.

Verification

You should now be able to answer:

• What are the three buffer regimes, and how does the choice between them depend on the source type?
• How do you size a disk buffer for a known peak line rate?
• What is the difference between the symptoms of buffer overflow and network stall?
• Why does an unbounded disk buffer cause a different failure shape than a bounded one?

Quiz