The Logging Pipeline

At 03:14 the on-call phone rings. A user reports a failed checkout. The application logs that explain the failure live in a Loki tenant. Grafana shows the dashboard. LogQL returns no rows.

The pipeline is the chain of components that took that line from the application’s stdout and put it somewhere a query can reach it. Every component in the chain is a failure domain. The lesson that follows names the chain, names the failure domain of each link, and walks the topology choices a sysadmin makes when sizing it.

What it is

A logging pipeline is the ordered set of components that turns a byte stream emitted by a process into queryable records in a log store. In a Loki stack the destination is a Loki distributor and the source is whatever process wrote the line. The collector of choice in 2026 is Grafana Alloy or the OpenTelemetry Collector; Promtail remains installed only in fleets mid-migration.

The canonical shape has four stages:

1. Source. A process emitting to stdout or stderr, a container runtime writing JSON to a log file, an application that writes directly to a socket, or a syslog producer. The source owns the format and the rotation policy.
2. Collect / transform. The on-host agent tails the source, parses the format, applies labels, optionally redacts or enriches, and packages the line for transport. In Alloy this is the loki.source.* and loki.process components; in OTel Collector it is the filelog receiver plus the attributes processor.
3. Buffer / transport. A queue between the on-host agent and the central store, on disk for durability, in memory for latency. This is where backpressure shows up first.
4. Central store and query. The Loki distributor validates, shards by hash, and hands the batch to the ingester. The ingester compresses into chunks and writes to the object store. Grafana queries through LogQL on read.

Why a sysadmin cares

Three failure shapes appear when the pipeline is treated as invisible plumbing rather than a service in its own right:

1. The “we have logs” fallacy. The pipeline has been silent for two days because of a misconfigured loki.write endpoint. Every dashboard says “no data”. The team only learns when an incident hits and the investigation stalls on the missing evidence.
2. The buffer overflow that nobody noticed. A spike in application output overwhelmed the on-host buffer. The agent dropped the oldest entries and continued. The pipeline metrics showed loki_source_files_dropped_entries_total rising steadily, but nobody had an alert on it.
3. The topology that did not scale. A 4,000-host fleet running one DaemonSet-isolated sidecar per pod drove the central distributor to its connection cap. Distributor logs filled with rpc error: client connection closed and Loki refused new writes. The fix was a topology change, not a Loki change.

None of these are caught by the application. They are caught by understanding the pipeline as four named stages with known failure modes.

How it works

The mental model. A log line moves left to right, and every arrow is a failure domain with its own metrics:

  +---------+    +-----------+    +---------+    +---------+
  | Source  |--->| Collect / |--->| Buffer /|--->| Central |
  | (stdout,|    | transform |    | ship    |    | store   |
  |  files, |    |  (Alloy,  |    | (disk + |    | (Loki   |
  |  socket)|    |   OTel)   |    |  HTTP)  |    |  ingester)
  +---------+    +-----------+    +---------+    +---------+
       |               |               |              |
   source_bytes   parse_failures   buffer_fill_pct  ingest_rate
   drop?          redacted_count   send_retries     rejected_lines

Each arrow is a place where the line can vanish. Each box emits its own metrics. The sysadmin’s job is to instrument every box and to know which metric corresponds to which failure.

On-host versus central pipeline

There are two topologies in production today:

On-host pipeline (the default). One collector process runs on every host. The host is the boundary. The collector tails local files, transforms, buffers on local disk, and ships to the central store over the network. In Kubernetes this is a DaemonSet. In bare-metal or VM fleets it is a systemd unit.

Central relay pipeline. A small fleet of relay processes ingest from many sources (syslog, fluent-bit forwarders, vendor agents) and re-emit to Loki. Used when the producers cannot run an Alloy or OTel collector themselves — managed appliances, legacy hosts, third-party software that speaks only syslog or Fluent Forward.

The trade-off is operational simplicity against blast radius:

  On-host                              Central relay
  --------                             -------------
  +------+    +------+   +------+       +------+    +------+
  | host |--->| Alloy|---| Loki|       | src1 |--->|      |
  +------+    +------+   +------+       +------+    | relay|---> Loki
  +------+    +------+   +------+       +------+    |      |
  | host |--->| Alloy|---| Loki|       | src2 |--->|      |
  +------+    +------+   +------+       +------+    +------+
  blast radius: a single host          blast radius: the relay
  cost per host: 80-200 MiB RAM        cost per host: trivial
  config drift: per-host               config drift: centralised

The on-host topology has the smaller blast radius and the larger configuration surface. The relay topology has the inverse. Most production observability stacks converge on “on-host for everything we own, central relay for everything we do not”.

How to configure it

A minimal on-host Alloy pipeline that tails /var/log/*.log, applies a host label, and ships to Loki. Real config, annotated:

// /etc/alloy/config.alloy
// "loki.source.file" tails files and emits log entries.
loki.source.file "system" {
  targets = [
    {
      __path__ = "/var/log/*.log",
      job      = "system",
      host     = constants.hostname,
    },
  ]
  forward_to = [loki.process.system.receiver]
}

// "loki.process" applies labels, parses, and drops.
loki.process "system" {
  stage.static_labels {
    values = {
      pipeline = "on-host",
      env      = "prod",
    }
  }

  // Drop noisy debug lines from a known chatty app.
  stage.match {
    selector = "{job=\"system\"}"
    stage.drop {
      expression  = ".*DEBUG.*connection_pool_reset.*"
      drop_counter_reason = "noisy_debug"
    }
  }

  forward_to = [loki.write.loki.receiver]
}

// "loki.write" ships to Loki. Real auth, real TLS, real endpoint.
loki.write "loki" {
  endpoint {
    url = "https://loki.internal.example.com/loki/api/v1/push"

    // Tenant ID for multi-tenant Loki.
    tenant_id = "prod"

    // Basic auth sourced from a file with 0600 perms.
    basic_auth {
      username = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }

    // Retry and backoff are governed here.
    retry_on_http_429 = true
    min_backoff_period = "1s"
    max_backoff_period = "1m"
    max_backoff_retries = 10
  }

  // External labels attach to every batch.
  external_labels = {
    collector = "alloy",
    region    = "eu-west-1",
  }
}

Each block is one stage. Three blocks, three failure domains, three places to put metrics on. The retry_on_http_429 and max_backoff_retries knobs are the backpressure surface for the final hop.

How to validate it

After applying the config, validate in order from local to remote.

# CONFIGURATION: alloy fmt checks syntax without running anything.
alloy fmt --check /etc/alloy/config.alloy

# (no output means the file is well-formed)

# CONFIGURATION: alloy validate parses and type-checks.
alloy validate /etc/alloy/config.alloy

ts=2026-08-13T11:42:01Z level=info msg="config valid"

# SERVICE-IMPACT: SIGHUP the running process to apply.
systemctl reload alloy

# READ-ONLY: confirm the agent reloaded the right file.
journalctl -u alloy -n 20 --no-pager | grep -E "config loaded|reload"

ts=2026-08-13T11:43:12Z level=info msg="loaded config" path=/etc/alloy/config.alloy

# READ-ONLY: confirm the agent is tailing the files it should be.
curl -s http://alloy-host:12345/metrics | grep loki_source_file

loki_source_file_target_last_parsed_timestamp_seconds{...} 1.726e+09
loki_source_files_failed_total{...} 0

# READ-ONLY: ship a known line and find it in Loki.
logger "rb-academy-smoke-$(date +%s)"

logcli query --since=2m \
  '{collector="alloy"} |~ "rb-academy-smoke"' \
  --addr=https://loki.internal.example.com

... 2026-08-13 11:43:55 ... rb-academy-smoke-1723559035

The smoke line appearing in Loki within a few seconds closes the loop from logger on the host through to a LogQL result.

How it can fail

Five failure modes recur in production observability stacks.

1. The silent source. A log file is rotated while the collector’s file handle is still on the renamed inode. New writes go to the new file but the agent keeps reading the empty one. Symptom: loki_source_files_failed_total flat, loki_source_file_target_last_parsed_timestamp_seconds frozen on a stale value, Loki has no recent entries for that job.
2. The label explosion. An application stamps a request UUID into a Loki label. The cardinality of the stream set explodes; the distributor rejects writes with 429 stream rate exceeded. Symptom: loki_distributor_ingester_append_failures_total rising, loki_distributor_streams orders of magnitude higher than expected.
3. The stuck buffer. A network partition between the collector and Loki fills the on-host disk buffer. The agent does not drop; it stops reading new entries. Symptom: loki_source_files_dropped_entries_total flat (it has not dropped; it has stalled), loki_write_dropped_entries_total also flat, but the host disk on /var/lib/alloy is at 100%.
4. The TLS time bomb. Loki’s certificate is renewed every 90 days; the collector’s CA bundle has not been refreshed since deploy. Writes fail with x509: certificate signed by unknown authority. Symptom: loki_write_client_errors_total rising, loki_write_remote_write_errors_total non-zero, no entries appearing in Loki.
5. The clock drift. A host’s ntp service stopped six hours ago. Loki rejects entries with timestamps more than two hours in the future or past. Symptom: Loki returns no rows for queries on the host’s labels in the last six hours, even though the agent metrics show entries leaving the host.

How to troubleshoot it

When Loki returns no rows for a host you believe is shipping, the diagnostic order matters. Diagnose before changing state.

1. Is the agent running? systemctl status alloy. If not, journalctl -u alloy -n 50 --no-pager for the cause.
2. Is it tailing the right files? curl -s http://host:12345/metrics | grep loki_source_file for loki_source_files_failed_total and the last_parsed_timestamp gauges.
3. Is it shipping successfully? curl -s http://host:12345/metrics | grep loki_write_ for loki_write_dropped_entries_total, loki_write_remote_write_errors_total, and loki_write_client_errors_total.
4. Is the network reachable? curl -v https://loki/loki/api/v1/push with a -u ingest:$PASS. A 401 means auth, a 403 means tenant, a connection refused means DNS or routing.
5. Is Loki accepting from this tenant? Look at the Loki distributor logs for stream_too_many or tenant_too_many_streams. The fix is upstream of the agent — on the application, not the pipeline.
6. Is the host clock sane? chronyc tracking or timedatectl status. A drift above two hours causes silent rejection.

Security implications

The pipeline crosses three trust boundaries: process to collector, collector to buffer, buffer to central store. Each crossing is a chance for exposure.

• Source to collector. Logs may contain credentials, PII, or tokens. The collector process must run as a low-privilege user and read only the files it is configured to read. Drop a chmod 0640 on /var/log/app/*.log and run the collector in a group that owns them.
• Collector to Loki. The HTTP push must be TLS. Basic auth or bearer token is mandatory in any environment that has more than one tenant or any human with access to the network. The password lives in a file with 0600, owned by the collector user.
• Loki to Grafana. Read-side auth is separate. Anyone with the query URL sees every label value, including host names, user IDs, request IDs, and IP addresses that may have been captured in structured metadata.

The defaults on a fresh install are deliberately permissive so the operator can prove the pipeline works. Production hardens them in the order above.

Performance implications

The pipeline is the highest-cardinality workload on most production hosts. Three knobs dominate:

• Label cardinality. Loki indexes every unique labelset. A labelset that varies per request (request ID, container ID) will blow the index. The discipline is to put high-cardinality values in the line body or in structured metadata, not in labels.
• Batch size and flush interval. Smaller batches mean lower latency and lower throughput. Larger batches mean the inverse. The defaults are a reasonable starting point; the right tuning depends on the line rate and the acceptable query latency.
• Buffer disk size. The disk buffer is the on-host safety net during a Loki outage. Set it to enough to absorb a one-hour outage at peak line rate; size it smaller only if the host disk cannot spare the space.

CPU is rarely the bottleneck. RAM scales with the in-memory queue depth, which scales with the line rate and the network latency to Loki. Disk I/O spikes during buffer flushes; an SSD-backed host disk is appropriate.

Production guidance

• Instrument every stage. The loki_source_*, loki_process_*, and loki_write_* counter families are published on the agent’s /metrics. Scrape them and alert on rate(loki_write_dropped_entries_total[5m]) > 0 and on loki_source_files_failed_total > 0.
• Pin versions in the platform manifest. Grafana Alloy releases monthly; OTel Collector follows a similar cadence. Read the release notes before bumping. Breaking changes to the pipeline config do happen.
• Test the buffer. Pull the network cable (or block the Loki port with iptables) for ten minutes. Confirm the agent buffers, then drains when the network returns. Confirm the metrics tell the story.
• Document the rollback. Keep the previous config file on the host. The reload is systemctl reload alloy; the rollback is the same command with the previous file restored.

Verification

You should now be able to answer:

• What are the four stages of a log pipeline and which failure mode does each one own?
• When does the on-host pipeline beat the central relay pipeline, and vice versa?
• What is the first metric to check when Loki has no rows for a host you believe is shipping?
• How does the symptom of buffer overflow differ from the symptom of network partition?

Quiz