Chunks

At 03:14 an ingester pod is rescheduled to a new node by the Kubernetes scheduler. The pod restarts in eight seconds. Loki accepts pushes again. Grafana queries return the last six minutes of data. The on-call engineer looks at the ingester metrics and sees a gap: the eight seconds during the restart, and the forty seconds before the head block caught up. The missing time is the in-memory head block that did not survive the move. The team did not enable the write-ahead log.

This is the failure shape of a chunk: an in-memory structure that depends on the WAL to survive a restart. Without the WAL, every restart costs the team the most recent time window.

What it is

A chunk is a time-bounded, compressed bundle of log lines for a single stream. The chunk is the unit of long-term storage: Loki writes a chunk to the object store as a single gzip-compressed file with structured headers and a content-addressed name. The chunk is immutable once written; later writes for the same stream go into a new chunk.

Three kinds of chunk live in the system:

   Kind            Where it lives     Lifetime
   --------------  -----------------  --------------------------
   head block      ingester memory    stream open for writes
   WAL segment     ingester disk      until replay completes
   flushed chunk   object store       until retention deletes it

The head block is the in-memory structure that accumulates lines for an open stream. It is a per-stream buffer plus an index entry that points at it. The head block is not a chunk in the long-term sense; it has no immutable file and no object store path. The moment the stream is flushed, the head block becomes a chunk.

The WAL segment is a write-ahead log on local disk. Every push that lands on an ingester is also appended to the WAL before the ingester acknowledges. The WAL is replayed on ingester startup to reconstruct head blocks that were lost in the restart.

The flushed chunk is the gzip-compressed file in the object store. Once written, it is read-only. The querier and the compactor read it; nobody writes to it.

Why a sysadmin cares

Chunks are the unit the bucket charges for, the unit the querier fetches, and the unit the compactor ages out. Four operational pains appear in every Loki cluster that does not have its chunk configuration tuned:

1. Chunks too small. A short chunk_idle_period produces many small chunks per stream. The querier pays the cost of fetching many chunks per query; the bucket pays the cost of many GetObject calls per query; the compactor pays the cost of many small files to merge.
2. Chunks too old. A long max_chunk_age produces over-large chunks. Loki v13 caps a chunk at chunk_target_size (default 1.5 MiB), but a single chunk spanning twelve hours holds many gigabytes of label-set metadata and is slow to query.
3. Restart loses recent data. Without the WAL, an ingester restart loses every head block in memory. The query_ingester_within window (default 30 minutes) hides the loss for queries that include the recent window, but any query for the exact restart window returns no data.
4. Disk fills with WAL. The WAL is append-only. Without a checkpoint-and-truncate cycle, the WAL grows until the disk is full and the ingester refuses to write.

How it works

A chunk has four flush triggers:

   +-------------------+
   | push arrives      |
   +-------------------+
            |
            v
   +-------------------+
   | append to head    |
   | block + WAL       |
   +-------------------+
            |
            | trigger:
            |   1. chunk_idle_period (default 30m) of no writes
            |   2. max_chunk_age     (default 2h)
            |   3. chunk_target_size (default 1.5 MiB)
            |   4. SIGTERM (graceful shutdown)
            v
   +-------------------+
   | encode + gzip     |
   | write to object   |
   | store             |
   +-------------------+
            |
            v
   +-------------------+
   | evict head block  |
   | from memory       |
   +-------------------+

The four triggers are not equally weighted. chunk_idle_period is the most common path: a quiet stream flushes within thirty minutes of its last push. max_chunk_age is the safety net: a noisy stream flushes every two hours regardless of activity. chunk_target_size is the hard ceiling: a single chunk never exceeds 1.5 MiB of compressed bytes; beyond that, Loki opens a new chunk for the same stream.

The 1.5x rule: max_chunk_age should be approximately 1.5x chunk_idle_period. The relationship guarantees that a stream that is being written at a steady rate gets to flush because it hit the size ceiling before it hit the age ceiling. A max_chunk_age that is too close to chunk_idle_period produces many small chunks. A max_chunk_age that is too far above chunk_idle_period produces over-large chunks.

   chunk_idle_period   max_chunk_age   behaviour
   ----------------    -------------   ----------------------------
   30m (default)       2h (default)    quiet streams flush on idle,
                                       noisy streams flush on age
   30m                 45m             <1.5x: many small chunks
   30m                 3h              >1.5x: chunks grow to size
                                       ceiling before age ceiling
   5m                  2h              noisy stream, ~24 flushes/day

How to configure it

The chunk lifecycle is governed by the ingester block.

# /etc/loki/config-write.yaml (extract)

ingester:
  # Idle flush. A stream with no writes for this long is
  # flushed. The default is 30m; lower it (e.g. 5m) for
  # workloads that need faster query availability for
  # quiet streams. Raising it past 1h produces fewer
  # small chunks but longer query latency for the
  # last-write.
  chunk_idle_period: 30m

  # Age ceiling. A stream that has been open for this long is
  # flushed regardless of activity. The default is 2h; the
  # rule of thumb is 1.5x chunk_idle_period to let chunks
  # grow to chunk_target_size before being force-closed.
  max_chunk_age: 2h

  # Target chunk size. The Loki binary uses this as the
  # soft ceiling for a single chunk. The default 1572864
  # bytes (~1.5 MiB) is a balance between compaction
  # efficiency and object-store read cost.
  chunk_target_size: 1572864

  # Maximum number of chunks in memory per ingester. The
  # safety net against a runaway stream creating too many
  # head blocks. The default 1000000 is high enough for
  # production; lower it only if memory is constrained.
  max_chunks_per_query: 2000000

  # Write-ahead log. Enable to survive an ingester restart.
  wal:
    enabled: true
    dir: /var/lib/loki/wal

    # Checkpoint and truncate the WAL every 5m (default).
    # A checkpoint marks the WAL segment as "all flushed";
    # the segment can then be removed. A checkpoint that
    # never advances means flushes have stalled.
    checkpoint_duration: 5m

  # Lifecycler controls how this ingester joins the ring.
  lifecycler:
    ring:
      kvstore:
        store: consul
      replication_factor: 3
    heartbeat_period: 5s
    join_after: 30s
    observe_period: 10s
    final_sleep: 0s

Three details to call out:

• chunk_target_size is the soft ceiling, not the hard cap. Loki will close a chunk slightly above this value rather than refuse the write. The hard ceiling is the available ingester memory.
• wal.checkpoint_duration is the interval at which the WAL is truncated. A checkpoint that fails (because flushes have stalled) means the WAL grows.
• lifecycler.final_sleep is the delay between SIGTERM and SIGKILL on shutdown. The default is 0; for graceful shutdown set it to a value that exceeds the longest expected flush.

How to validate it

Severity: READ-ONLY.

1. Confirm the ingester is accepting and writing chunks:

curl -s http://loki-write:3100/metrics | grep loki_ingester_chunks_created_total
# loki_ingester_chunks_created_total 1842

curl -s http://loki-write:3100/metrics | grep loki_ingester_chunk_age_seconds
# loki_ingester_chunk_age_seconds_bucket{le="60"} 1842
# loki_ingester_chunk_age_seconds_bucket{le="300"} 1842
# loki_ingester_chunk_age_seconds_bucket{le="1800"} 1842
# loki_ingester_chunk_age_seconds_bucket{le="7200"} 1842

2. Confirm chunks are reaching the bucket:

# READ-ONLY: list chunk files for a recent day.
aws s3 ls s3://prod-loki-chunks/tenant/fake/2026-08-13/ \
  --recursive | head -5
# 2026-08-13 09:14:22      12345 fake/08/xxx.../01Hxx...
# 2026-08-13 09:14:22       8934 fake/08/yyy.../01Hyy...
# ...

3. Confirm the WAL exists, is being written, and is being checkpointed:

ls -la /var/lib/loki/wal/
# -rw------- 1 loki loki 67108864 checkpoint.00012345
# -rw------- 1 loki loki 41943040 00012345

curl -s http://loki-write:3100/metrics | grep loki_ingester_wal
# loki_ingester_wal_bytes 41943040
# loki_ingester_wal_loaded 0

4. Confirm a flush happens on graceful shutdown:

# CONFIGURATION: send SIGTERM to the ingester and watch logs.
kill -TERM $(pidof loki)
journalctl -u loki-write -n 50 -f | grep -E '(flushing|flush|wal)'
# loki_write.go: flushing chunks
# loki_write.go: flushed 1842 chunks
# loki_write.go: shutting down wal

How it can fail

Six shapes cover the most common chunk-related incidents:

1. WAL disabled. The ingester starts and accepts pushes, but a restart loses every in-memory head block. Symptom: the gap between the last push before the restart and the first push after the restart is silent in queries. loki_ingester_wal_loaded is missing.
2. WAL fills the disk. wal.checkpoint_duration is too long for the flush rate, or flushes are failing. Symptom: the disk fills; the ingester refuses to write; pushes return 500. loki_ingester_wal_bytes rises monotonically.
3. max_chunk_age too close to chunk_idle_period. A busy stream hits the age ceiling before the size ceiling and produces many small chunks. Symptom: loki_ingester_chunks_created_total rises faster than loki_ingester_chunks_flushed_total and the bucket has many small files per stream.
4. max_chunk_age too far above chunk_idle_period. A busy stream hits the size ceiling but stays open for hours in ingester memory. Symptom: ingester memory usage tracks active stream count; loki_ingester_chunk_age_seconds histogram p99 climbs.
5. Flush to object store failing. The ingester tries to write a chunk; S3 returns 403 or the network times out. Symptom: head blocks accumulate in memory; the ingester eventually rejects pushes with OOM; the WAL grows because checkpoints cannot advance.
6. Chunk format drift. The chunk was written under v12 but the current schema_config says v13. Symptom: the querier returns chunk not found for any query that targets a stream whose chunks were written under the old schema. The loki_index_request_duration_seconds histogram spikes.

How to troubleshoot it

The diagnostic order for a chunk-related incident:

1. Is the WAL enabled and being checkpointed? loki_ingester_wal_bytes and loki_ingester_wal_checkpoint should both advance on a healthy ingester. A flat WAL checkpoint means flushes have stalled.
2. Are chunks being created at the right rate? loki_ingester_chunks_created_total rate should match the rate of loki_ingester_chunks_flushed_total. A gap is a stalled flush path.
3. What is the chunk age distribution? loki_ingester_chunk_age_seconds bucket histogram. A p99 above max_chunk_age means streams are being closed late; a p50 below chunk_idle_period means streams are being closed too early.
4. What is the chunk size distribution? loki_ingester_chunk_size_bytes histogram. A p99 at chunk_target_size means chunks are being closed on size; a p99 well below means streams are being closed on age (small chunks).
5. Is the object store reachable? loki_objstore_request_duration_seconds and loki_ingester_chunk_encode_duration_seconds. A flat encode time with a growing object-store time means the flush path is the bottleneck.
6. Is the schema version right? Compare schema_config.configs against the chunk’s recorded version. Mismatches produce chunk not found errors.

Security implications

Chunks live in the object store under the bucket the operator controls. Three surfaces:

• Bucket credentials. A leaked AWS key with s3:GetObject is a read-only breach. A leaked key with s3:PutObject is a write breach; an attacker could overwrite chunk files with garbage and cause silent data loss for that time window.
• Chunk file access. S3 object locks, bucket versioning, and IAM policies are the access boundaries. The Loki process should use scoped credentials (a role with s3:GetObject, s3:PutObject, s3:ListBucket, and s3:DeleteObject only on the chunks bucket).
• WAL on local disk. The WAL contains every line that has not yet been flushed. A stolen disk is a privacy incident. Encrypt the WAL volume at rest.

Performance implications

The cost of chunk design is paid at four points:

• Flush cost. A flush writes a chunk to the object store and updates the TSDB index. The cost is amortised across the chunk size; a 1.5 MiB chunk is cheaper per MB than a 100 KiB chunk because the per-flush overhead is fixed.
• Querier fetch cost. A query that targets a stream with many small chunks pays one GetObject per chunk. A query against a stream with 12 chunks per day costs 12 fetches; one with 48 costs 48.
• Compactor cost. The compactor merges index entries for retired chunks. A bucket with many small chunks has many index entries; the merge pass is correspondingly slower.
• WAL replay cost. A long WAL segment takes longer to replay on startup. The first minutes after an ingester restart show reduced availability.

The right sizing:

• Default values for most deployments. 30m idle, 2h age, 1.5 MiB target.
• Quiet streams (less than one push per hour). Lower chunk_idle_period to 5m. The stream is rarely open for long; the size and age ceilings never bind.
• Noisy streams (more than one push per second). Raise chunk_idle_period to 1h. The stream is always open; the idle period never binds. The age and size ceilings govern.
• NVMe-backed WAL disk. The WAL is append-only and fsync-on-write. A network disk is the wrong answer.

Production guidance

• Enable the WAL on every ingester. The cost is a few GB of disk; the benefit is the absence of restart-induced data loss.
• Pin the WAL to a fast, dedicated disk. The WAL is the bottleneck on a restart, a flush storm, or a slow object store.
• Keep the default chunk_target_size (1.5 MiB) unless the bucket is unusually expensive per GetObject. The default is a balance.
• Set max_chunk_age to 1.5x chunk_idle_period as a starting point. Adjust to the traffic profile after one week of metrics.
• Monitor loki_ingester_wal_bytes and alert when the WAL exceeds 80% of the disk. The WAL is the only safety net before the disk fills and pushes start failing.
• Monitor loki_ingester_chunk_age_seconds p99. A p99 that climbs past max_chunk_age means the flush path is slow.

Verification

You should now be able to answer:

• What are the four flush triggers for a Loki chunk, and which is the most common in steady-state traffic?
• What is the 1.5x rule for max_chunk_age and why does it matter?
• Where does the WAL live, what does it contain, and when is it replayed?
• What is the difference between the head block, the WAL segment, and the flushed chunk?
• Which metric shows whether the WAL is being checkpointed?

Quiz