Ingester Lifecycle

At 02:14 a push to Loki succeeds in fourteen milliseconds. At 02:47 the same push takes eleven seconds and returns 503 three times in a row. The on-call engineer inspects the distributor metrics and sees loki_distributor_ingester_append_failures_total rising. The ingesters are returning errors. The cause is a rolling restart of the ingester pool: three new pods joined the ring during the restart, the old pods entered LEAVING, and the handoff between the old pods and the new pods is failing because the WAL is empty and the new pods have no streams to take ownership of.

This is the failure shape of the ingester ring: a stateful role whose membership changes cause distributor-side errors that the dashboard does not surface for a minute.

What it is

The ingester is the only stateful role on the Loki write path. It holds the head blocks for every stream it owns in memory, plus a write-ahead log on local disk. When a stream arrives at the distributor, the distributor hashes the stream’s label set, looks up the ingester ring to find the ingesters that own that hash range, and forwards the push to each of them. The number of replicas that receive each push is the replication factor (RF).

Three concepts bound the ingester’s role:

   Concept              What it owns                          What changes it
   ------------------   ----------------------------------   --------------------
   stream               a unique label set                    push arrival, idle timeout
   ingester instance    the head blocks + WAL for its owned   ring membership change
                        streams                               (join, leave, handoff)
   replication factor   the number of ingesters that          per-tenant override
                        receive each push                     via runtime config

The ingester is stateful because the head blocks are not durable in the bucket. Flushed chunks are durable; in-memory head blocks are not. Restarting an ingester without WAL replay loses the head blocks for every stream it owns. The ring exists so that Loki can tolerate ingester failures: when an ingester fails, another ingester takes ownership of its streams.

Why a sysadmin cares

The ingester is the role that breaks the most often. Three operational pains appear in every Loki cluster that has not had its ingester configuration and capacity tuned:

1. Ring churn. Adding or removing an ingester pod reshuffles the hash space. Every distributor must re resolve the ring and may send pushes to the wrong ingester for a few seconds. Symptom is distributor-side connection refused errors during the roll.
2. Insufficient RF. A push that arrives when one ingester is down and the RF is 3 against a pool of 2 replicas waits for the third replica that does not exist. Symptom is distributor latency climbing and pushes failing with ingester not found.
3. Handoff on graceful shutdown. A SIGTERM that is not honoured means the ingester exits without handing off its streams. The streams become unowned for the duration of final_sleep; pushes for those streams fail. Symptom is a window of failed pushes every time an ingester restarts.

How it works

A single push flows through the distributor and lands on N ingesters, where N is the replication factor.

   push arrives at distributor
            |
            v
   +-------------------+
   | distributor       |
   | hash(stream)      |-----> ring lookup
   | stream -> N ingester IDs
   +-------------------+
            |
            +--------+
            |        |
            v        v
   +-------+--+   +--+--------+
   | ingester A| | ingester B | ... (RF total)
   +----------+  +-----------+
        |
        v
   head block + WAL

The ring is a consistent hash ring. Each ingester owns a range of hash space. The distributor computes the hash of the stream’s label set, walks the ring, and picks the first N ingesters whose ranges cover that hash. The selection is deterministic: the same label set always lands on the same set of ingesters, regardless of which distributor does the lookup.

Ring membership states

An ingester moves through five states during its life:

   PENDING   -- start of life, not in the ring yet
       |
       v
   JOINING   -- announced itself to the ring KV, observing peers
       |
       v
   ACTIVE    -- accepting pushes
       |
       v
   LEAVING   -- SIGTERM received, transferring ownership
       |
       v
   gone      -- removed from the ring

The transitions are driven by the lifecycler:

   join_after:        time to wait after PENDING before JOINING
   observe_period:    interval between JOINING and ACTIVE
   heartbeat_period:  interval between ACTIVE heartbeats
   final_sleep:       time to wait in LEAVING before exit

The transition from ACTIVE to LEAVING is the handoff. During LEAVING, the ingester hands off every stream it owns to a successor ingester in the ring (the next ingester in the hash range). The handoff is what makes graceful shutdown possible; without it, every restart is a hard restart that loses ownership.

The handoff

The handoff is a per-stream transfer of ownership from the leaving ingester to a successor. The leaving ingester flushes every chunk that has been idle for chunk_idle_period to the object store, then signals the successor to take ownership. The successor reconstructs the head blocks by reading the chunks from the object store or by replaying the WAL on its own disk (if the WAL is mounted on shared storage).

   SIGTERM
        |
        v
   +-------------------+
   | flush idle chunks |
   | to object store   |
   +-------------------+
        |
        v
   +-------------------+
   | for each owned    |
   | stream:           |
   |   pick successor  |
   |   transfer tokens |
   |   transfer WAL    |
   +-------------------+
        |
        v
   +-------------------+
   | sleep final_sleep |
   | (default 0s)      |
   +-------------------+
        |
        v
   exit

A handoff that runs out of time is a hard exit. The streams become unowned for join_after seconds (the time it takes for the next ingester to claim them) and pushes for those streams fail.

How to configure it

The ingester’s ring membership is governed by the ingester.lifecycler block.

# /etc/loki/config-write.yaml (extract)

ingester:
  # Chunk lifecycle settings (covered in 03-chunks).
  chunk_idle_period: 30m
  max_chunk_age: 2h
  chunk_target_size: 1572864

  wal:
    enabled: true
    dir: /var/lib/loki/wal
    checkpoint_duration: 5m

  # Lifecycler controls ring membership.
  lifecycler:
    # Address advertised to the ring KV. Must be reachable
    # by the distributor and other ingesters.
    address: loki-write-0.loki-write-headless.loki.svc.cluster.local

    # Ring KV store. consul is the right choice for
    # multi-cluster; memberlist is the right choice for
    # single Kubernetes cluster.
    ring:
      kvstore:
        store: consul
        consul:
          host: consul.loki.svc.cluster.local:8500

      # Replication factor. RF=3 means every push lands on
      # three ingesters. The replica count must be >= RF.
      replication_factor: 3

    # How long to wait in PENDING before JOINING. Allows
    # caches and connections to warm up.
    join_after: 30s

    # How long to observe peers between JOINING and ACTIVE.
    observe_period: 10s

    # Heartbeat interval. The KV is updated on this cadence.
    heartbeat_period: 5s

    # Heartbeat timeout. If the KV has not seen a heartbeat
    # from this ingester in this long, it is eligible for
    # re-assignment.
    heartbeat_timeout: 1m

    # Time to sleep in LEAVING. Set this to longer than the
    # longest expected flush to ensure the handoff completes.
    final_sleep: 30s

Three details to call out:

• replication_factor: 3 against a replica count of 2 means pushes fail when one ingester is down. The replica count must be greater than or equal to RF.
• final_sleep: 30s is the time the LEAVING ingester waits for the handoff to complete. The right value is the longest expected flush time plus a margin. A value of 0 skips the handoff entirely.
• heartbeat_timeout: 1m is the time after which the KV considers an ingester unhealthy. The value should be several heartbeat_periods to tolerate transient network issues.

How to validate it

Severity: READ-ONLY.

1. Confirm the ingester is in the ring:

curl -s http://loki-write:3100/ingester/ring | jq '.shards'
# {
#   "0": {
#     "state": "ACTIVE",
#     "tokens": [1, 2, 3, ...]
#   },
#   ...
# }

curl -s http://loki-write:3100/ingester/ring \
  | jq '[.shards | to_entries[] | select(.value.state != "ACTIVE")] | length'
# 0  (every ingester is ACTIVE)

2. Confirm the ring has the right number of replicas:

curl -s http://loki-write:3100/ingester/ring | jq '.members | length'
# 3  (matches RF and replica count)

3. Confirm a push lands on three ingesters:

curl -sG http://loki-write:3100/loki/api/v1/push \
  --data-urlencode 'streams[0].stream.job=ring-test' \
  --data-urlencode 'streams[0].values[0][1]=hello' \
  -w '%{http_code}\n'
# 204

# Inspect each ingester directly to confirm it received the push.
for ing in loki-write-0 loki-write-1 loki-write-2; do
  curl -sG http://${ing}:3100/loki/api/v1/query \
    --data-urlencode 'query={job="ring-test"}' \
    | jq '.data.result | length'
done
# 1
# 1
# 1

4. Confirm the ingester metrics are healthy:

curl -s http://loki-write:3100/metrics | grep loki_ingester_lifecycler
# loki_ingester_lifecycler_ring_inconsistencies_total 0
# loki_ingester_lifecycler_tokens_owned{...} 33554432

curl -s http://loki-write:3100/metrics | grep loki_ingester_streams
# loki_ingester_streams{tenant="fake"} 8123

5. Confirm the WAL is being replayed correctly after a restart:

# After a SIGTERM and restart, check the WAL load metric.
curl -s http://loki-write-3100/metrics | grep loki_ingester_wal_loaded
# loki_ingester_wal_loaded_total 1842  (one record per stream recovered)

How it can fail

Six shapes cover the most common ingester-related incidents:

1. Replica count below RF. A cluster has 2 ingesters with RF=3. Every push waits for the third replica that does not exist. Symptom is distributor latency climbing and pushes failing with ingester not found; the distributor logs show replication factor not met.
2. Ring churn during a rolling restart. Adding or removing ingesters triggers ring reshuffles. Symptom is loki_ingester_lifecycler_ring_inconsistencies_total rising and the distributor logging connection refused during the roll.
3. Handoff timeouts. final_sleep is shorter than the flush time. The LEAVING ingester exits before the handoff completes. Symptom is unowned streams for the duration of join_after; pushes for those streams fail.
4. WAL not shared across pods. The ingester restart reads the local WAL but the new pod has no WAL. Symptom is loki_ingester_wal_loaded_total flat at zero after the restart and the head block reconstruction misses every stream that was in flight.
5. KV store unreachable. The ring KV (consul or memberlist) is down. Symptom is the distributor unable to resolve the ring; every push returns 500; loki_distributor_ingester_clients drops to zero.
6. Per-tenant stream imbalance. One ingester owns most of the streams for a noisy tenant. Symptom is one ingester’s memory usage tracks that tenant’s stream count; the other ingesters are idle.

How to troubleshoot it

The diagnostic order for an ingester-related incident:

1. Is the ring KV reachable? Inspect the KV (consul UI or memberlist health). A down KV is a global outage; every distributor returns 500.
2. Is every ingester ACTIVE? curl /ingester/ring on any ingester and inspect .shards[].state. An ingester stuck in JOINING has not completed the observe period.
3. What is the ring inconsistency count? loki_ingester_lifecycler_ring_inconsistencies_total. Any non-zero rate is a sign of ring churn.
4. Are pushes landing on RF ingesters? loki_distributor_ingester_clients{status="success"}. A value below RF means the distributor is missing replicas.
5. What is the per-tenant stream count? loki_ingester_streams{tenant="..."}. An unbalanced distribution means one ingester is hot.
6. Is the WAL replaying on restart? loki_ingester_wal_loaded_total after a restart. A flat value means the WAL is missing or the new pod has the wrong mount.

Security implications

The ingester holds every line that has not yet been flushed. Three surfaces:

• WAL on local disk. The WAL contains every line in flight. A stolen disk is a privacy incident. Encrypt the WAL volume at rest.
• Ring KV credentials. The KV store credentials (consul token, memberlist secret) are the boundary between legitimate ingesters and impostors. A leaked token lets an attacker join the ring and accept pushes.
• mTLS between distributor and ingester. The gRPC connection between distributor and ingester carries the push payload. mTLS prevents an attacker on the network from reading the streams.

Performance implications

The performance cost of the ingester is paid at three points:

• Head block memory. Every active stream holds a struct in memory (the label set plus a pointer to the head block). The cost is ~few KiB per stream. A cluster with 1 million active streams pays ~few GiB in ingester RAM.
• WAL disk bandwidth. Every push is appended to the WAL before being acknowledged. The WAL bandwidth is the write bandwidth of the disk. NVMe is the right answer; network storage is the wrong one.
• Distributor ring cache. The distributor caches the ring for a few seconds. A short cache TTL means more KV reads; a long TTL means more ring churn events going undetected.

The right sizing:

• Single tenant, low cardinality. RF=3 against 3 ingesters. The defaults are fine.
• Multi-tenant. RF=3 is still the default, but consider per-tenant RF overrides via the runtime config file for tenants that need higher durability.
• High cardinality. Add ingester replicas before the head-block memory ceiling is reached. The rule of thumb is head_block_bytes < 50% of available memory.

Production guidance

• Set RF=3 for production. RF=1 is single point of failure; RF=2 is insufficient for a zone-aware deployment.
• Set final_sleep to at least the longest expected flush time. The default of 0s skips the handoff.
• Pin the WAL to a fast, dedicated disk. The WAL is the bottleneck on a restart, a flush storm, or a slow object store.
• Monitor loki_ingester_lifecycler_ring_inconsistencies_total and alert on any non-zero rate. Ring churn is the leading indicator of distributor-side errors.
• Monitor loki_ingester_streams{tenant="..."} per tenant. An unbalanced distribution means one ingester is hot.
• Roll ingesters one at a time during a deploy. A batch restart of the entire ingester pool triggers a wave of handoffs that can stall the write path.

Verification

You should now be able to answer:

• Why is the ingester the only stateful role on the Loki write path?
• How does the distributor use the ring to pick the ingesters for a push?
• What are the five ring membership states, and what is the role of the handoff?
• What is the difference between RF=3 against 3 replicas and RF=3 against 2 replicas?
• What is the role of final_sleep in the lifecycler, and why should it be non-zero?

Quiz