Loki Architecture Overview · Observability

At 02:14 a push of a single line to Loki succeeded in fourteen milliseconds. At 02:47 the same line took eleven seconds and returned 503 three times in a row. The on-call engineer, trained on simple-scalable mode, restarted the Loki pod and watched the latency get worse. The pod was not the bottleneck. A new ingester replica had joined the ring during a rolling restart, half the distributors were holding stale ring state, and the compactor was three days behind on retention sweep. Restarting the pod made everything ring-inconsistent.

This lesson is the architecture. The seven components that make up Loki, the two deployment modes that bind them together, and the order in which to scale them.

What it is

Loki is a label-indexed log store backed by an object store. The label index answers “which streams match this label selector” in milliseconds; the object store holds the actual log bytes in time-bounded compressed chunks. Loki does not index log content. A LogQL query matches streams by label, then scans only the chunks whose streams matched.

The pipeline has seven named components. Three are mandatory in any deployment; the rest are added as load grows or as tenancy is introduced.

   Role              Stateful?  Scales on                Default target
   ----------------  ---------  -----------------------  ----------------
   distributor       no         pushes per second        distributor
   ingester          yes        streams + chunks in mem  ingester
   querier           no         concurrent queries       querier
   query-frontend    no         long LogQL queries       query-frontend
   compactor         no         bucket marker count      compactor
   ruler             no         rule evaluation cost     ruler
   index-gateway     no         index file fan-out       index-gateway

The first three are mandatory in any deployment that wants to write and read. The query-frontend is optional but pays off once the querier pool exceeds two replicas. The compactor is mandatory in any deployment that retains data beyond a single ingester flush window. The ruler is mandatory in any deployment that evaluates LogQL alerting rules. The index-gateway is mandatory in any microservices deployment using the TSDB index.

Why a sysadmin cares

Three operational pains push a sysadmin to care about the architecture rather than treat Loki as a black box:

One role is the bottleneck and the rest are idle. A platform with heavy ingest but light query load wastes CPU when distributor and querier share a process. Splitting them lets each scale on its own metric.
Failure isolation. A crash of the ingester in simple-scalable mode also restarts the querier running in the same process. In microservices mode only the ingester pod restarts; queriers keep serving from long-term storage.
Multi-tenant isolation. Multi-tenant deployments need per-tenant rate limits at the distributor and per-tenant compaction at the compactor. Both are easier when each role is its own workload with its own config map.

The cost of caring is operational surface. A microservices Loki needs seven or more StatefulSets / Deployments, a per-role SLO, and an operator who can read ingester ring state.

How it works

The pipeline stays the same in both modes. The difference is that in simple-scalable mode every box in the picture is the same binary launched with -target=all; in microservices mode each box is its own pod, with its own address, and the wiring between them is explicit in the config.

   Application / Alloy / Promtail
             |
             v
   +-------------------+      +-------------------+
   | distributor x N   | ---> | ingester x N      | ---> S3 / GCS / Azure
   | rate-limit + hash |      | head block + WAL  |
   | fan-out to RF     |      | ring, RF=3        |
   +-------------------+      +-------------------+
                                      |
                                      | (chunk flush)
                                      v
                              +---------------+
                              | object store  |
                              | /<tenant>/... |
                              +---------------+
                                      ^
                                      | (chunk fetch)
             +-------------------+   |
             | query-frontend    |---+ (split, cache, fan-out)
             +-------------------+
                      |
                      v
             +-------------------+
             | querier x N       |
             | LogQL execution    |
             +-------------------+
                      ^
             +-------------------+
             | index-gateway     |----> serves TSDB index files
             +-------------------+
             +-------------------+
             | compactor (one)   |----> retention + index compaction
             +-------------------+
             +-------------------+
             | ruler             |----> alerting rules over LogQL
             +-------------------+

The roles, in order of how often they need attention:

Distributor. Stateless. Validates streams, applies per-tenant rate limits, hashes the stream label set, and forwards the push to the ingesters that own that hash range. CPU-bound on JSON / Protobuf decoding.
Ingester. Stateful. Accumulates log lines into head blocks in memory, fsyncs a write-ahead log to local disk, and flushes completed chunks to long-term storage. Runs as a StatefulSet with a memberlist or consul ring KV.
Querier. Stateless. On a LogQL request, fetches chunks from long-term storage, merges with chunks from in-memory ingesters, applies the LogQL filter and pipe expressions, returns results. CPU- and network-bound on chunk fetches.
Query-frontend. Stateless. Splits long LogQL queries into smaller intervals, caches subresults in an embedded FIFO cache, and parallelises work across queriers. Required when the querier pool exceeds two replicas.
Compactor. Mostly stateless but must run as a singleton. Walks the bucket, merges small TSDB index files, applies retention. Runs as a single replica by default; sharded compaction is available but rarely necessary below millions of active streams.
Ruler. Stateless but with storage. Evaluates LogQL alerting rules on a schedule and ships firing alerts to Alertmanager.
Index-gateway. Stateless. Serves the per-tenant TSDB index files from long-term storage to queriers. Mandatory in microservices mode; absent in simple-scalable mode (the querier reads the index directly).

The two deployment modes

Loki ships in two shapes:

  Mode              Workloads       When to use
  ----------------  --------------  ----------------------------------
  simple-scalable   one binary,     single-tenant, low-to-medium load,
                    four targets    small team, "I want Loki to just
                    (read, write,   work"
                    backend, ruler)
  microservices     seven binaries  multi-tenant, medium-to-high load,
                    each as its     "I want one role to be able to
                    own workload    scale independently of the rest"

In simple-scalable mode, one binary runs the union of every component section in the config. The -target flag picks which components are actually loaded. A typical production simple-scalable deployment runs four replicas: -target=write (distributor + ingester), -target=read (querier + query-frontend), -target=backend (compactor + index-gateway), -target=ruler.

In microservices mode, each role is its own process with its own config and its own replicas. The wiring between roles is explicit: the distributor must know the ingester gRPC address; the querier must know the query-frontend address; the compactor must know the object store.

How to configure it

A simple-scalable config covers the four -target values. The sections below correspond to the four targets. Microservices mode splits each section into its own config map.

# /etc/loki/config-write.yaml
# Mode: simple scalable, write target. -target=write on the CLI.

auth_enabled: false

server:
  http_listen_port: 3100    # Alloy, Promtail, and the read path push here
  grpc_listen_port: 9095    # distributor fans out to ingesters over gRPC
  log_level: info

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-write-0.loki-write-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# Distributor fans pushes out to ingesters over the ring.
distributor:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500

# Ingester controls chunk lifetime and WAL.
ingester:
  chunk_idle_period: 30m
  max_chunk_age: 2h              # 1.5x chunk_idle_period under steady load
  wal:
    enabled: true
    dir: /var/lib/loki/wal
  lifecycler:
    ring:
      kvstore:
        store: consul
      replication_factor: 3

# /etc/loki/config-read.yaml
# Mode: simple scalable, read target. -target=read on the CLI.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-read-0.loki-read-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

querier:
  query_timeout: 60s
  query_ingester_within: 30m    # also ask live ingesters for recent data

query_range:
  split_queries_by_interval: 24h
  parallelise_sharded_queries: true
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 500

# /etc/loki/config-backend.yaml
# Mode: simple scalable, backend target. -target=backend on the CLI.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-backend-0.loki-backend-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

compactor:
  working_directory: /var/lib/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: s3

index_gateway:
  mode: simple

Four details worth calling out:

common.ring.kvstore is shared by every target that needs a ring (distributor, ingester, compactor). A typo in the store name (e.g. consul vs memberlist) is caught only at first use, not at parse time. Validate before restarting.
ingester.lifecycler.ring.replication_factor sets the replication of every push. RF=3 means each push is written to three ingesters before the distributor returns 200. The value must be less than or equal to the ingester replica count.
query_range.parallelise_sharded_queries toggles the querier-side parallelisation of split queries. Off by default in older versions; on by default in Loki 3.x.
index_gateway.mode: simple is the right choice for single-tenant deployments. Multi-tenant deployments need mode: ring so the gateway ring balances per-tenant index reads across replicas.

How to validate it

Severity: READ-ONLY unless restarting a service.

Confirm the binary accepted the config:

loki -config.file=/etc/loki/config-write.yaml -target=write -verify-config
# expected: "config is valid" on stdout; exit code 0.

Confirm each role is /ready:

for role in write read backend ruler; do
  echo -n "$role: "
  curl -s "http://loki-${role}:3100/ready"
  echo
done
# write: ready
# read: ready
# backend: ready
# ruler: ready

Confirm the ingester ring has the right membership:

curl -s http://loki-write:3100/ingester/ring | jq '.members | length'
# 3
curl -s http://loki-write:3100/ingester/ring \
  | jq '.shards | to_entries[].value | {state: .state, tokens: (.tokens | length)}'
# {"state": "ACTIVE", "tokens": 33554432}
# {"state": "ACTIVE", "tokens": 33554432}
# {"state": "ACTIVE", "tokens": 33554432}

Confirm pushes arrive via the LogQL API:

curl -sG http://loki-write:3100/loki/api/v1/push \
  --data-urlencode 'streams[0].stream.job=test' \
  --data-urlencode 'streams[0].values[0][1]=hello'
# expected: 204 No Content

curl -sG http://loki-read:3100/loki/api/v1/query \
  --data-urlencode 'query={job="test"}' | jq '.data.result | length'
# 1

Confirm the querier and compactor are making progress:

curl -s http://loki-read:3100/metrics | grep loki_querier_query_seconds_count
# loki_querier_query_seconds_count{...} 1842

curl -s http://loki-backend:3100/metrics \
  | grep loki_compactor_oldest_processed_age_seconds
# loki_compactor_oldest_processed_age_seconds 86400

How it can fail

Six shapes appear repeatedly in production Loki incidents:

Ingester ring unstable across replicas. A misconfigured join_after or heartbeat_timeout causes ingesters to flap between JOINING and ACTIVE. Symptom is loki_ingester_lifecycler_ring_inconsistencies_total rising and the distributor returning 503 on every push.
Compactor missing. The binary tolerates its absence, the bucket does not. Symptom is loki_compactor_oldest_processed_age_seconds flat at zero while the bucket grows. The storage bill arrives first; the query latency second.
Distributor-to-ingester address drift after a Helm upgrade. A chart rename changes the ingester DNS (loki-write to lokiwrite). Distributors keep the old DNS in their config and lose contact. Symptom is the distributor logging connection refused and pushes returning 500.
Wrong replication_factor. A config that says RF=3 against an ingester pool of two replicas means every push waits for a third replica that does not exist. Symptom is distributor latency climbing and pushes failing with ingester not found.
Query-frontend cache misconfigured. A TTL of 0s makes the cache useless; a TTL of 24h returns stale results after a config change. Symptom is the same query returning different results across two consecutive Grafana clicks.
Ruler store wrong. The ruler reads its rules from a separate bucket (prod-loki-ruler). A typo in ruler.storage.s3.bucketnames means the ruler loads zero rules. Symptom is loki_ruler_rules_loaded flat at zero while the Alertmanager reports no Loki alerts.

How to troubleshoot it

The diagnostic order matters. Each step rules out one role:

Is every target /ready? /ready per target. A 404 points to a wrong -target flag in the unit file or to a Helm value that disables the role.
Is the ring healthy? /ingester/ring and /distributor/ring. An empty members list means the KV store (consul or memberlist) is not reachable between pods.
Are pushes arriving? loki_distributor_bytes_received_total and loki_ingester_chunks_created_total. A flat distributor counter is a client-side problem; a flat ingester counter is a wiring problem.
Are chunks flushing? loki_ingester_chunk_age_seconds and the local WAL checkpoint. Stale checkpoints point at object-store failures.
Is the bucket compacting? loki_compactor_oldest_processed_age_seconds. A flat counter for hours means the compactor is wedged.
Is the query path responsive? loki_querier_query_seconds and loki_request_duration_seconds{route="/loki/api/v1/query"}. p99 latency over five seconds almost always points at the bucket, not the querier.

Security implications

Microservices mode exposes more ports because each role has its own service. Three surfaces to lock down:

Per-role addresses become stable DNS names. Bind every role’s HTTP and gRPC listen address to a private interface. Treat the distributor port as service-internal; do not expose 3100 publicly.
mTLS between roles. The ring KV and the querier-to-query-frontend link are cluster-internal but should not be plaintext in a shared cluster. Use cert-manager or your platform’s mTLS mesh.
Bucket credentials. The ingester and compactor both write to the bucket. A leaked AWS key with s3:PutObject is a trace-data breach. Use scoped credentials per role and rotate them on the same cadence as the rest of the cloud credentials.

Performance implications

The cost profile differs by role:

Distributor. CPU-bound on decoding. Memory grows with the rate-limit queue.
Ingester. Memory-bound on the head block; disk-bound on the WAL.
Querier. CPU- and network-bound on chunk fetches.
Query-frontend. Memory-bound on the in-memory cache; CPU on query splitting.
Compactor. CPU- and network-bound during the compaction window.
Ruler. Memory-bound on the in-memory rule set per tenant.
Index-gateway. Network-bound on TSDB index file fetches.

The right scale order, when one role is the bottleneck:

Querier. Read load grows with the number of Grafana users. Add pods first when p99 query latency rises.
Distributor. Write load grows with the number of push clients. Add pods when CPU is saturated and loki_distributor_bytes_received_total is still rising.
Ingester. Only when the head-block memory ceiling is reached, because each new pod rebalances the ring and may cause a brief 503 window.
Index-gateway. When p99 query latency rises but the querier is healthy.
Compactor. Almost never. Singleton is the default and is sufficient for millions of marker files.
Ruler. When rule evaluation cost grows; this is per-tenant and rarely the bottleneck.

Production guidance

Start on simple-scalable mode. Move to microservices when one role is provably the bottleneck.
Pin the ingester WAL to a dedicated, fast disk. NVMe-backed local SSD is the right answer; network storage is the wrong one.
Run exactly one compactor by default; only enable sharded compaction when the bucket exceeds what a single compactor can process per cycle.
Front the querier with the query-frontend once the querier pool exceeds two replicas.
Use the same ring KV store address in every config. A divergence produces a cluster where distributors and ingesters see different ring memberships.

Verification

You should now be able to answer:

Name the seven Loki components and which are stateless, stateful, or mandatory in any deployment.
Which role do you scale first when query latency rises?
What does the query-frontend add when the querier pool exceeds two replicas?
Why does the compactor run as a singleton by default?
What is the difference between the distributor ring and the ingester ring?