Loki Installation Modes

A team deploys Loki in single-binary mode at 50 GB/day. Six months later they are at 800 GB/day and the process OOMs every Tuesday night at 02:00 when the nightly log sweep runs. They have no replica. They have no rolling restart. They have no metrics path that shows why the process is OOMing. They cannot redeploy without losing the in-flight ingester chunks. Three months of incident-driven work go into a migration they should have planned on day one.

This is the cost of choosing a Loki deployment mode without reading the second-year consequences. The mode you pick on day one decides how you will spend the next three years of operational budget.

What it is

Loki 3.x ships in three deployment modes. They are the same code, configured differently. The binary accepts a -target flag that turns one process into one of: all, read, write, backend, ruler, ingester, querier, query-frontend, index-gateway, compactor, distributor, or any single component name. The mode is not a separate binary. It is a flag and a config file.

The three modes are:

• Single binary. One process running the -target=all target. All components in one address space. One config file.
• Simple scalable. A split between read-path components (-target=read) and write-path components (-target=write), plus a separate -target=backend process for the compactor and index-gateway. Three logical deployments, one config file per target. The community Helm chart and the Docker Compose examples use this shape.
• Microservices. Each component runs as its own deployment, scales independently, has its own replicas, and has its own config. Twelve or more Kubernetes Deployments per cluster. This is what Grafana Labs runs internally.

The mode you pick decides your scaling ceiling, your failure isolation, your upgrade choreography, and how many config files you have to keep in version control.

Why a sysadmin cares

A sysadmin cares because every deployment mode has a different blast radius when it breaks:

• Single binary: blast radius is the host. The whole stack is down. Restart loses the in-memory ingester chunks unless WAL is on.
• Simple scalable: blast radius is split. The read path can be degraded without taking ingestion. The write path losing ingesters loses in-flight chunks but not historical data.
• Microservices: blast radius is per component. An ingester fleet can lose one replica without losing data. A query-frontend crash does not affect ingestion.

The mode also decides the operational headcount. Single binary can be run by one person. Simple scalable needs two — one for the write path, one for the read path, both coordinating compactor and index-gateway upgrades. Microservices needs an SRE-shaped team that owns a Kubernetes platform, a metrics stack, and a rollout choreography that respects per-component version skew.

How it works

  +-------------------------+    +-------------------------+
  |       Single binary     |    |    Simple scalable      |
  |                         |    |                         |
  |   one process, one      |    |   -target=read          |
  |   config file, all      |    |   -target=write         |
  |   components inside     |    |   -target=backend        |
  |                         |    |                         |
  |   scrapes / push via    |    |   distributor, ingester,|
  |   embedded distributor  |    |   query-frontend,       |
  |                         |    |   querier, index-gateway|
  |                         |    |   compactor             |
  +-------------------------+    +-------------------------+

  +-----------------------------------------------------+
  |                   Microservices                      |
  |                                                      |
  |   distributor (>=2)        query-frontend (>=2)      |
  |   ingester  (N replicas)   querier      (N replicas) |
  |   index-gateway (>=2)      ruler        (>=1)        |
  |   compactor  (exactly 1)   cache-loader (>=1)        |
  |                                                      |
  |   each component: own deployment, own config,         |
  |   own metrics, own HPA, own version skew window      |
  +-----------------------------------------------------+

The common config block is shared by every target. The per-target config block (ingester, querier, etc.) is filtered to the component that consumes it. A single-binary config is the union of every per-target block. A microservices config has each block in isolation.

How to configure it

The choice of mode is made by the -target flag. Everything else is config. Below is the smallest viable config for each mode, with annotations.

Single binary

# /etc/loki/config.yaml
# Mode: single binary. -target=all is implicit if -target is unset.
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: inmemory         # single process, no ring needed
  instance_addr: 127.0.0.1
  path_prefix: /var/lib/loki
  storage_backend: filesystem
  filesystem:
    directory: /var/lib/loki/chunks

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 744h       # 31 days
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 24

# READ-ONLY: confirm the binary recognises the mode
loki -config.file=/etc/loki/config.yaml -target=all -print-config-stderr 2>&1 | head -1
# expected: msg="starting" target=all

Simple scalable (Docker Compose / Helm)

The Helm chart and the official docker-compose example both ship three targets: read, write, backend. The config is split into per-target files. The common block is identical between targets. Each per-target block differs.

# /etc/loki/config-write.yaml
# Mode: simple scalable, write target.
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul:8500
  instance_addr: loki-write-0.loki-write-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 2160h      # 90 days
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 48

# /etc/loki/config-read.yaml
# Mode: simple scalable, read target.
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul:8500
  instance_addr: loki-read-0.loki-read-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 500

# No ingester block. Read target does not ingest.

Microservices

Each component has its own config file and its own deployment. The config file for an ingester contains only the ingester and common blocks. The config file for a compactor contains only the compactor and common blocks. The Helm chart and the production/ksonnet directory under the Loki repository are the canonical templates.

How to validate it

Three commands, one per mode, that confirm the binary started in the expected mode and that the components are wired correctly.

# READ-ONLY: confirm the binary resolved the target it was given.
loki -config.file=/etc/loki/config.yaml -target=all -version
# expected: Loki version 3.x (date, branch, commit)

curl -s http://localhost:3100/ready | jq .
# expected: { "ingester": "ready", "querier": "ready", ... }

curl -s http://localhost:3100/services | jq .
# expected: includes the components present in the running target.
# single-binary: distributor, ingester, querier, query-frontend, compactor, ...
# read target: querier, query-frontend, index-gateway, ruler.
# write target: distributor, ingester.

# READ-ONLY: component membership is visible in the /services endpoint.
curl -s http://loki-write:3100/services | jq -r '.services[]'
# expected on a simple-scalable write target:
# distributor
# ingester

The /services endpoint is the first diagnostic. If the running component list does not match the mode you intended, you are looking at the wrong config file.

How it can fail

Five failure modes appear repeatedly across Loki deployments. Each is mapped to a symptom the operator can recognise.

1. Wrong target flag on the binary. The systemd unit passes -target=ingester instead of -target=all on a single-binary host. Symptom: the service is “up” (port bound, metrics endpoint serving) but no logs arrive. The /services endpoint shows only ingester, not distributor. The distributor listener on 3100 is absent. Result: silent ingestion outage.

2. Single-binary OOM on ingest spike. Single binary has no per-component memory isolation. A traffic spike hits the ingester’s chunk cache, the queryer’s results cache, and the ruler’s rules state simultaneously. The process is OOM-killed. Symptom: journalctl -u loki shows the kernel oom-killer event followed by a clean restart. In-flight chunks for the last max_chunk_age are lost unless WAL is enabled.

3. Simple-scalable ring split. The consul backend becomes unreachable from the write-path replicas but reachable from the read-path replicas. The write path drops out of the ring. New ingesters cannot register. Symptom: distributor starts rejecting pushes with 500 instead of 400. The loki_ingester_ring_members metric drops to zero for the write path while the read path is still healthy.

4. Microservices compactor not singleton. Two compactor pods running simultaneously after a Helm rollback that did not wait for the old pod to terminate. Symptom: the compactor logs show another compactor is holding the lock and refuses to run. Retention stops advancing until the duplicate is removed.

5. Version skew across components. A rolling upgrade of the querier to a newer Loki version while ingesters stay on the old version introduces a protobuf wire-format mismatch. Symptom: loki_querier_query_failed_total spikes with the reason rpc error: code = Unimplemented desc = unknown method. Reads degrade while writes continue.

How to troubleshoot it

The diagnostic order is the same regardless of mode:

1. Is the binary up? systemctl status loki or kubectl get pods -n loki. If the process is not running, the mode is irrelevant.
2. Which target is it running? loki -version does not show the target. curl /services does. If the component list is wrong, the -target flag is wrong.
3. Is the ring healthy? curl /ring. On a healthy ring every replica appears in JOINED state. If replicas appear in PENDING or LEAVING, the ring KV store is the suspect.
4. Is the object store reachable? The metrics loki_objstore_request_duration_seconds and the error counter loki_objstore_request_errors_total show the latency and failure rate per request type. A spike in 5xx from the object store affects reads and compactor work.
5. Are the rate limits firing? The metrics loki_distributor_lines_rejected_total and loki_discarded_samples_total{reason="rate_limit"} show the shape of the rejection. The application note in 06-loki-rate-limited walks through the per-reason breakdown.

Security implications

Each mode exposes the same HTTP surface for the components it runs. In single-binary, the operator has one listener to lock down. In simple-scalable, the read and write listeners are on different hosts and can have different network policies. In microservices, each component has its own listener and the operator must lock down twelve of them.

In every mode, the distributor and querier are the only components that need to be reachable from the application fleet. The ingester, compactor, index-gateway, and ruler should be reachable only from inside the Loki namespace. A common production mistake is exposing the compactor to the public internet because it is a “Loki endpoint” — the compactor has no authentication, no rate limit, and the bucket-level admin API. Lock it down with a NetworkPolicy.

The authentication chain (auth_enabled) is mode-independent. The operator must still issue a tenant ID and an auth proxy, or accept that the Loki instance is single-tenant. The lesson in 02-loki-tenant-isolation covers the multi-tenant path.

Performance implications

The performance ceiling of each mode is different:

• Single binary. CPU and memory are shared across all components. A single 8 vCPU / 32 GiB host will OOM around 50-100 GB/day depending on label cardinality. There is no path to scale beyond one host.
• Simple scalable. The write path can scale vertically to a single fat host or horizontally with three or more write replicas. The read path scales horizontally. 100 GB/day to 500 GB/day fits comfortably on commodity hosts.
• Microservices. Each component scales independently. The compactor needs one replica (it is a singleton). The distributor and query-frontend are stateless. The ingester and querier are the bulk of the resource budget. 1 TB/day and beyond is the realistic territory.

The trap: choosing simple scalable because “we might grow” and finding that the read path needs a separate autoscaler and the write path needs WAL and the compactor needs its own state and the whole migration becomes a quarter of work. Choose the smallest mode that fits the 12-month projection, not the 36-month one.

Production guidance

• Pick the mode at design time, not at first incident.
• Validate the mode before writing the first log line. The /services endpoint is the source of truth.
• Plan the upgrade to the next mode before you need it. Migrating from single binary to simple scalable while under load is how outages start.
• Monitor loki_target_info for every component. A missing component in the metric set means a missing component in the process.
• Document the mode in the runbook. The on-call engineer at 03:00 should not need to read config files to know which mode is running.

Verification

You should now be able to answer:

• Which Loki deployment mode fits a 20 GB/day staging cluster? Which fits a 500 GB/day production cluster?
• What does the /services endpoint show for a simple-scalable write target? What does it show for the read target?
• What is the failure mode of a single binary under a memory spike, and what Loki feature mitigates it?
• How do you confirm a Loki process is running in the target you intended?
• What is the operational cost difference between simple scalable and microservices in terms of headcount and on-call burden?

Quiz