Loki Configuration

The Loki binary accepts a single YAML file. The file is the authoritative declaration of what the process is. The -target flag selects which sections of the file are loaded into running components. A single-binary config is the union of every per-target section. A microservices config has each section in isolation. A broken section does not stop the file from parsing — Loki starts, the affected subsystem fails, and the on-call engineer finds out from an alert that fired an hour later.

This lesson is the file map. Every top-level section in loki.yaml, what it controls, and the failure mode of a wrong value.

What it is

Loki configuration is a YAML file consumed by a Go binary. The binary parses the file into a struct that is read by each component on startup. Every section is optional. The default values are in the source code (pkg/loki/modules.go). A config with only the auth_enabled key starts a Loki instance that uses the filesystem backend, in-memory ring, and the v13 schema.

The seven sections that every production config sets:

• auth_enabled — top-level boolean.
• server — HTTP and gRPC listener addresses.
• common — shared by every component. Ring, instance address, path prefix, storage backend.
• schema_config — versioned schema declarations that map a date range to an index store, an object store, and a schema version.
• limits_config — ingestion limits, retention, query limits. Can be overridden per tenant.
• ruler — alerting rule storage and evaluation.
• Per-component blocks (ingester, querier, query_range, distributor, compactor, frontend_worker, index_gateway, cache_loader).

Why a sysadmin cares

A sysadmin cares because every value in this file is a knob that either helps the next incident or hurts it. Three knobs matter more than the rest:

• schema_config.configs[].schema is irreversible. Changing the schema version requires a migration window during which the old schema is still being written and the new schema is being written in parallel. A wrong schema version produces a cluster that cannot read its own data.
• common.storage_backend decides where every byte of every chunk lives. A wrong value means the cluster is silently writing to a different bucket than the dashboard is reading from.
• limits_config.retention_period is a global setting. A wrong value is a compliance event. Logs older than the value are deleted by the compactor on its next sweep.

The lesson in 04-loki-retention-config walks through retention in detail.

How it works

  loki.yaml
    |
    +--- auth_enabled           (top-level, no nesting)
    |
    +--- server:                (top-level, no nesting)
    |       http_listen_port: 3100
    |       grpc_listen_port: 9095
    |
    +--- common:                (shared by every component)
    |       ring: { kvstore: ... }
    |       instance_addr: ...
    |       path_prefix: ...
    |       storage_backend: s3
    |
    +--- schema_config:
    |       configs:
    |         - from: '2024-01-01'
    |           store: tsdb
    |           object_store: s3
    |           schema: v13
    |
    +--- limits_config:         (global; can be overridden per tenant)
    |       retention_period: 2160h
    |       ingestion_rate_mb: 32
    |       max_query_length: 30d
    |
    +--- ingester:              (only loaded by -target=ingester,
    |                            -target=all, -target=write)
    |       chunk_idle_period: 30m
    |       max_chunk_age: 2h
    |
    +--- querier:               (only loaded by -target=querier,
    |                            -target=all, -target=read)
    |       query_timeout: 60s
    |
    +--- query_range:           (query-frontend only)
    |       split_queries_by_interval: 24h
    |       results_cache:
    |         cache:
    |           embedded_cache:
    |             enabled: true
    |
    +--- compactor:             (compactor only)
    |       working_directory: /var/lib/loki/compactor
    |       retention_enabled: true
    |
    +--- ruler:                 (ruler only)
    |       storage:
    |         type: s3
    |         s3:
    |           s3: s3://eu-west-1
    |           bucketnames: prod-loki-ruler

The binary parses the file and then loads only the sections that match the active -target. A microservices ingester config has only common, server, schema_config, limits_config, and ingester. A query-frontend config has common, server, and query_range. A single-binary config has all of them.

How to configure it

A production config for a simple-scalable write target. Every section is annotated.

# /etc/loki/config-write.yaml
# Mode: simple scalable, write target. -target=write on the CLI.

# Auth is multi-tenant when true, single-tenant when false.
# This Loki is single-tenant; auth lives at the gateway.
auth_enabled: false

server:
  http_listen_port: 3100    # Promtail, Alloy, and the read path push here.
  grpc_listen_port: 9095    # Used by distributor to fan out to ingesters.
  log_level: info

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-write-0.loki-write-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

# schema_config defines the chain of schemas Loki has used.
# Each entry says: from this date forward, use this index store
# (tsdb), this object store (s3), this schema version (v13).
# The list is append-only. Removing a period breaks reads of data
# written under that period.
schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# Global limits. Overridable per tenant via the runtime config file.
limits_config:
  retention_period: 2160h      # 90 days
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 48
  max_query_length: 30d
  max_query_parallelism: 32
  reject_old_samples: true
  reject_old_samples_max_age: 168h   # 7 days

# Ingester controls how ingested log lines become chunks.
ingester:
  chunk_idle_period: 30m       # flush after 30m of no writes
  max_chunk_age: 2h            # flush at most every 2h
  wal:
    enabled: true              # survives a restart
    dir: /var/lib/loki/wal

# Distributor fans pushes out to ingesters.
distributor:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500

# /etc/loki/config-read.yaml
# Mode: simple scalable, read target. -target=read on the CLI.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-read-0.loki-read-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# Querier does the heavy lifting of fetching chunks from the
# object store and applying the query expression.
querier:
  query_timeout: 60s
  query_ingester_within: 30m   # also ask live ingesters

# query_range is the query-frontend section. It splits long
# queries into smaller intervals and caches results.
query_range:
  split_queries_by_interval: 24h
  parallelise_sharded_queries: true
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 500

# /etc/loki/config-backend.yaml
# Mode: simple scalable, backend target. -target=backend on the CLI.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-backend-0.loki-backend-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# Compactor applies retention and merges index files.
# retention_enabled must be true to delete chunks older than
# limits_config.retention_period.
compactor:
  working_directory: /var/lib/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: s3

# index_gateway serves index files for queries.
index_gateway:
  mode: simple

# /etc/loki/config-ruler.yaml
# Mode: simple scalable, ruler target. -target=ruler on the CLI.

auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-ruler-0.loki-ruler-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3

ruler:
  storage:
    type: s3
    s3:
      s3: s3://s3.eu-west-1.amazonaws.com
      bucketnames: prod-loki-ruler
      region: eu-west-1
  alertmanager_url: http://alertmanager.monitoring.svc.cluster.local:9093
  rule_path: /var/lib/loki/rules
  evaluation_interval: 1m

The four files above are the canonical simple-scalable shape. The Helm chart and the official docker-compose example ship them in this form.

How to validate it

Three commands that confirm the config is syntactically valid and that the binary accepted it.

# READ-ONLY: print the resolved config to stderr.
# Sections not relevant to the target are filtered out.
loki -config.file=/etc/loki/config-write.yaml -target=write -print-config-stderr 2>&1 \
  | grep -E '^(auth_enabled|server|common|schema_config|limits_config|ingester|distributor):' \
  | head
# expected: the section headers that match the target. Missing
# sections means missing YAML keys. Extra sections in the output
# means a typo was loaded into the wrong struct.

# READ-ONLY: validate the config without starting the binary.
loki -config.file=/etc/loki/config-write.yaml -target=write -verify-config
# expected: "config is valid" on stdout; exit code 0.

# READ-ONLY: confirm the schema config loaded as expected.
curl -s http://localhost:3100/config | jq '.schema_config'
# expected: the configs[] list as the running binary sees it.
# If a config is missing from the list, the binary did not load it.

# READ-ONLY: confirm the storage backend is the one expected.
curl -s http://localhost:3100/config | jq '.common.storage_backend'
# expected: "s3" for a production deployment. "filesystem" for
# staging only.

How it can fail

Six failure modes cover the most common production incidents tied to Loki configuration. Each is mapped to a symptom.

1. Schema config typo in the from date. A typo ('2024-13-01' instead of '2024-01-01') causes Loki to log failed to parse schema config from at startup. Symptom: the process exits with code 1 before binding any port. The fix is a corrected date.

2. Write and read targets disagree on bucketnames. The write target writes to prod-loki-chunks. The read target was reconfigured to point to prod-loki-chunks-v2. Symptom: new log lines are visible in the S3 bucket but absent from Grafana queries. The metric loki_objstore_request_duration_seconds shows the read target hitting chunks-v2 with no objects.

3. limits_config.retention_period shorter than compliance requires. Compliance requires 365 days. The config says 744h (31 days). The compactor runs at its next interval and deletes everything older than 31 days. Symptom: the loki_compactor_oldest_processed_age_seconds metric drops to the retention value. The operator discovers the deletion only when a compliance audit requests a year-old log and finds nothing.

4. Compactor not singleton. Two compactor pods started simultaneously. Symptom: the compactor logs show lock already held and compactor is not the leader. Retention does not advance.

5. Distributor ring pointed at a stale consul host. The consul backend was migrated; the consul.host value still points at the old address. Symptom: distributors log connection refused on the ring registration path; the loki_distributor_ingester_clients gauge drops to zero; every push returns 500.

6. query_range.split_queries_by_interval set to 0. The query-frontend reads 0 as “split by zero seconds” and refuses to start. Symptom: the read target crashes with the error invalid split interval. The fix is a non-zero value (24h, 12h, 6h are common choices).

How to troubleshoot it

The diagnostic order for a Loki config problem:

1. Did the binary accept the file? Check the systemd unit or the pod status. If the process exited at startup, the error is in the journal. journalctl -u loki -n 100 or kubectl logs loki-write-0 --previous shows the exact line.
2. Are the section keys correctly named? A typo in chunk_idle_period produces a silent default change. Run loki -print-config-stderr and diff against the file you intended to load.
3. Are the limits being enforced? The loki_discarded_samples_total metric with the reason label shows the rate-limit rejections. The loki_request_duration_seconds histogram on /loki/api/v1/push shows the push latency.
4. Is the compactor running and the lock held? The loki_compactor_oldest_processed_age_seconds metric shows the age of the oldest compaction processed. If the value stops advancing, the compactor is stuck.
5. Is the schema version what you expected? The loki_index_request_duration_seconds histogram shows the read path’s index lookups. A spike in latency can mean the schema version differs between writers and readers.

Security implications

Every section that takes a credential (s3.access_key_id, consul.host + ACL, ruler.storage.s3.*) is a credential boundary. The credential files should be mounted from a secret manager, not committed to the repository. The auth_enabled key controls whether the Loki API expects an X-Scope-OrgID header on every request; turning it on without a reverse proxy in front that issues tenant IDs produces an unauthenticated multi-tenant cluster.

The server.http_listen_address defaults to all interfaces (0.0.0.0). On a single-host installation, this binds the distributor and querier to the public network. A firewall rule on port 3100 must restrict the source to known push clients (Promtail, Alloy, application agents). The production guidance in 01-loki-auth walks through the reverse-proxy path.

Performance implications

The performance ceiling of a Loki config is set by:

• ingester.chunk_idle_period and ingester.max_chunk_age. Short values flush more often and reduce query latency; long values reduce object-store write throughput.
• query_range.split_queries_by_interval. Short values produce more sub-queries; long values produce fewer, larger ones. The tradeoff is between parallelism and overhead.
• query_range.results_cache.cache.embedded_cache.max_size_mb. More cache memory means more query reuse; less means more cache misses.
• limits_config.max_query_parallelism. The cap on per-query parallelism. A value that is too high lets one user exhaust the querier fleet.

The lesson in 06-loki-perf-troubleshooting covers the runtime diagnostics.

Production guidance

• Validate the config with loki -verify-config before every restart.
• Keep schema_config.configs append-only. Removing an entry breaks reads of data written under that schema.
• Pin loki_version_verified in the lesson frontmatter to the schema version you tested. Loki 3.x ships with the v13 schema as the default.
• Store schema_config, limits_config, and the per-target sections in version control. Diff every change in code review.
• Rotate the s3.access_key_id and secret_access_key on the same cadence as the rest of the cloud credentials.

Verification

You should now be able to answer:

• Which section of loki.yaml controls the index version, and why is the section append-only?
• Which section controls ingestion limits and retention, and how does the per-tenant override path work?
• Which section controls the chunk flush cadence, and what is the tradeoff between chunk_idle_period and max_chunk_age?
• How does loki -verify-config differ from loki -print-config-stderr?
• Why does a typo in common.ring.kvstore.store only fail at first use rather than at parse time?

Quiz