Loki Config Migration

A team upgrades Loki from 2.9 to 3.4. The team prints the new config with -print-config-stderr and diffs it against the file on disk. The diff shows a long list of changes. The team reads the diff, accepts the changes, and applies the new config. The cluster restarts. The distributor logs show unknown storage backend: cassandra. The team searches the release notes for the error. The notes do not mention the cassandra backend. The team discovers the cassandra backend was removed in 3.0 and the migration tool does not flag the removal because the key is still accepted as an unknown value.

Config migration is not a tool. It is a discipline. The tool produces a diff. The diff is the input. The discipline is the human review of the diff against the release notes, the cluster configuration, and the per-tenant overrides.

What it is

A config migration is the process of updating a loki.yaml file to the syntax expected by the target version. The migration is necessary because every Loki release renames, removes, or restructures some YAML keys. A config that was valid in 2.9 may parse in 3.4 but produce a cluster that behaves differently.

The categories of config change between Loki versions:

• Renames. A key was renamed. The old key is no longer recognised; the new key takes the same value. chunk_store_config was renamed to common.storage_backend and the per-store configuration was moved under common.{s3,gcs,azure,filesystem}.
• Restructures. A block was reorganised. The values are the same; the nesting changed. auth_enabled moved from per-tenant to a top-level key that controls the global behaviour.
• Removals. A key was removed entirely. The value is no longer accepted. The cassandra backend was removed in 3.0. The config parser logs a warning and continues with the default.
• New defaults. A key was introduced with a default that differs from the previous behaviour. The config does not need to change but the cluster behaves differently. The query_range.split_queries_by_interval default changed from 0s to 1h in 3.x.

The Loki binary includes a migrate subcommand that reads a config file in the syntax of one version and writes the equivalent config in the syntax of another. The tool is helpful for the rename and restructure categories. It is silent on the removals and the new defaults.

Why a sysadmin cares

The config migration is the most common upgrade failure mode. Three production scenarios apply:

1. A renamed key is silently defaulting. The team did not notice the rename. The cluster runs with the default value. The default is different from what the team intended. The cluster behaves differently from the previous version.
2. A removed key is silently ignored. The cassandra backend is no longer accepted. The cluster logs a warning and continues with the default backend. The cluster writes to the default backend instead of the configured one. The log volume goes to the wrong place.
3. A new default is silently accepted. The query_range.split_queries_by_interval default changed. The cluster now splits queries differently. The query latency changes. The new behaviour is not wrong, but it is not what the team tuned.

The cost of a botched config migration is the cluster that runs but produces unexpected results. The team discovers the regression when the S3 bill arrives, the query latency spikes, or the alert fires.

How to configure it

The migration is a four-step process: diff, review, apply, verify. The output of the migrate tool is the diff. The diff is the input to the review.

# /etc/loki/loki.yaml
# Loki 3.x single-binary config. The auth_enabled key is now
# top-level; the storage backend is under common; the runtime
# config file is loaded via the runtime_config flag.
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-0.loki-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 2160h
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 48
  max_query_length: 30d
  max_query_parallelism: 32
  reject_old_samples: true
  reject_old_samples_max_age: 168h

# The runtime config file is loaded via the CLI flag, not the
# YAML block. The path is mounted from a ConfigMap.
# loki -config.file=/etc/loki/loki.yaml -runtime-config.file=/etc/loki/runtime-config.yaml

# /etc/loki/runtime-config.yaml
# The per-tenant overrides moved to a runtime config file in 3.x.
# The overrides are pushed to Loki over HTTP; the file is the
# source of truth.
overrides:
  tenant-a:
    ingestion_rate_mb: 64
    retention_period: 2160h
  tenant-b:
    ingestion_rate_mb: 16
    retention_period: 720h

# /etc/loki/config-write.yaml
# The write target on simple-scalable. The block structure is the
# same as the single-binary; the per-target sections are loaded
# based on the -target flag.
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-write-0.loki-write-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 2160h
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 48

distributor:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500

ingester:
  chunk_idle_period: 30m
  max_chunk_age: 2h
  wal:
    enabled: true
    dir: /var/lib/loki/wal

How to validate it

Five commands that confirm the config migration is complete and the cluster is running on the target version.

# READ-ONLY: produce a diff between the current config and the
# target version. The migrate tool reads the file in the syntax
# of the source version and writes the equivalent in the syntax
# of the target version.
loki migrate --source-version=2.9 --target-version=3.4 \
  --input=/etc/loki/loki.yaml.previous \
  --output=/etc/loki/loki.yaml.migrated
diff -u /etc/loki/loki.yaml.previous /etc/loki/loki.yaml.migrated
# expected: the diff shows the renames, restructures, and new
# defaults. The review is the operator reading the diff against
# the release notes.

# READ-ONLY: validate the parsed structure.
loki -config.file=/etc/loki/loki.yaml -verify-config
# expected: "config is valid" on stdout; exit code 0.
# A non-zero exit means the YAML parser rejected the file.

# READ-ONLY: resolve the config and print it.
loki -config.file=/etc/loki/loki.yaml -print-config-stderr 2>&1 > /tmp/config.log
diff /tmp/config.log /etc/loki/loki.yaml
# expected: the printed config should match the file intended to
# load, with defaults filled in for unset keys. A large diff
# means the operator forgot to set a key.

# READ-ONLY: confirm the runtime config file is loaded.
curl -s http://loki-0:3100/config | jq '.limits_config'
# expected: the runtime overrides for the tenants are visible in
# the limits_config section. The runtime config is merged into
# the global limits_config at query time.

# READ-ONLY: confirm the canary metric is reporting success.
curl -s http://loki-canary-0:3100/metrics | grep 'loki_canary_last_success'
# expected: a timestamp within the last 60 seconds.

How it can fail

Six failure modes cover the most common production incidents tied to a Loki config migration.

1. A renamed key was not updated. The cluster runs with the default value. Symptom: the cluster behaves differently from the previous version. The loki_ingester_chunks_flushed_total metric shows a different flush rate than expected.

2. A removed key is silently ignored. The cassandra backend was removed in 3.0. The config still has the cassandra block. The binary logs a warning and uses the default. Symptom: the cluster writes to the default backend. The chunks end up in the wrong bucket.

3. A new default changes the query path. The query_range.split_queries_by_interval default changed from 0s to 1h. Symptom: queries that used to be split at every second are now split at every hour. The query latency changes. The loki_request_duration_seconds histogram shows a different profile.

4. The runtime config file is not loaded. The CLI flag -runtime-config.file is missing. The per-tenant overrides are not applied. Symptom: a tenant that should have a 64 MB ingestion rate is limited to the global 32 MB. The loki_discarded_samples_total metric shows the tenant exceeding the limit.

5. The config was migrated but the runtime config was not. The per-tenant overrides file is in the 2.x syntax. The 3.x binary does not recognise the old keys. Symptom: the overrides are silently ignored. The tenant limits revert to the global defaults.

6. The config was applied without the canary. The cluster is rolled without loki-canary running. Symptom: a regression in the read path is not detected until a user reports an empty query result. The canary is the first signal of a regression.

How to troubleshoot it

The diagnostic order for a config migration that does not behave as planned:

1. Did the binary accept the new config? loki -verify-config exits with 0 for a valid file. The startup log shows the parse result.
2. What did the binary resolve? loki -print-config-stderr shows the resolved config. Diff against the file intended to load. A large diff means a key was not set.
3. Is the runtime config loaded? curl /config | jq .limits_config shows the merged limits. The runtime overrides must be visible.
4. Is the cluster using the right backend? The loki_ingester_chunks_flushed_total metric, broken down by path, shows the per-tenant path. The path must match the configured bucket.
5. Is the canary reporting success? The loki_canary_last_success metric should be within the last 60 seconds. A stale value means the read path is failing.
6. Is the compactor keeping up? The loki_compactor_oldest_processed_age_seconds metric should stay close to the current time.

Security implications

The config migration is a privilege escalation window. The new config may have different default values for auth_enabled, server.http_listen_address, or the per-tenant paths. A single-binary Loki that was behind a reverse proxy in 2.x may now bind the distributor to all interfaces.

The CLI flag for the runtime config file is a privileged path. The file contains per-tenant overrides that may include retention periods or ingestion rates. The file must be mounted from a secret manager and have read-only access for the Loki service account.

Performance implications

A config migration is a workload change. The new defaults may have different flush behaviour, different query parallelism, or a different cache size. The performance baseline from the previous version is the only reliable reference. Capture the loki_ingester_chunks_flushed_total and loki_request_duration_seconds histograms before the migration and diff them after.

The runtime config file is loaded at query time. A large runtime config file (thousands of tenants) increases the query latency per request. The loki_request_duration_seconds histogram on /config shows the load time. A spike in the histogram is the symptom of a runtime config file that is too large.

Production guidance

• Diff the resolved config with -print-config-stderr against the file intended to load. The diff is the input to the review.
• Use the loki migrate tool as a starting point, not as the final config. The tool handles renames and restructures. The review handles removals and new defaults.
• Read the release notes for every version between the running version and the target. The breaking changes are listed.
• Copy the per-tenant overrides from the 2.x config into the 3.x runtime config file. The overrides do not migrate automatically.
• Run loki-canary continuously. The canary is the first signal of a config regression.
• Pin loki_version_verified in the lesson frontmatter to the version that introduced the config change.

Verification

You should now be able to answer:

• What is the difference between a rename, a restructure, a removal, and a new default in a Loki config migration?
• Why does the loki migrate tool not catch a removal or a new default?
• What is the runtime config file, and why must it be loaded via the CLI flag?
• Why is the per-tenant override the most common migration failure?
• What is the difference between loki -verify-config and loki -print-config-stderr?

Quiz