Loki Upgrade Basics

A team is two minor versions behind on Loki. The new version promises a per-tenant query parallelism fix and a smaller index file. The team schedules the upgrade for a Wednesday afternoon. The cluster runs read-write during the cutover. After the roll, the distributor logs show failed to load schema config: unknown schema version v13. The cluster ingests nothing. Engineers spend the evening reverting, taking the API down for forty minutes while the previous version is reinstalled.

Loki upgrades are not a single action. They are three independent decisions that have to be sequenced: config compat, schema compat, and storage compat. The wrong order produces a cluster that cannot read its own data. The right order produces a cluster that does not notice the cutover.

What it is

A Loki upgrade is the process of moving a running cluster from one version to a newer version while preserving the ability to read all data written by the previous version. The upgrade is composed of three distinct concerns:

• Configuration compat. The YAML keys the cluster accepts. Keys renamed, removed, or restructured between versions. A 3.x config uses long-form keys (store: tsdb) that did not exist in 2.x.
• Schema compat. The on-disk format of the index and the chunk store. Loki 3.x ships with the v13 schema as the default; older clusters writing v11 or v12 still work, but new writes use the new schema.
• Storage compat. The object store layout and the index backend. 3.x defaults to the TSDB index and deprecates the boltdb-shipper index; a 2.x cluster using boltdb-shipper continues to read and write until the migration in 05-boltdb-to-tsdb runs.

Each is a separate decision. A configuration-only upgrade can be applied at the rolling restart. A schema upgrade requires a new entry in schema_config.configs and a window during which the old schema is still being written. A storage upgrade requires a tooling job that reads the old index and writes the new one.

Why a sysadmin cares

The Loki release cadence is roughly monthly. Standing still is not an option: bug fixes and security patches land in the supported series, and the gap between the running version and the supported version grows every cycle. The cluster that skips two minor versions pays the surcharge in a single larger upgrade window.

Three risks dominate the upgrade conversation:

1. Read regression. A version bump that changes the default schema or the index backend produces a cluster that writes the new format but reads the old format inconsistently. Queries against the gap fail with schema not found or index not found.
2. Write regression. A config key that was renamed in the new version causes the binary to start with defaults that silently change behaviour. The cluster starts but writes elsewhere.
3. Storage cost. A migration that was supposed to compact the index runs over the bucket and produces an S3 bill the capacity plan did not predict.

The cost of a botched upgrade is not the alert that fires. It is the two days of post-incident analysis while the team reconstructs what the cluster was actually doing before the restart.

How it works

   Pre-upgrade
       |
       v
  +---------+
  | Plan    |   read release notes; map breaking changes to your config
  +---------+
       |
       v
  +---------+
  | Backup  |   snapshot the bucket; export schema_config and limits_config
  +---------+
       |
       v
  +---------+
  | Stage   |   run the new version against a copy of the bucket;
  |         |   execute the migration; query and write
  +---------+
       |
       v
  +---------+
  | Canary  |   one pod at a fraction of traffic; watch metrics
  +---------+
       |
       v
  +---------+
  | Roll    |   rolling restart of every component in dependency order
  +---------+
       |
       v
  +---------+
  | Validate|   query a stream from every schema period; check metrics
  +---------+
       |
       v
  Production

The plan is the highest-leverage step. The release notes for the target version list every breaking change. The plan answers three questions for each one: does it apply to my config, does it apply to my schema, does it apply to my storage. The answer is one of three verbs: rename, migrate, or no-op.

How to configure it

The config produced by the upgrade is rarely a single file. The minimum cutover set is five.

# /etc/loki/config-write.yaml
# Schema config -- the v13 entry is added; the v11 entry stays.
# Loki reads the entry whose `from` date is the most recent before
# the write timestamp. The old v11 entry is needed for reads of
# data written before the cutover.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# /etc/loki/config-read.yaml
# Read target carries the same schema_config list. The list must
# match across write and read targets. A mismatch produces a read
# path that selects the wrong schema for the timestamp.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# /etc/loki/common.yaml
# The common block carries the storage backend. The bucket name
# does not change across the upgrade.
common:
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

# /etc/loki/runtime-config.yaml
# Tenant limits moved to a runtime config file in 3.x. The keys
# here override the global limits_config per tenant.
overrides:
  tenant-a:
    ingestion_rate_mb: 64
    retention_period: 2160h
  tenant-b:
    ingestion_rate_mb: 16
    retention_period: 720h

# /etc/loki/limits-config.yaml
# Global limits. The retention_period and ingestion_rate_mb are
# the two values that break production most often.
limits_config:
  retention_period: 2160h
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 48
  max_query_length: 30d
  max_query_parallelism: 32

How to validate it

Four commands that confirm the upgrade is intact and the new version is doing what the plan requires.

# READ-ONLY: confirm the binary loaded the schema config the
# plan specifies.
curl -s http://loki-write-0:3100/config | jq '.schema_config.configs[].schema'
# expected: ["v11", "v13"]
# If only one entry appears, the second was not loaded.

# READ-ONLY: confirm the version the running binary reports.
curl -s http://loki-write-0:3100/metrics | grep loki_build_info
# expected: loki_build_info{branch="...", version="3.x.y", ...} 1

# READ-ONLY: confirm the new schema is being written.
curl -s http://loki-write-0:3100/metrics | grep 'loki_index_request_duration_seconds_count'
# expected: the count for the new index prefix is non-zero within
# one chunk_idle_period of the upgrade.

# READ-ONLY: confirm the old schema is still being read.
curl -s http://loki-read-0:3100/metrics | grep 'loki_tsdb_index_request_duration_seconds'
# expected: a non-zero count for both the old and new index periods
# during the cutover window.

How it can fail

Six failure modes cover the most common production incidents tied to Loki upgrades.

1. Schema config typo in the new from date. A wrong date ('2026-09-31') causes the parser to reject the entire schema_config block. Symptom: the pod crashes with failed to parse schema config from. The fix is a corrected date and a pod restart.

2. Write and read targets disagree on the new schema entry. The write target gains the v13 entry; the read target does not. Symptom: new log lines are visible in the bucket but absent from queries. The loki_index_request_duration_seconds histogram shows the write target hitting the new prefix without corresponding reads.

3. Three minor versions jumped at once. A team upgrades from 2.9 to 3.4 in a single change. The release notes between 2.9 and 3.4 contain four breaking changes. The team caught three of them. Symptom: the fourth, a renamed configuration key, defaults to a value that doubles the index file size. The team discovers the growth a week later when the S3 bill arrives.

4. The new binary rejects an old CLI flag. loki -target=all was the documented way to run single-binary in 2.x. The 3.x binary uses -target=single for the same mode. Symptom: the process exits with unknown target: all. The fix is a flag rename.

5. The canary pod stays on the old version by mistake. A readiness probe passes on the old version because the API surface is unchanged. Symptom: the canary pod never migrates to the new version; the rollout stalls. The fix is a startup probe that checks the version string.

6. The migration job runs against the live bucket. A bot runs the boltdb-shipper to tsdb migration on the production bucket. The job rewrites every index file. The querier reads mid-migration and finds an incomplete index. Symptom: queries return partial results for the duration of the migration. The fix is a separate staging bucket for the first run.

How to troubleshoot it

The diagnostic order for an upgrade that does not behave as planned:

1. Did the binary accept the new config? Check the pod status and the startup log. kubectl logs loki-write-0 --previous shows the parser error, if any.
2. Is the new version actually running? curl /metrics | grep loki_build_info confirms the version. A readiness probe that passes on the old version is a common trap.
3. Is the new schema being written? The loki_index_request_duration_seconds_count metric on the write target should show traffic against the new prefix.
4. Is the new schema being read? The same metric on the read target should show reads. A write-only new prefix means the read target is still on the old schema config.
5. Is the migration job (if any) running? The loki_tsdb_index_completed_sections metric on the index-gateway shows the section count for the new index. A stuck count is the sympom of a migration that has failed silently.
6. Is the compactor keeping up? The loki_compactor_oldest_processed_age_seconds metric should stay within compaction_interval of the current time. A growing gap means the compactor is the bottleneck.

Security implications

The upgrade window is a privilege escalation window. The new binary uses the same credentials as the old one. If the credentials are rotated during the upgrade, the new binary starts with the old credentials and the rollback uses the new ones. The fix is to rotate credentials between upgrades, not during them.

The -auth_enabled=true flag is the easiest way to leave a Loki instance open during an upgrade. A multi-tenant Loki that was behind a reverse proxy for the previous version may now bind the distributor to all interfaces. The firewall rule on port 3100 must be in place before the rolling restart.

Performance implications

A Loki upgrade is a workload change. The new binary may have different flush behaviour, different query parallelism, or a different cache default. The performance baseline from the previous version is the only reliable reference. Capture the loki_ingester_chunks_flushed_total and loki_request_duration_seconds histograms before the upgrade and diff them after.

The largest performance delta in Loki 3.x is the TSDB index. The index file is smaller and the lookup is faster, but the compactor runs more often. The first week after the upgrade shows elevated compactor CPU. Plan for the spike.

Production guidance

• Read the release notes for every version between the running version and the target. The breaking changes are listed. Diff your config against the lists.
• Pin the upgrade to a single minor version jump. Two minor versions is the largest safe jump in a single change window.
• Run loki -verify-config against the new binary before every restart.
• Capture the resolved config with -print-config-stderr before the canary. Diff against the file you intended to load.
• Write a synthetic log line to the cluster every minute. The validation step in the next lesson uses it.
• Schedule the upgrade for a low-traffic window. The first signal that the upgrade is good is the synthetic line appearing in Grafana.

Verification

You should now be able to answer:

• What are the three categories of Loki upgrade risk, and why does each require a different migration plan?
• Why must the schema_config.configs list be append-only across an upgrade, and what is the consequence of removing an old entry?
• Why is the read target rolled after the write target, not before?
• What does loki -verify-config validate that loki -print-config-stderr does not?
• Why is the canary pod checked against the version string, not just the readiness probe?

Quiz