Loki Schema Config

A team operates a Loki cluster on the v11 schema. The upgrade target is v13. The team edits schema_config.configs, removes the v11 entry, and replaces it with v13. The pod restarts. The new binary reads the config and starts writing with the v13 schema. The querier receives a query for a log line from six months ago. The querier selects the v13 schema because that is the only entry in the list. The index lookup finds nothing. The query returns no results. Six months of data is invisible to the dashboard.

The team recovers by reverting the schema_config change. The v11 entry is restored. The data is visible again. The lesson is the rule that the schema_config block is append-only: the new schema is added, the old schema stays, and Loki picks the schema per timestamp.

What it is

The schema_config block in loki.yaml is the versioned declaration of how Loki stores its index and what object store holds the chunks. It is a list of PeriodConfig entries. Each entry is a date range that says: from this date forward, use this index store, this object store, and this schema version.

  schema_config.configs
       |
       +--- [0]  from: '2024-01-01'  store: boltdb-shipper  schema: v11
       |                                      object_store: s3
       |
       +--- [1]  from: '2026-09-01'  store: tsdb            schema: v13
       |                                      object_store: s3
       |
       v
  Loki selects the entry whose `from` is the most recent before
  the timestamp of the write or the query.

The list is sorted by from. The active schema for a given timestamp is the most recent entry whose from is before that timestamp. Loki keeps every entry because every entry may be the active one for some historical timestamp.

Three versions matter for a Loki 3.x upgrade:

• v11. The original schema for the boltdb-shipper index. The index file is per-day, per-tenant, per-stream. The chunk format is the same as v12 and v13.
• v12. Daily index files were kept. The chunk format is unchanged. The migration from v11 to v12 is a no-op in the sense that v11 chunks are still readable; the v12 entry was introduced to claim the schema at the time of a future migration.
• v13. The schema paired with the TSDB index. The index file is per-day, per-tenant, in a single TSDB file. The migration from v12 to v13 is the moment Loki switches from the older boltdb-shipper to the newer TSDB index.

The version bump is not arbitrary. Each version is a code-level contract inside Loki. The binary parses the schema string and looks it up in a registry. An unknown version fails the parse. A known version declares the on-disk format and the index backend.

Why a sysadmin cares

The schema version is the contract between the write path and the read path. A wrong version produces a cluster that writes one format and reads another. Three production scenarios apply:

1. The read path cannot serve historical data. The schema_config list was reduced. The active schema for an old timestamp is the wrong one. The data is in the bucket; the index is missing. Queries return empty.
2. The write path produces chunks the read path cannot decode. A schema version was added but the read target was not rolled. The write target uses the new schema; the read target still uses the old. Queries for recent timestamps fall back to the old schema; the old index has no entries for the new writes.
3. The chunk format and the index format are out of sync. A v13 entry with a boltdb-shipper store is not a valid combination. The binary logs a warning and uses the closest matching format. The cluster runs but produces unexpected results.

The cost of a wrong schema is not a crash. It is a silent gap in the data that the operator discovers only when a query against the gap returns nothing.

How it works

  Kubernetes rolling restart of the write target
       |
       v
  Pod N+1 starts with the new schema_config
       |
       v
  +-----------+      +-----------+
  | Ingester  | ---> | flush     | ---> s3://bucket/chunks/...
  | writes    |      | chunk     |      schema=v13
  | under v13 |      | with v13  |
  +-----------+      +-----------+
       |
       v
  +-----------+
  | Index     | ---> s3://bucket/index_/...
  | update    |      schema=v13
  +-----------+

  Pod N is still running with the old schema_config
       |
       v
  +-----------+      +-----------+
  | Ingester  | ---> | flush     | ---> s3://bucket/chunks/...
  | writes    |      | chunk     |      schema=v11
  | under v11 |      | with v11  |
  +-----------+      +-----------+

During the rolling restart, both schemas are being written in parallel. The bucket receives v11 chunks from the older pods and v13 chunks from the newer pods. The schema_config list has both entries. The active schema for any write is the one that matches the active pod. The read path selects the schema per timestamp.

How to configure it

The migration is append-only. The new entry is added; the old entry stays. The new entry uses a from date in the future so the change can be applied without affecting current writes.

# /etc/loki/config-write.yaml
# The v11 entry was the original schema. The v13 entry is added
# at the migration date. The list is append-only.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# /etc/loki/config-read.yaml
# The read target carries the same list. A divergence produces a
# read path that selects the wrong schema for some timestamps.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# /etc/loki/config-backend.yaml
# The backend target (compactor + index-gateway) reads from both
# schemas. The schema_config list is the same; the differences
# are in the per-component blocks.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

The from field is the date at which the new schema becomes active. A future date lets the operator roll the config without changing behaviour. The new schema becomes active at midnight UTC on the date. The first new-prefix chunk is flushed at the next chunk_idle_period after midnight.

How to validate it

Five commands that confirm the schema_config is correct and the migration is healthy.

# READ-ONLY: confirm the running binary sees the expected entries.
curl -s http://loki-write-0:3100/config | jq '.schema_config.configs'
# expected: the list with both v11 and v13 entries.
# If only one entry appears, the second was not loaded.

# READ-ONLY: confirm the schema selection for a recent timestamp.
curl -s -G http://loki-read-0:3100/config \
  --data-urlencode 'since=2026-09-02T00:00:00Z' | jq '.schema_config.configs'
# expected: the same list. The active schema for the timestamp
# is selected at query time.

# READ-ONLY: confirm the new schema is being written.
curl -s http://loki-write-0:3100/metrics | grep 'loki_tsdb_index_writes_total'
# expected: a non-zero count within one chunk_idle_period of the
# migration date.

# READ-ONLY: confirm the old schema is still being read.
curl -s http://loki-read-0:3100/metrics | grep 'loki_boltdb_shipper_request_duration_seconds'
# expected: a non-zero count for the old schema as long as the
# retention window contains data written under v11.

# READ-ONLY: query a stream that straddles the migration date.
# The query should return results from both schemas.
logcli query --addr=http://loki-read-0:3100 \
  '{cluster="prod"} |= "synthetic-upgrade-test"' \
  --since=2026-08-25T00:00:00Z --until=2026-09-05T00:00:00Z
# expected: log lines from both sides of the migration date.
# An empty result for the pre-migration period means the v11
# index is missing or the read target is selecting the wrong
# schema.

How it can fail

Six failure modes cover the most common production incidents tied to the schema_config migration.

1. The v11 entry was removed during the upgrade. The schema_config.configs list was reduced from two entries to one. The active schema for a 2024 timestamp is now v13. The v13 index has no entries for 2024. Symptom: queries for old timestamps return empty. The fix is to restore the v11 entry.

2. The from date is in the past at the time of the apply. The operator intends to migrate on 2026-09-01 but applies the change on 2026-08-30. The new schema is active immediately. Writes before the rolling restart are v11; writes after are v13. The querier selects v13 for the 2026-08-30 timestamp because v13 is the most recent entry. The v11 chunks for that timestamp are invisible. Symptom: a gap in the data on the migration date.

3. The two object_store values disagree. The v11 entry uses object_store: s3 and the v13 entry uses object_store: azure. The bucket the v11 chunks live in is different from the bucket the v13 chunks live in. Symptom: queries for old timestamps hit the S3 bucket; queries for new timestamps hit the Azure bucket. The dashboard shows two disjoint sets of log lines.

4. The store and the schema are inconsistent. The v13 entry uses store: boltdb-shipper. The boltdb-shipper does not support v13. The binary logs a warning and falls back to the closest matching format. Symptom: the cluster runs but produces index files that are neither valid v11 nor valid v13. Queries return partial results.

5. The index.period was changed in the middle of the list. The v11 entry uses period: 24h and the v13 entry uses period: 1h. The size of the index files changes. The querier selects the per-hour files for v13 and the per-day files for v11. The metric loki_tsdb_index_files reports a much larger file count than planned. Symptom: the compactor runs out of disk during the first merge.

6. The chunk format and the schema version disagree. A v13 entry was added but the chunk encoder was not updated. The ingester writes v13-format index entries against v11-format chunks. The querier reads the chunk header, sees v11, applies the v11 decoder, and produces a corrupted string. Symptom: log lines arrive in Grafana as garbage bytes.

How to troubleshoot it

The diagnostic order for a schema_config regression:

1. Did the binary accept the new config? The startup log shows the parse result. A typo in from fails the parse with a clear error.
2. Is the active schema the one expected? curl /config | jq .schema_config shows the list. Compare against the planned list.
3. Is the new schema being written? The loki_tsdb_index_writes_total metric on the write target should show traffic within one chunk_idle_period of the migration date.
4. Is the old schema still being read? The loki_boltdb_shipper_request_duration_seconds_count metric on the read target should show traffic for as long as the retention window includes data written under v11.
5. Do queries straddle the migration date? logcli query with a since before the date and an until after the date produces a result that contains both schemas. A missing side is the symptom of a missing entry in the list.
6. Is the compactor keeping up? The loki_compactor_oldest_processed_age_seconds metric should stay close to the current time. A growing gap means the compactor is the bottleneck.

Security implications

The schema_config block is a data-routing block. A wrong entry points the read path or the write path at a different bucket. A bucket that is shared with a different Loki cluster or a different team is a data leak. The fix is to give each Loki instance its own bucket and to enforce bucket-level IAM policies that restrict the access to the Loki service account.

The index.prefix value is the prefix under which the index files live in the bucket. A wrong prefix collides with another tenant or another cluster. The compactor merges index files from both sources. The result is a corrupted index that neither side can read. The fix is a unique prefix per cluster.

Performance implications

The schema version changes the index format. The TSDB index is smaller per entry than the boltdb-shipper index, but the index file count is larger. The compactor runs more often during the first week after the migration. The loki_compactor_oldest_processed_age_seconds metric should be monitored during the migration window.

The query path for a v13 index is faster than the query path for a v11 index. The historical data (v11) is served by the older path. The mixed-mode cluster has two interesting performance profiles: the v11 path and the v13 path. The loki_request_duration_seconds histogram, broken down by the route label, shows the two paths separately.

Production guidance

• Add the new schema entry with a from date at least one week in the future. Apply the change. Watch the metrics for the week. Let the date arrive.
• Keep the old schema entry for the duration of the retention window. Removing it before the retention window expires produces a period of unreadable historical data.
• Use a date that is at midnight UTC. The schema is evaluated at the second level, but a date with a time component is parsed differently across operator mistakes.
• Pin loki_version_verified in the lesson frontmatter to the version that introduced the schema. The release notes call out the schema bump.
• Run a logcli query that straddles the migration date as part of the validation. The query must return results from both sides.

Verification

You should now be able to answer:

• Why is the schema_config.configs list append-only, and what is the consequence of removing an entry?
• What is the difference between the schema version and the store name, and why must they be paired consistently?
• Why must the new from date be in the future when the config is applied, and what is the operational consequence of an immediate-trigger date?
• What is the metric that proves the new schema is being written after the migration date?
• What is the metric that proves the old schema is still being read?

Quiz