boltdb-shipper to TSDB

A team operates Loki on the boltdb-shipper index. The upgrade target is Loki 3.x, which defaults to the TSDB index. The team appends the v13 entry to schema_config.configs and rolls the cluster. The new pods start with the TSDB index. The pods detect old boltdb-shipper files in the bucket and log a deprecation warning. The team reads the warning, runs the migration, and the index files are rewritten. The team removes the v11 entry from schema_config.configs and rolls the cluster again. The new pods start with the TSDB index only. The index-gateway returns an error: no index files found for tenant prod for the requested time range. The team investigates. The migration rewrote the files under a different prefix. The original files were deleted by the migrator. The new prefix is empty for the historical time range.

The boltdb-shipper to TSDB migration is not a config change. It is a one-shot operation that rewrites the index files in place. The prefix is the same; the file format changes. A migration that is not verified before the cleanup produces a gap in the data.

What it is

The boltdb-shipper to TSDB migration is the process of rewriting the Loki index files from the older boltdb-shipper format to the newer TSDB format. The migration is a one-shot operation that runs against the index bucket. The migrator reads each boltdb-shipper file, deserialises the index entries, and writes the equivalent TSDB files. The migrator then deletes the source files.

Three versions of the migration exist:

• Schema v11 with boltdb-shipper to schema v13 with tsdb. The default migration path. The migration rewrites every index file under the schema’s prefix.
• Schema v12 with boltdb-shipper to schema v13 with tsdb. The same migration, but the v12 entry is in the schema_config list. The migration is typically a no-op for the v12 index because the v12 and v13 file formats are similar.
• Schema v13 with boltdb-shipper to schema v13 with tsdb. A late upgrade where the cluster was on v13 with the old index format. The migration rewrites the files under the same schema version.

The migration is online. Loki continues to read from the boltdb-shipper files while the migrator is running. The new TSDB files are written into a separate prefix until the migration is complete. The migrator atomically renames the new prefix to the active prefix when the rewrite for a tenant is complete.

Why a sysadmin cares

The migration is a one-shot operation that touches every index file in the bucket. A failure produces a gap in the data that the operator cannot recover without a backup. Three production scenarios apply:

1. The migrator ran but the verification failed. The migrator claims success but the TSDB files are missing some entries. The cluster reads from the TSDB files. Queries for the affected time range return empty.
2. The migrator was interrupted mid-run. The migrator crashed or was killed. The source files were partially deleted. The new files are partially written. The cluster sees a mix of old and new files. Queries return inconsistent results.
3. The schema_config was updated before the migration. The schema_config.configs list was updated to v13 only. The migration is running against the old prefix. The cluster reads from the new prefix, which is empty. Queries for the historical time range return empty.

The cost of a failed migration is the days of post-incident analysis while the team reconstructs the index from the S3 versioning history, or accepts the gap and rolls the cluster back to the boltdb-shipper index.

How to configure it

The migration is a CLI operation. The CLI takes the config file and the source schema as arguments. The output is the progress and the completion signal.

# /etc/loki/config-write.yaml
# The schema_config list is the source of truth for the migration.
# The v11 entry is the source; the v13 entry is the target. The
# migration rewrites the v11 files into the v13 format.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# /etc/loki/config-read.yaml
# The read target must continue to serve from the boltdb-shipper
# files during the migration. The boltdb-shipper store is still
# declared in the schema_config list. The cluster reads from both
# stores during the window.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# /etc/loki/config-backend.yaml
# The backend target (compactor + index-gateway) reads from both
# stores. The schema_config list is the same.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

The migration is invoked from the CLI. The config file is the same; the migration is a separate process that does not require a cluster restart.

# Run the migration as a one-shot job. The CLI reads the
# config file, identifies the boltdb-shipper files, and rewrites
# them as TSDB files. The job is idempotent; re-running skips
# files that have already been migrated.
loki tsdb-migrate \
  --config.file=/etc/loki/config.yaml \
  --source-schema=v11 \
  --target-schema=v13 \
  --tenant-filter='prod|stage' \
  --batch-size=100
# expected: the job prints progress per tenant. The completion
# signal is the migrated file count matching the source file count.

How to validate it

Five commands that confirm the migration is complete and the cluster is reading from the TSDB index.

# READ-ONLY: confirm the migrator job is finished.
loki tsdb-migrate \
  --config.file=/etc/loki/config.yaml \
  --source-schema=v11 \
  --target-schema=v13 \
  --dry-run
# expected: the dry-run lists the files that would be migrated.
# A non-zero count means the migration is incomplete.

# READ-ONLY: confirm the index-gateway is serving from the TSDB
# index.
curl -s http://loki-backend-0:3100/metrics | grep 'loki_tsdb_index_request_duration_seconds'
# expected: a non-zero count for the v13 schema. A flat count
# means the index-gateway is still serving from the boltdb-shipper.

# READ-ONLY: confirm the boltdb-shipper files are gone.
aws s3api list-objects-v2 \
  --bucket prod-loki-chunks \
  --prefix 'index_/prod/' \
  --output json | jq '.KeyContents | length'
# expected: zero. The migrator deletes the source files after
# the rename is complete.

# READ-ONLY: confirm the TSDB files are present.
aws s3api list-objects-v2 \
  --bucket prod-loki-chunks \
  --prefix 'index_/prod/tsdb/' \
  --output json | jq '.KeyContents | length'
# expected: a non-zero count matching the historical file count.

# READ-ONLY: query a stream that straddles the migration date.
# The query should return results from both the old and new
# schemas.
logcli query --addr=http://loki-read-0:3100 \
  '{cluster="prod"} |= "synthetic-upgrade-test"' \
  --since=2026-08-25T00:00:00Z --until=2026-09-05T00:00:00Z
# expected: log lines from both sides of the migration date.
# An empty result for the pre-migration period means the boltdb-
# shipper files were deleted before the TSDB files were verified.

How it can fail

Six failure modes cover the most common production incidents tied to the boltdb-shipper to TSDB migration.

1. The migrator ran but the TSDB files are missing entries. The migrator claimed success but the deserialisation skipped some entries. Symptom: queries for the affected time range return empty. The loki_tsdb_index_request_duration_seconds_count metric shows the index-gateway hitting the new prefix but the chunk IDs match fewer chunks than expected.

2. The migrator was interrupted mid-run. The migrator crashed or was killed. The source files were partially deleted. The new files are partially written. Symptom: the cluster sees a mix of old and new files. The loki_objstore_request_duration_seconds histogram shows NoSuchKey errors for the partially-deleted files.

3. The schema_config was updated before the migration. The schema_config.configs list was updated to v13 only. The migration is running against the old prefix. The cluster reads from the new prefix, which is empty. Symptom: queries for the historical time range return empty. The loki_tsdb_index_files metric shows zero files for the old prefix.

4. The migrator exceeded the S3 API rate limit. The migration ran at a fraction of the expected throughput. The S3 API returned throttling errors. Symptom: the migration takes much longer than expected. The loki_objstore_request_duration_seconds histogram shows elevated latency.

5. The cleanup ran before the verification. The migrator deleted the source files without verifying the TSDB files. Symptom: a gap in the data for the time range that was migrated. The loki_tsdb_index_files metric shows the new prefix is empty for the affected time range.

6. The migrator was run on the wrong bucket. A typo in the config file pointed the migrator at a staging bucket. The migration ran against the staging data. Symptom: the staging bucket was rewritten; the production bucket is unchanged. The cluster reads from the production bucket. The TSDB files are missing.

How to troubleshoot it

The diagnostic order for a migration that does not behave as planned:

1. Is the migrator running? The loki_tsdb_migrate_in_progress metric shows the job state. A non-zero value means the migration is in progress.
2. Is the migration making progress? The loki_tsdb_migrate_files_migrated_total metric shows the file count. A flat count means the migration is stuck.
3. Is the index-gateway serving from the TSDB index? The loki_tsdb_index_request_duration_seconds metric shows the read traffic. A flat count means the index-gateway is still serving from the boltdb-shipper.
4. Are the boltdb-shipper files gone? aws s3api list-objects-v2 on the boltdb-shipper prefix. The count should be zero after the cleanup.
5. Are the TSDB files present? aws s3api list-objects-v2 on the TSDB prefix. The count should match the historical file count.
6. Does the S3 API rate limit cause throttling? The loki_objstore_request_duration_seconds histogram shows the per-operation latency. A spike is the symptom of throttling.

Security implications

The migration is a privilege escalation window. The migrator requires read access to the source prefix and write access to the target prefix. The IAM policy must be granted to the migration service account, not the Loki service account. The policy is revoked after the migration is complete.

The migrator deletes the source files. The deletion is logged and reversible via S3 versioning. The reversal requires PutBucketVersioning enabled on the bucket before the migration. A bucket without versioning cannot recover the deleted files.

Performance implications

A migration is a workload change on the bucket. The API rate limit is the bottleneck. The migrator must be tuned with a batch size that keeps the API rate below the limit. The loki_objstore_request_duration_seconds metric, broken down by the operation label, shows the per-operation cost.

The TSDB index is smaller per entry than the boltdb-shipper index. The total bucket size shrinks after the migration. The loki_tsdb_index_files metric shows the file count. The compactor runs more often during the first week after the migration.

Production guidance

• Enable S3 versioning on the bucket before the migration. The versioning is the rollback path.
• Run the migration against a staging bucket first. The staging run validates the migrator arguments and the batch size.
• Use a tenant filter to limit the migration to a subset of tenants. The first run should be the lowest-traffic tenant.
• Schedule the migration for a low-traffic window. The API rate limit is shared with the cluster.
• Run loki-canary continuously. The canary is the first signal that the read path is serving from the TSDB index.
• Capture the resolved config with -print-config-stderr before and after the migration. The diff shows the schema_config changes.
• Document the rollback path. The path is the inverse of the migration: revert the schema_config, roll the cluster, delete the TSDB files.

Verification

You should now be able to answer:

• Why is the boltdb-shipper to TSDB migration a one-shot operation, not a config change?
• What is the verification step that must run before the cleanup, and what is the failure mode of a migration without it?
• Why is the schema_config update separate from the migrator run, and what is the symptom of doing it in the wrong order?
• What is the S3 API rate limit impact, and how is the migrator tuned to stay below it?
• What is the rollback path if the migration cannot be completed?

Quiz