Loki Storage Migration

A team is told to migrate Loki off a long-lived S3 bucket because the bucket is in a region that the company is decommissioning. The team writes a script that copies every object from the old bucket to the new one. The script runs over the weekend. On Monday, the team repoints Loki at the new bucket and rolls the cluster. The first query against a stream from six months ago returns nothing. The team investigates. The copy script used aws s3 cp with the default --recursive flag. The flag copies the immediate files but not the version history. The version IDs in the new bucket do not match the version IDs in the old bucket. Loki looks up the chunk by version ID, finds nothing, returns empty.

Storage migration is not a copy. It is a coordinated change of three things: the chunk store, the index store, and the schema config. The copy must be complete, versioned, and verified before the cluster is repointed.

What it is

A Loki storage migration is the process of moving the durable state of a Loki cluster from one object store to another. The durable state is composed of three kinds of objects:

• Chunks. Compressed log lines. Keyed by tenant ID and chunk ID. Each chunk is immutable once written.
• Index files. Per-tenant, per-day, per-store index files. The boltdb-shipper writes one file per day per tenant. The TSDB index writes one file per day per tenant. Both versions live in the bucket under the prefix declared in schema_config.
• Ruler state. Alert and recording rule state. Lives in the bucket the ruler block points at.

The three object kinds are typically in the same bucket, but Loki configuration allows them to be in different buckets. A migration that touches one bucket must touch the index prefix and the ruler prefix too.

Three destinations justify a storage migration:

1. Region change. The bucket is in a region the company is decommissioning. The migration moves the data to a region in the same provider or a different one.
2. Storage class change. The bucket is on standard storage; the migration moves it to infrequent-access storage for retention-bound data.
3. Provider change. The bucket is on S3; the migration moves it to GCS, Azure, or an on-premises MinIO.

The migration is an online process. Loki continues to read and write during the migration. The dual-write window is the period during which the cluster is configured to write to both the old and the new bucket. The cutover is the moment the cluster reads from the new bucket and stops reading from the old one.

Why a sysadmin cares

A storage migration is a one-shot operation. A failure produces gaps in the data that the operator cannot recover. Two production risks dominate:

1. Incomplete copy. The copy script missed files, lost version history, or stopped before the bucket was fully replicated. The cluster reads from the new bucket; some chunks are missing. Queries for the gap return empty.
2. Repointed before the index caught up. The chunk copy completed; the index copy did not. The cluster reads chunks from the new bucket but the index from the old bucket. The index points to chunk IDs that do not exist in the new bucket. Queries for the recent time range fail.

The cost of a failed storage migration is not the alert that fires. It is the days of post-incident analysis while the team reconstructs what was in the old bucket and decides whether to restore from S3 versioning or accept the gap.

How to configure it

The migration is a four-step process: snapshot, copy, verify, cutover. The configuration below is the cutover config that the cluster runs after the copy is complete.

# /etc/loki/common.yaml
# The common block carries the new bucket. The change is the
# bucket name; the rest of the S3 configuration is unchanged.
common:
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-2.amazonaws.com
    bucketnames: prod-loki-chunks-v2
    region: eu-west-2
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

# /etc/loki/config-write.yaml
# The write target points at the new bucket. The schema_config
# list is unchanged; the chunks are written under the new bucket
# prefix.
schema_config:
  configs:
    - from: '2024-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: '2026-09-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# /etc/loki/config-ruler.yaml
# The ruler block points at the new bucket. The ruler state
# files must be migrated too; otherwise the cluster loses alert
# evaluation history.
ruler:
  storage:
    type: s3
    s3:
      s3: s3://s3.eu-west-2.amazonaws.com
      bucketnames: prod-loki-ruler-v2
      region: eu-west-2

# /etc/loki/loki-canary.yaml
# loki-canary writes a synthetic log line every minute and
# queries it back. The metrics it reports are the foundation
# of the migration validation.
loki-canary:
  base_url: http://loki-read-0:3100
  push_url: http://loki-write-0:3100
  query_timeout: 30s
  read_timeout: 30s
  max_entries: 100

The migration configuration is the post-cutover config. The pre-cutover config is the same file with the old bucket name. The cutover is the diff between the two files.

How to validate it

Five commands that confirm the migration is complete and the cluster is reading from the new bucket.

# READ-ONLY: confirm the bucket the running binary sees.
curl -s http://loki-write-0:3100/config | jq '.common.s3.bucketnames'
# expected: "prod-loki-chunks-v2" for the post-cutover config.
# If the value is the old bucket, the cluster was not repointed.

# READ-ONLY: confirm the chunk count in the new bucket matches
# the chunk count in the old bucket.
aws s3api list-object-versions \
  --bucket prod-loki-chunks \
  --prefix 'fake/' \
  --output json | jq '.Versions | length'
aws s3api list-object-versions \
  --bucket prod-loki-chunks-v2 \
  --prefix 'fake/' \
  --output json | jq '.Versions | length'
# expected: the new bucket count is at least the old bucket count.
# A lower count means the copy missed files.

# READ-ONLY: confirm the loki-canary metric is reporting success.
curl -s http://loki-canary-0:3100/metrics | grep 'loki_canary_last_success'
# expected: a timestamp within the last 60 seconds. The canary
# writes a line, queries it back, and records the time of the
# last successful round trip.

# READ-ONLY: confirm the read path is serving from the new bucket.
curl -s http://loki-read-0:3100/metrics | grep 'loki_objstore_request_duration_seconds_count'
# expected: the count for the new bucket is non-zero and increasing.
# A flat count for the new bucket means the read path is still
# serving from the old bucket.

# READ-ONLY: query a stream from a time range that only exists
# in the new bucket. The query should return results.
logcli query --addr=http://loki-read-0:3100 \
  '{cluster="prod"} |= "synthetic-upgrade-test"' \
  --since=2026-09-15T00:00:00Z --until=2026-09-15T01:00:00Z
# expected: log lines from the synthetic window. An empty result
# means the new bucket is missing the chunks for this window.

How it can fail

Six failure modes cover the most common production incidents tied to a Loki storage migration.

1. The copy preserved the chunk IDs but not the version IDs. The index file points to version IDs that do not exist in the new bucket. Symptom: queries for the recent time range return empty. The metric loki_objstore_request_duration_seconds shows the read path hitting the new bucket with NoSuchVersion errors.

2. The copy ran but the index prefixes were missed. The chunks were copied; the index files were not. Symptom: queries for the old time range return empty because the index is missing. The metric loki_tsdb_index_files shows a much smaller count than expected.

3. The dual-write window was too short. The cutover happened before the ingester had written to the new bucket under the new schema. Symptom: a gap in the data at the cutover time. The metric loki_ingester_chunks_flushed_total shows the new bucket has no chunks for the cutover window.

4. The ruler state was not migrated. The bucket was migrated; the ruler bucket was not. Symptom: alert evaluation history is lost. The metric loki_ruler_wal_replay_duration_seconds shows a replay that starts from empty.

5. The cutover was applied to the read target before the write target. The read target was rerouted to the new bucket; the write target was still writing to the old bucket. Symptom: new log lines are visible in the old bucket but absent from the new bucket. Queries for the recent time range return empty.

6. The copy job was throttled by the S3 API rate limit. The copy ran at a fraction of the expected throughput. Symptom: the migration takes longer than the dual-write window. The cluster is repointed to the new bucket before the copy is complete. A gap in the data for the un-copied time range.

How to troubleshoot it

The diagnostic order for a storage migration that does not behave as planned:

1. Which bucket is the cluster reading from? curl /config | jq .common.s3.bucketnames shows the running bucket. The value is the source of truth for the read path.
2. Are the chunk counts equal? aws s3api list-object-versions on both buckets. The new bucket count should be at least the old bucket count.
3. Is the index caught up? The loki_tsdb_index_files metric on the index-gateway should match the count in the new bucket. A divergence means the index migration is incomplete.
4. Is the dual-write window active? The loki_ingester_chunks_flushed_total metric on the write target should show traffic against both buckets during the window. A single-bucket count means the dual-write is not configured.
5. Are the canary metrics passing? The loki_canary_last_success metric should be within the last 60 seconds. A stale value means the read path is failing.
6. Is the compactor keeping up? The loki_compactor_oldest_processed_age_seconds metric should stay close to the current time. A growing gap means the compactor is retrying on the new bucket and failing.

Security implications

The migration is a privilege escalation window. The new bucket requires a new IAM policy. The old bucket IAM policy must remain active until the dual-write window closes. A policy that is removed before the close produces a write target that fails to flush.

The new bucket must be in a region that complies with the data residency requirements. A migration that crosses a residency boundary is a compliance event. The fix is to verify the destination region against the residency policy before the migration starts.

Performance implications

A storage migration is a workload change on the bucket. The new bucket may have a different API cost model, a different latency profile, or a different throughput ceiling. The loki_objstore_request_duration_seconds histogram, broken down by the operation label, shows the per-operation cost. A migration to a different storage class typically shows a slower GET latency and a faster LIST latency.

The migration job itself is a workload. The aws s3 cp job saturates the network and the bucket. The dual-write window doubles the write load. The cluster must be sized for the peak load during the window, not the steady-state load.

Production guidance

• Use S3 batch operations or aws s3 sync with the --copy-source flag to preserve version IDs. A bare aws s3 cp is not safe.
• Run the migration job against a staging bucket first. The staging run validates the copy logic and the verification commands.
• Keep the dual-write window open for at least one retention period. The window is the only guarantee that new data is in the new bucket.
• Run loki-canary continuously. The canary is the first signal that the read path is serving from the new bucket.
• Capture the resolved config with -print-config-stderr before the cutover. Diff against the file intended to load.
• Document the rollback path. The path is the inverse of the cutover: repoint the read target back to the old bucket, stop the dual-write, and wait for the chunk ID gap to close.

Verification

You should now be able to answer:

• What is the dual-write window, and why must it be open for at least one retention period?
• Why is the version ID important in the index file, and what is the failure mode of a copy that does not preserve it?
• Why is the ruler state migrated separately, and what happens if it is not?
• What is the difference between the chunk copy and the index copy, and why must both be complete before the cutover?
• What is the rollback path if the migration cannot be completed?

Quiz