Break/Fix: Upgrade Breaks Configuration

Reported symptoms

The Loki upgrade from 2.9 to 3.4 went in on Saturday morning in a planned window. It was, by every measure the team owned, a good upgrade. The release notes were read. The config was migrated. loki -verify-config passed against the new binary before the rollout and again afterwards. The rolling restart completed at 09:14. Every pod came back Ready, request latency did not move, the ingest rate did not move, and the team closed the change ticket.

It is now Tuesday, and someone investigating a Friday incident cannot find Friday.

• Explore stops at 09:14 on Saturday. For every tenant. The retention period is 2160h and no retention job ran.
• The Friday query does not fail. It returns an empty result set. Two people looked at it and concluded they had written the LogQL wrong, which is a completely reasonable conclusion to draw from an empty result.
• The bucket has stopped growing. Its daily object count has not moved since Saturday. The finance dashboard picked up the collapse in PUT requests on Sunday and someone annotated it as a saving from the new index format.
• The write pods are filling up. The local data volumes have gone from roughly 3 GB each to roughly 137 GB each in three days. Nothing alerted, because the alert is on percent-free of a 2 TiB volume and 137 GB is nowhere near it.
• tenant-a is being rate-limited. It has a documented ingestion override of 64 MB/s. It is getting 429s. Nobody edited tenant-a, and nobody edited the global limit either.
• The compactor says it has nothing to compact. Filed Monday as a separate bug.

Six symptoms, four tickets, three teams, and every health signal the platform publishes about itself is green.

Evidence provided

Start with the thing everybody skips, because reading the file on disk has already been done twice and told nobody anything. Ask the running process what configuration it is actually using:

$ kubectl exec -n loki loki-write-0 -- loki -config.file=/etc/loki/config-write.yaml -print-config-stderr 2>&1 | grep -A6 '^common:'

common:
path_prefix: /var/lib/loki
storage_backend: filesystem
replication_factor: 3
ring:
  kvstore:
    store: consul

The file on disk still contains a fully populated S3 section with the bucket name, the region and the credentials. The merged configuration the binary resolved does not reference it at all.

The external witness agrees, and it is the one source in this incident that cannot be wrong about where the data went, because it is not part of the cluster:

$ aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name NumberOfObjects --dimensions Name=BucketName,Value=prod-loki-chunks Name=StorageType,Value=AllStorageTypes --start-time 2026-08-13T00:00:00Z --end-time 2026-08-19T00:00:00Z --period 86400 --statistics Average --output text

NumberOfObjects
DATAPOINTS	41827104.0	2026-08-14T00:00:00Z	Count
DATAPOINTS	42019663.0	2026-08-15T00:00:00Z	Count
DATAPOINTS	42106918.0	2026-08-16T00:00:00Z	Count
DATAPOINTS	42106918.0	2026-08-17T00:00:00Z	Count
DATAPOINTS	42106918.0	2026-08-18T00:00:00Z	Count

And the data that should be there is somewhere else:

$ kubectl exec -n loki loki-write-0 -- sh -c 'du -sh /var/lib/loki/chunks; df -h /var/lib/loki'

137G	/var/lib/loki/chunks
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    2.0T  141G  1.9T   8% /var/lib/loki

The tenant-a 429 is worth reading carefully rather than skimming, because the error names the number:

429 Too Many Requests
Ingestion rate limit exceeded for user tenant-a (limit: 33554432 bytes/sec)

33554432 bytes per second is 32 MiB per second. That is the global ingestion_rate_mb. tenant-a is documented at 64. The limiter is not misbehaving; it has never heard of the override.

Which leads to the arguments the write pods are running with:

$ kubectl get sts -n loki loki-write -o json | jq -r '.spec.template.spec.containers[0].args[]'

-config.file=/etc/loki/config-write.yaml
-target=write

Finally, the migration itself. Compare what the old file declared against what the new file declares, and note what has no counterpart:

# Substitute your own paths before running:
OLD=/srv/loki-config/2.9/config-write.yaml
NEW=/srv/loki-config/3.4/config-write.yaml

# Which top-level blocks exist in each file?
grep -E '^[a-z_]+:' "$OLD" | sort
grep -E '^[a-z_]+:' "$NEW" | sort

The old file carries storage_config and chunk_store_config. The new one carries neither, and its common block carries no storage backend. Nothing in the startup log mentions either fact.

Work the evidence before reading on

You have a cluster that starts, passes its own validator, reports itself healthy, and is doing something nobody asked for.

1. A query for Friday returns an empty result rather than an error. What does that tell you about where the cluster looked, and why is “empty” a more dangerous answer than “failed”?
2. verify-config passed. Be precise about what that command proves and what it cannot prove. The distinction is the whole incident.
3. The bucket stopped receiving objects on Saturday morning and the write pods started accumulating data on Saturday morning. Those are not two facts.
4. tenant-a is limited at exactly the global value. What single mechanism would produce both a wrong storage backend and a missing tenant override, without logging anything about either?
5. The compactor has nothing to compact and the finance dashboard recorded a cost saving. Both were filed as good news or as unrelated bugs. What were they actually reporting?

And before you read on, the question that decides how the next hour goes: the fix is a configuration change and a restart. What is on those write pods right now, and what happens to it the moment you restart them?

Root cause

1. An unrecognised key is not an error

The binary parses the YAML into a struct that has a field for every configuration key the version knows about. A key in the file that has no field in the struct is discarded during parsing. The struct is then merged with the defaults, and the merged result is what every component reads.

There is no error for this and no warning worth the name. From the parser’s point of view, nothing went wrong: the document was well-formed, every field it recognised was populated, and the rest was not its business.

The 2.9 configuration declared the object store under storage_config and chunk_store_config. In 3.x the backend is resolved from the common block. So both of the old blocks were read and dropped, common was left with no backend, and the default applied. The default is filesystem, and path_prefix - which did survive the migration, because it is still a real key - supplied a directory to use.

The cluster then behaved impeccably. It started, it accepted writes, it flushed chunks on schedule, and it wrote every one of them to /var/lib/loki/chunks on the pod that happened to receive them.

2. Reads are consistent with writes, so nothing errors

This is why three days passed.

A query for a timestamp after 09:14 looks up the index for that period in the store the cluster is configured with, finds it, fetches the chunks from the same store, and returns them. Correct data, correct latency, no complaint.

A query for a timestamp before 09:14 looks up the index for that period in the same store. It is not there and never was. An index lookup that returns nothing is not an error condition - it is the ordinary answer for a stream that has no data in the window - so Loki returns an empty result set, with a 200, in normal time.

Ninety days of logs are sitting intact in prod-loki-chunks. Nothing deleted them, nothing corrupted them, and nothing can reach them, because there is no longer any layer of this cluster that knows the bucket exists.

An empty result is the most expensive shape a failure can take, because it is indistinguishable from a correct answer to a slightly different question, and the first instinct of a competent engineer who gets one is to check their own query.

3. The tenant override is the same mechanism, smaller

In the 2.9 configuration the per-tenant overrides sat inside the main YAML. 3.x reads them from a separate runtime configuration file, loaded by -runtime-config.file.

So the override block in the migrated file is a key with no field: parsed, discarded, silent. And the replacement was never wired up, because the StatefulSet arguments were rewritten during the upgrade and the new flag was not among them. tenant-a therefore falls through to the global ingestion_rate_mb of 32, and the 429 body reports 33554432 bytes per second because that is what the limiter genuinely believes tenant-a is entitled to.

Nothing about this is a second bug. It is the same sentence twice: a key the new version does not read there, dropped without comment, replaced by whatever the default happens to be.

4. Every green signal was answering a different question

loki -verify-config parses the file and validates the structure. It returns success when the document is a valid configuration for this version. It has no opinion about whether the resulting cluster resembles the one you were running on Friday, and it could not have one - it has never seen the 2.9 configuration.

Readiness probes report that the process is up and serving. The ingest rate is normal because ingestion genuinely is normal. Request latency is normal for the same reason. The compactor reporting nothing to compact was a true statement about an empty store, and it was filed as a bug in the compactor. The finance dashboard reporting a collapse in PUT requests was a true statement about a bucket receiving nothing, and it was annotated as a saving.

Two instruments detected this failure within 48 hours. Both were read as good news, because neither was framed as a question about correctness.

Resolution

1. Freeze movement before you diagnose any further. Suspend anything that could reschedule or scale the write pods: cluster autoscaler, node drains, planned maintenance, and any GitOps controller that would reconcile the StatefulSet. The data you are about to protect is pinned to three specific pods on three specific nodes.
2. Snapshot the three local volumes. This is the step that makes every later decision reversible, and it is the only step in this list that gets harder the longer you wait.
3. Decide what the misplaced 71-hour window is worth, and record who decided. Option one: copy the volumes off and serve them from a separate read-only instance pointed at the filesystem store for as long as the window matters, accepting a second thing to operate and eventually retire. Option two: re-ingest from the origin hosts, which covers only what the agents can still see and requires a pipeline that preserves original timestamps rather than stamping ingestion time - a gap is better than a plausible lie about when things happened. Option three: write the window off and document the gap where an incident reviewer will find it, which is defensible for a log platform and not defensible if anything in that window is under a retention obligation.
4. Correct the configuration: move the object store under the common block so the backend resolves to s3, and restore -runtime-config.file to the StatefulSet arguments with the tenant overrides in it. Both halves, in the same change - fixing storage and leaving tenant-a limited is how a follow-up incident gets created.
5. Verify the merged configuration before you restart anything. Run the new file through the binary and read the resolved storage_backend, bucket and region. The file has been wrong for three days precisely because nobody looked at what it resolved to.
6. Restart the write path, then the read path, and watch the bucket rather than the pods. Readiness told you nothing on Saturday and will tell you nothing now.
7. Do not roll back to 2.9. It does not relocate the misplaced data, schema_config now carries a v13 tsdb entry with three days of writes beneath it, and a 2.9 binary cannot read a layout that postdates it. There is no pre-upgrade bucket snapshot, so the precondition for a clean rollback was never met. This is a forward fix and saying so early stops somebody proposing it at 03:00.
8. Close the compactor ticket and the finance annotation with the real cause rather than leaving them as an unrelated bug and a cost saving. Both were correct detections that were filed as something else, and leaving them mislabelled removes two working detectors from the estate.
9. If you hold rather than act, hold with an owner and an end time. The disks have roughly five weeks of headroom at the current rate, so a hold is defensible - but every day of it adds a day of logs to a store you will have to deal with, and the day the volumes fill, ingestion stops hard for every tenant simultaneously.

Verification

1. Verify the cluster, not the file. Print the merged configuration from the running binary and confirm storage_backend resolves to s3 with the bucket and region you expect. The file on disk was never the thing that was wrong.
2. Diff the merged output against the pre-upgrade merged output. If you never captured it, capture it now by running the 2.9 binary against the old file offline. Building that artefact late is worth doing; it is the thing the upgrade plan should have carried, and you will want it for the next version bump.
3. Verify from outside Loki. The decisive check is that the bucket starts receiving objects again, observed through the object store own API. A component pointed at the wrong backend will report healthy writes to that backend all day, so no Loki metric can settle this question.
4. Verify reads at three separate points and expect three different answers: a stream from before the upgrade must return data from the bucket; a stream from inside the misplaced window must behave the way your chosen disposition says it should; a stream from after the fix must return data. One passing query is evidence about one window.
5. Verify the tenant limit by pushing, not by reading configuration. Send tenant-a above the global 32 MB/s and below its 64 MB/s override and require success; then send it above the override and require the 429. A check that only exercises the passing case proves a limit exists, not that it is the right one.
6. Confirm the local volume growth on the write pods returns to its previous baseline. That is the direct statement that writes have moved, independent of anything the configuration claims.
7. Confirm the compactor starts compacting. It has been idle for three days because there was nothing in the store it was pointed at, and its return to work is a second, independent confirmation that the store is real again.
8. Confirm no write pod was rescheduled between the decision to preserve the window and the completion of the snapshots. If one was, find out what was on it before you tell anyone the window is safe.

Prevention

• Diff the merged configuration, not the file. Capture the resolved config from the current version and from the target version, and read the difference before the canary. Every element of this failure was a line in that diff.
• Make the diff a CI artefact rather than a human habit. The pull request that bumps the version should produce it, and a reviewer should have to look at it. A discipline that depends on somebody remembering is a discipline that survives until the first busy week.
• Stop treating a successful start as evidence. verify-config, a green readiness probe and a normal ingest rate are statements about parsing and process health. This cluster produced all three while writing every log line to the wrong disk.
• Own an external witness for every storage backend. Newest-object timestamp, object count or PUT rate, observed through the object store, cannot be fooled by the component being wrong about where it writes. No self-published metric has that property.
• Alert on the absence of writes to storage, not only on write errors. Zero is the shape this failure takes, and almost nobody alerts on zero.
• Watch the local disk of anything you believe is stateless on the data path. A component that suddenly starts consuming local storage is telling you something about its configuration before it tells you anything else.
• Turn the release notes into a table. For every breaking change, record rename, migrate or no-op against your own configuration, and treat an unanswered row as a blocker rather than as a nice-to-have.
• Make the canary read across the schema boundary and write into the bucket. A canary that only becomes Ready tests the thing that was never in doubt.
• Prepare the rollback before you need it. A bucket snapshot and a merged config are worthless taken afterwards, and this upgrade had neither.