Break/Fix: Object Storage Unavailable · Break/Fix · Observability

Reported symptoms

Four tickets are open against the observability platform. On the face of it they belong to four different teams, and three of them have already been closed once.

Logs work, history does not. The checkout team reports that Loki is broken. The platform team looks, runs a query over the last thirty minutes, gets lines back, and closes the ticket. The checkout team reopens it with a screenshot: the same selector over yesterday afternoon returns an error, not an empty result. Every query that stays inside roughly the last two hours succeeds. Every query that reaches past it fails.

Traces have the same shape and nobody connects them. A separate ticket says Tempo has lost old traces. A trace ID copied from a log line five minutes ago resolves and renders. A trace ID from a ticket filed yesterday returns not found. The reporter assumes retention was shortened.

The bucket is growing and nothing is being deleted. Finance flags the object-storage line item. It has risen three days running. The storage team confirms the growth is real and points out something stranger: the daily delete count against the bucket is zero, and has been since Tuesday.

An alert did not fire. Overnight, a log-based alert that should have paged did not. The Loki ruler pod was restarted routinely at 06:10 as part of node maintenance and came back with no rule groups loaded. That ticket is filed against node maintenance.

And the thing that is not a symptom, which is what kept all four tickets separate: Prometheus is perfect. Every scrape target is up, every panel renders, every Prometheus-based alert has fired and resolved normally all week. Ingestion is green too. The push rate into Loki, the distributor accept rate and the ingester flush rate are all sitting at their usual values.

Nobody has deployed Loki, Tempo, Grafana or a collector in three weeks.

Evidence provided

The cheapest question to ask first is not “is the bucket up” but “what is the bucket saying to us”. Loki emits a histogram for every backend operation, labelled by operation and by status code, so one query separates a reachability problem from an authorisation problem without touching the cloud console at all.

Read-only / Safetwo operations succeed, two are refused

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=sum by (operation, status_code) (rate(loki_objstore_request_duration_seconds_count[10m]))' \
| jq -r '.data.result[] | .metric.operation + " " + .metric.status_code + " " + .value[1]'

PUT    200  38.412
LIST   200  0.851
HEAD   200  2.960
GET    403  14.203
DELETE 403  0.612

Illustrative output

That single result is the whole incident, and it is worth sitting with before reading further. A bucket that is down does not return 200 to some operations and 403 to others. A network partition does not either.

Read-only / Safethe querier names the failure precisely

$ kubectl -n loki logs deploy/loki-querier --since=15m | grep -i -m 2 denied

level=error caller=fetcher.go msg="error fetching chunks" err="AccessDenied: Access Denied, status code: 403"
level=error caller=fetcher.go msg="error fetching chunks" err="AccessDenied: Access Denied, status code: 403"

Illustrative output

The compactor tells the same story in a different vocabulary. Its progress metric reports the age of the oldest window it has finished processing, so a healthy compactor keeps it small and a stalled one lets it climb.

Read-only / Safejust over three days of un-swept retention

$ curl -s http://loki-compactor.internal.example.com:3100/metrics | grep -E '^loki_compactor_oldest_processed_age_seconds'

loki_compactor_oldest_processed_age_seconds 268140

Illustrative output

Tempo, queried the same way, splits neatly along the same seam: its read path is failing and its write path is not.

Read-only / Safereads failing, flushes clean

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=sum(rate(tempo_querier_query_errors_total[10m])) or vector(0)' \
--data-urlencode 'time=now' \
| jq -r '.data.result[0].value[1]'
curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=sum(rate(tempo_ingester_failed_flushes_total[10m])) or vector(0)' \
| jq -r '.data.result[0].value[1]'

3.417
0

Illustrative output

Two checks falsify the obvious hypothesis. The bucket answers a head-bucket from an operator workstation, using the operator’s own credentials rather than the platform’s, which separates “the bucket is reachable” from “the platform is allowed to use it”.

Read-only / Safethe bucket is present and reachable

$ aws s3api head-bucket --bucket prod-observability --region eu-west-1; echo exit=$?

exit=0

Illustrative output

And the policy simulator answers the question directly, for the identity that actually matters.

Read-only / Safetwo of the four required actions are gone

$ aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::123456789012:role/observability-storage \
--action-names s3:GetObject s3:PutObject s3:DeleteObject s3:ListBucket \
--resource-arns arn:aws:s3:::prod-observability \
                arn:aws:s3:::prod-observability/tenant-checkout/probe \
--query 'EvaluationResults[*].[EvalActionName,EvalDecision]' --output text

s3:GetObject     implicitDeny
s3:PutObject     allowed
s3:DeleteObject  implicitDeny
s3:ListBucket    allowed

Illustrative output

The change record shows an IAM least-privilege consolidation applied to the shared observability role at 02:05 on Tuesday. No Loki or Tempo pod has restarted since Monday.

Work the evidence before reading on

Four tickets, four teams, one bucket that is demonstrably up.

A bucket outage and a network partition both fail everything. This failure returns 200 to PUT and LIST and 403 to GET and DELETE. What kind of fault can distinguish between operations on the same bucket from the same client?
Loki answers for the last two hours and fails beyond that. Which component serves each of those two ranges, and which of them talks to the bucket?
Prometheus is entirely unaffected while Loki and Tempo are both degraded in the same way. What does Prometheus not have in that bucket?
The ruler was fine for four days and broke at 06:10 when it restarted. What does a ruler do at startup that it does not do while it is running?

Before continuing: no pod has restarted since Monday and the platform’s credentials have not been reissued, yet the behaviour changed at 02:05 on Tuesday. What does that tell you about where the decision to refuse these requests is being made, and what does it rule out?

Root cause

1. The store was never unavailable; the path to it was

The bucket was up for the whole incident. Objects were intact, the region was healthy, the network path was clean, and an operator could list and download at will. What changed was the identity’s right to use it.

At 02:05 on Tuesday an IAM consolidation replaced the inline policy on the shared observability-storage role. The replacement was generated from a write-path template, so it grants s3:PutObject and s3:ListBucket and omits s3:GetObject and s3:DeleteObject. Those four actions are the set the platform needs, and losing half of them does not take the platform down. It takes half of the platform down, which is much harder to see.

2. Reads and writes are different actions, so the failure is asymmetric

This is the property that turned one policy edit into four unrelated-looking tickets.

The ingesters only need to write. They kept flushing chunks and blocks on schedule, so push rate, accept rate and flush rate never moved and every ingestion dashboard stayed green. That greenness is what let the first Loki ticket be closed as unreproducible.

The queriers only need to read. In Loki, a query over a recent range is answered from the ingesters, which hold the window between “in memory” and “in object storage” — typically about two hours. A query that reaches beyond that window has to fetch chunks from the bucket, and every one of those fetches now returns 403. That is the two-hour boundary the checkout team measured without knowing what it was. Tempo behaves identically: spans still land and still flush, but a trace whose block has already left the ingester has to be downloaded, and the download is refused.

The compactor needs to read its marker files and delete expired chunks. It lost both. Retention stopped on Tuesday morning, which is why the bucket has grown for three days and why the delete count is exactly zero rather than merely low. Nothing alerted, because no rule watched the retention sweep.

The ruler needs to read its rule groups from the bucket at startup. It had them in memory, so it kept evaluating happily for four days. The 06:10 restart was the first moment it needed the read, and the read was refused. The alert that did not fire had nothing to do with node maintenance.

3. Prometheus was fine because it is not in the bucket

Prometheus 2.55 writes samples to a local TSDB. Its data is not in the object store, so no policy on that bucket can affect it. Every Prometheus panel and every Prometheus-based alert therefore stayed correct throughout, and that correctness is the single biggest reason the storage hypothesis was slow to arrive: half the platform was healthy, and it was the half people look at first.

Under the hood

Two mechanisms explain why nobody had to do anything for the outage to start, and why nobody has to do anything for it to stop.

IAM is evaluated on every request. The role’s credentials are minted by the cloud provider and refreshed roughly hourly, but the policy attached to the role is consulted at the moment the request arrives at the bucket, not at the moment the credential is issued. A policy edit therefore takes effect for running processes that have not restarted and hold perfectly valid credentials — which is exactly what the evidence shows, and exactly why “the credentials must be stale, restart the pods” is the wrong instinct here. The same mechanism runs in reverse: restoring the actions restores service without a restart.

The two-hour boundary is not a Loki setting anyone chose. The ingester holds acknowledged writes in memory, backed by a write-ahead log, and periodically flushes them to object storage; after a successful flush the WAL segment for that window can be truncated. Until the flush, the data exists in two places. After it, the bucket is the only place. Any fault that affects only the bucket is therefore invisible for as long as the recent window lasts, and becomes visible the moment a reader asks for something older. The window is the reason the incident took a day to surface and the reason it looked like a retention change rather than an outage.

Two attractive moves, both of which make this worse

Restarting the Loki and Tempo pods. The last object-store incident this team had was a rotated static access key that needed a restart to pick up, so the muscle memory is strong. Here it is actively harmful. The deny is evaluated per request, so a restart changes nothing about the 403s — and it destroys the ingester window, which is currently the only part of the read path that still works. For the duration of the WAL replay, the platform goes from “readable for the last two hours” to “not readable at all”, during an incident where people are trying to read.

Failing over to the replica bucket. The runbook for object storage says to fail over when a bucket is unavailable, and this ticket is titled as an availability problem. But the replica is reached through the same role, so the same deny applies and the failover changes nothing except which stale copy you are being refused. Even if a separate identity existed, the replica’s contents lag the source by the replication lag, so the move would trade a read outage for a read outage against older data — and it would introduce a second, harder incident inside the first one at the point where you have to switch back.

Both moves share a shape worth naming: they are responses to the ticket title rather than to the evidence. The evidence says two operations succeed and two are refused, and nothing in either move addresses that.

Resolution

Announce that no pods are to be restarted and no bucket failover is to be attempted. Both are already being suggested; say why they are wrong once, in the channel, so it does not have to be relitigated.
State the working picture explicitly: Prometheus is intact and trustworthy, Loki and Tempo are readable only inside the ingester window, and an empty Loki result for yesterday means "cannot read" rather than "nothing was logged". This matters because other people are drawing conclusions from those queries right now.
Restore s3:GetObject and s3:DeleteObject to the observability-storage role, scoped to the bucket ARN and the object ARNs beneath it, alongside the s3:PutObject and s3:ListBucket the consolidation kept.
Prove the grant before you look at the platform. Re-run aws iam simulate-principal-policy for all four actions against both the bucket ARN and an object ARN. If a query still fails after this returns allowed, the problem is in the data plane and not in the policy — that separation is the point of checking in this order.
Watch the 403 rate on loki_objstore_request_duration_seconds_count fall to zero without touching a single pod. This is both the confirmation of the fix and the proof that the cause was the policy rather than a cached credential.
Watch the compactor resume. It has three days of arrears and will issue a burst of LIST and DELETE against one prefix, which is the exact shape that earns a 503 SlowDown from the bucket. Watch the sweep rather than assuming it, and be ready to let it run over several cycles rather than forcing it.
Confirm the ruler reloads its rule groups. They are still in the bucket: the role lost the ability to read them and never had the ability to remove them, so nothing was deleted.
Only then fix the policy process. The technical fix takes one change; the reason a write-path template was applied to a read-and-write consumer takes a conversation, and it belongs after service is restored, not during.

Verification

aws iam simulate-principal-policy returns allowed for all four of s3:GetObject, s3:PutObject, s3:DeleteObject and s3:ListBucket, against both the bucket ARN and an object ARN beneath it. Both resources matter: ListBucket is granted on the bucket, the other three on the objects.
The rate of loki_objstore_request_duration_seconds_count with a 4xx status code is zero for every operation. Not lower — zero. A 403 against the object store has no normal steady state.
A Loki query over a range from two days ago returns lines. Pick the range deliberately so it cannot be answered from the ingester window; a query over the last hour proves nothing here and will pass even if the fix failed.
A Tempo trace ID captured before the incident began resolves and renders. Same reasoning: the trace has to be old enough that its block is in the bucket.
loki_compactor_oldest_processed_age_seconds is falling rather than climbing. This is the check that tells you retention is genuinely moving again rather than merely permitted.
The bucket object count begins to drop once retention_delete_delay has elapsed. The delay is deliberate — deletes are marked before they are executed — so do not read the gap as a failed sweep.
The ruler reports its rule groups loaded and is evaluating them. Confirm from the ruler, not from the alert firing, because an alert that is correctly not firing looks identical to one that cannot fire.
The new guard can fail. Remove s3:GetObject from the staging role and confirm the alert fires inside its for window. An alert that has only ever been green has never been tested.

Prevention

Alert on the object store’s response codes rather than on its availability. A 403 on any operation is never normal, so a rule on the 4xx rate split by operation catches this entire class within a scrape interval and names the missing action in the alert text. Availability checks would have stayed green for the whole incident, because the bucket was available.
Alert on the retention sweep separately. A compactor that cannot delete is invisible on every other panel until the bill arrives, and the bill is a three-day feedback loop. loki_compactor_oldest_processed_age_seconds climbing past a known ceiling is the cheapest signal available.
Run the policy simulator on a schedule and page on a denial. It is the only check that catches a bad policy before a reader does, it needs no access to the data plane, and it costs nothing:

# Substitute your own values before running:
ROLE_ARN=arn:aws:iam::123456789012:role/observability-storage
BUCKET=prod-observability

aws iam simulate-principal-policy \
  --policy-source-arn "$ROLE_ARN" \
  --action-names s3:GetObject s3:PutObject s3:DeleteObject s3:ListBucket \
  --resource-arns "arn:aws:s3:::$BUCKET" "arn:aws:s3:::$BUCKET/probe" \
  --query 'EvaluationResults[?EvalDecision!=`allowed`].EvalActionName' \
  --output text

Keep the per-component action table next to the policy, so the next person writing a least-privilege template has the requirement in front of them instead of inferring it from a sample of traffic:

{
  "Sid": "ObservabilityObjects",
  "Effect": "Allow",
  "Comment": "ingester PUT; querier and index-gateway GET; compactor GET+DELETE; ruler GET+PUT",
  "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
  "Resource": "arn:aws:s3:::prod-observability/*"
}

Split the read role from the write role. Separation of read and write makes rotation simpler and limits blast radius if a credential leaks, and it also means a single policy edit cannot remove both halves of the storage path at once.
Put the observability principals into the blast-radius review for IAM consolidations. The observability platform is a consumer of storage that nobody thinks to consult, and a permission boundary that cannot be reduced below the platform’s requirement is a stronger control than a review anyway.
Write down the asymmetry, because it is the part that made this quiet: reads and writes are separate actions and fail independently. A green ingestion dashboard is not evidence that the storage path is healthy, and this incident is the proof.

What if the consolidation had removed s3:PutObject instead?

The incident would have been far louder and far shorter.

Ingesters could not flush. In-memory chunks would accumulate, tempo_ingester_failed_flushes_total would rise immediately, and the WAL would grow because nothing could drain it. Memory pressure follows, then an OOM kill, and at that point the loss window is whatever the WAL cannot replay. Every ingestion dashboard would go red inside a scrape interval and somebody would be paged within minutes.

That version threatens data. This version does not — every write landed safely throughout. And yet this version ran for three days across four teams, because the visible half of the platform was healthy.

The lesson is not that read failures are worse than write failures. It is that the failure which threatens data announces itself, and the failure which merely makes the platform useless does not. Alerting has to be designed for the quiet one, because the loud one will find you regardless.

Why this matters

The ticket said the object store was unavailable and the object store was fine. That gap is the whole lesson.

Object storage in an observability platform is reached through three layers that fail independently: the network path to the bucket, the bucket itself, and the identity’s right to perform a specific action on a specific prefix. Only the third of those can fail for GET and not for PUT, and it is the one no availability check looks at. Teams instrument reachability because reachability is easy to instrument, then discover during an incident that reachability was never the question.

The second half of the lesson is about what the ingester window does to your sense of time. It gives every object-store fault a delay fuse of an hour or two, so the fault surfaces long after the change that caused it, in a component that did not change, filed by someone who did not make the change. Four tickets across four teams is the normal presentation, not an unusual one. When a platform breaks along a boundary rather than all at once, look for the thing that distinguishes the working half from the broken half — here, one is a write and the other is a read — and the cause is almost always a single edit on the far side of that boundary.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The store was never unavailable; the path to it was

2. Reads and writes are different actions, so the failure is asymmetric

3. Prometheus was fine because it is not in the bucket

Resolution

Verification

Prevention