Slow Object Storage

A Mimir query returns context deadline exceeded after 30 seconds. Loki’s querier logs are full of objstore: operation timeout. Tempo traces have empty spans. The Thanos Compactor has not made progress in two hours. None of these components are at fault. The bucket is at fault. A cross-region S3 bucket, a misconfigured proxy, or a cold-storage tier without a recent restore has put the four most expensive operations behind a slow link.

This lesson is about slow object storage: the bucket that sits behind every modern metrics, logs and traces platform, and the diagnostic order that gets from “queries are slow” to “the LIST call is the problem”.

What it is

Slow object storage is the condition where reads and writes against the metrics, logs or traces bucket exceed the platform’s expected latency. The most expensive operation is almost always LIST — the call that enumerates the objects in a prefix. LIST returns up to 1000 keys per request and may paginate through millions of objects to build a complete answer.

The bucket is the largest shared dependency in any modern observability stack. Thanos, Mimir, Loki and Tempo all store their durable data as objects in a bucket. A slow bucket degrades every component that touches it.

Why a sysadmin cares

Three production pains concentrate in slow object storage:

1. Query timeouts. Range queries that span hours must fetch the blocks or chunks that contain the relevant time range. A slow bucket means the fetch dominates the query budget and the query times out.
2. Compactor stalls. The compactor must LIST blocks to find what to merge. A slow LIST means the compactor cannot enumerate the bucket; the merge queue grows; the on-disk footprint doubles.
3. Ingest stalls on write failures. Recent samples and logs are written to the bucket. A bucket that is slow on PUT will eventually reject writes, and the platform begins to buffer. When the buffer fills, the platform starts dropping data.

The lesson is that the bucket is the platform’s storage substrate. Treat it as a first-class dependency with its own monitoring, its own alerting and its own capacity plan.

How it works

The four operations an object-storage client issues against an S3-compatible bucket are GET, PUT, HEAD and LIST. Each one has a different cost profile:

   client (Thanos Store, Loki querier, Mimir store-gateway)
        |
        v
   +-----------------+
   |  GET            |  --  fetch one object by key
   +-----------------+     cost: O(1) per call
        |
        v
   +-----------------+
   |  PUT            |  --  write one object
   +-----------------+     cost: O(1) per call
        |
        v
   +-----------------+
   |  HEAD           |  --  metadata for one object
   +-----------------+     cost: O(1) per call
        |
        v
   +-----------------+
   |  LIST           |  --  enumerate keys with a prefix
   +-----------------+     cost: O(N) per call

GET and PUT are usually fast — sub-100 ms for a same-region bucket. LIST is the dangerous one: it returns up to 1000 keys per call and paginates through the entire prefix. A bucket with 10 million blocks requires 10 000 LIST calls to enumerate. A bucket with 100 million chunks requires more.

The components that depend on LIST include the Thanos Compactor (which must enumerate all blocks to schedule merges), the Thanos Store gateway (which must enumerate the bucket for each tenant during a query), the Mimir store-gateway, the Loki compactor and the Loki querier. A slow LIST affects all of them simultaneously.

How to configure it

Slow object storage is mitigated at the bucket layer, the network layer and the cache layer. All three matter.

# Severity: CONFIGURATION
# /etc/thanos/store.yml
type: S3
config:
  bucket: "thanos-prod-eu-west-1"
  endpoint: "s3.eu-west-1.amazonaws.com"
  region: "eu-west-1"
  access_key: "${AWS_ACCESS_KEY_ID}"
  secret_key: "${AWS_SECRET_ACCESS_KEY}"
  http_config:
    idle_conn_pool_enabled: true
    max_idle_connections: 100
    timeout: 30s

# Severity: CONFIGURATION
# /etc/loki/config.yml
chunk_store_config:
  chunk_cache_config:
    enable_fifocache: true
    fifocache:
      max_size_bytes: 4GB
      ttl: 24h
  query_range_config:
    results_cache_config:
      cache:
        enable_fifocache: true
        fifocache:
          max_size_bytes: 4GB

Three rules govern these knobs:

1. The bucket must be in the same region as the query nodes. Cross-region traffic costs tens of milliseconds per call. A 30-second query that issues 100 calls pays 3 seconds of pure network latency.
2. The index cache must be sized to the working set. A 4 GB cache holds roughly 4 GB of index entries. A bucket with 10 TB of data benefits from a 10 GB cache.
3. The chunk cache must be sized to the recent working set. A 4 GB chunk cache holds roughly 4 GB of chunks. Loki’s chunk size defaults to 1.5 MB; a 4 GB cache holds roughly 2700 chunks.

How to validate it

Validation is read-only. Confirm the bucket latency, the cache hit ratio and the operation breakdown:

# Severity: READ-ONLY
rclone test object s3:thanos-prod-eu-west-1/01HXXX/ --s3-region eu-west-1

The expected result is a sub-second response. Anything above 1 second for a single GET is a sign the bucket is not in the same network.

The platform metrics tell the same story from the running process:

# Severity: READ-ONLY
histogram_quantile(0.99,
  sum by (operation) (rate(objstore_bucket_operation_duration_seconds[5m]))
)
sum by (operation) (rate(objstore_bucket_operation_failures_total[5m]))

A healthy platform shows GET p99 under 200 ms and LIST p99 under 1 second. Anything above those numbers is a slow bucket.

The cache hit ratio:

# Severity: READ-ONLY
sum(rate(thanos_store_index_cache_hits_total[5m])) /
  (sum(rate(thanos_store_index_cache_hits_total[5m]))
   + sum(rate(thanos_store_index_cache_misses_total[5m])))

A healthy platform shows a hit ratio above 0.95 for the index cache and above 0.80 for the chunk cache. Anything below 0.50 means the cache is too small for the workload.

How it can fail

Six failure shapes account for nearly every slow-object- storage incident:

1. Cross-region bucket. The bucket is in us-east-1 and the query nodes are in eu-west-1. Symptom: every GET and LIST pays tens of milliseconds of cross-region latency.
2. Cold-storage bucket class. A bucket transitioned to S3 Glacier or equivalent for cost reasons. Symptom: GET calls return minutes later or fail with SlowDown.
3. Mis-sized index cache. The cache is set to 256 MB but the working set is 4 GB. Symptom: cache hit ratio below 0.50 and LIST calls dominate the query budget.
4. Network between query node and bucket. A firewall rule, an MTU mismatch, or a misconfigured proxy adds 100 ms to every call. Symptom: GET p99 above 500 ms even on a same-region bucket.
5. Bucket throttling. The bucket provider has a per- prefix request rate. A high-cardinality tenant pushes the prefix past the rate. Symptom: 503 SlowDown on a fraction of GET and LIST calls.
6. LIST pagination with millions of objects. A single LIST returns 1000 keys. A bucket with 10 million blocks requires 10 000 LIST calls per enumeration. Symptom: the compactor’s LIST cycle takes minutes instead of seconds.

How to troubleshoot it

The diagnostic order is consistent across all six failure shapes:

1. Confirm the symptom is the bucket. Inspect objstore_bucket_operation_duration_seconds by operation. If GET p99 is high, the bucket is the problem. If GET p99 is fine and LIST p99 is high, the bucket is the problem but the workload is the trigger.
2. Identify the slow operation. Inspect the histogram by operation label. Compare GET, PUT, HEAD and LIST.
3. Inspect the cache hit ratio. If the cache hit ratio is low, the cache is too small. Raise it.
4. Inspect the bucket region. Confirm the bucket region and the query node region are identical. Cross- region traffic is the single most common cause.
5. Inspect the bucket class. Confirm the bucket is on the standard class. Cold classes cost minutes per GET.
6. Inspect the network. mtr or tcping from the query node to the bucket endpoint. Round-trip should be sub-10 ms.

Security implications

The bucket contains every metric, log or trace the platform has retained. Three disciplines matter in production:

1. Bucket credentials must be least-privilege. A service account that can only read the metrics prefix is preferable to one that can read and write the entire bucket.
2. The bucket must encrypt at rest. Server-side encryption with a managed key is the default on every major provider; verify it is enabled.
3. Cross-account access must be explicit. A bucket that is reachable from another account or another VPC is a leak; restrict access via bucket policy or VPC endpoint.

A leaked bucket credential is a full disclosure of the platform’s history. Treat the credential like a database password.

Performance implications

Bucket cost is a function of three variables: operation count, operation size, and round-trip latency. The arithmetic is:

query_bucket_cost_seconds
    = (GET_count * GET_latency)
    + (LIST_count * LIST_latency)
    + (HEAD_count * HEAD_latency)

A platform that issues 100 GETs and 10 LISTs at 50 ms each pays roughly 5.5 seconds per query. Halving the GET latency halves the cost. Halving the LIST count through better indexing halves the cost. The cache sitting in front of the bucket is the single largest lever.

Verification

You should now be able to answer:

• What are the four operations object-storage clients usually issue, and which one is most expensive at scale?
• Which metric exposes per-operation bucket latency, and what p99 threshold for GET indicates a slow bucket?
• Why is a same-region bucket critical for query latency, and how would you verify the bucket region?
• What does an empty index cache cost a Thanos Store gateway on a range query?
• How would you size the chunk cache for Loki?

Quiz