S3 Permissions

A Loki cluster has been running for six months. The on-call engineer rotates the access key on the long-lived IAM user the platform team uses for Loki-to-S3 access. Within an hour, the ingester logs fill with 403 Forbidden lines. The compactor’s loki_compactor_oldest_processed_age_seconds metric stops advancing. The platform’s write path returns 500 to every push agent. The new access key was applied to the secret store, but the Loki pods were not restarted to pick it up. The on-call engineer takes an hour to find the cause.

This is the recurring shape of Loki-S3 incidents. Static credentials are an operational liability: they expire, they get rotated, they leak in environment variables. IAM roles — instance profiles on EC2, IRSA on EKS, pod identity on GKE — eliminate the liability. The IAM role shape is also the discipline that enforces tenant isolation at the storage layer: the bucket policy scopes each tenant’s principal to its own prefix.

What it is

Loki-to-S3 permissions are the IAM configuration that allows each Loki component to read and write the right objects. There are two surfaces:

• IAM role. The identity assumed by the Loki pod. On EC2 the role is attached as an instance profile. On EKS the role is associated with a Kubernetes ServiceAccount via IRSA (IAM Roles for Service Accounts). On GKE the role is bound via Workload Identity. The role carries the policy that grants the necessary S3 actions.
• Bucket policy. The resource-based policy attached to the bucket. The policy grants the IAM role the actions it needs, scoped to the prefixes the role should reach. The bucket policy is also where tenant isolation is enforced at the storage layer.

The two surfaces compose. The IAM role’s policy grants identity; the bucket policy grants resource access. Loki needs both. A policy on the role that allows s3:GetObject on * is necessary but not sufficient; the bucket policy must also allow it.

Why a sysadmin cares

A sysadmin cares because the IAM shape is the boundary between a working Loki and a Loki that silently degrades. The failure modes are specific:

• Long-lived access keys. A static AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY pair in a Kubernetes Secret. The key rotates; the secret store is updated; the pods are not restarted. Symptom: 403 Forbidden on every S3 call. The bucket bill is paid; the data is not accessible.
• Wildcard IAM policy. A policy that grants s3:* on arn:aws:s3:::*. The role can read every bucket in the account. A compromised Loki pod exfiltrates every bucket. The blast radius is total.
• No prefix-scoped bucket policy. A bucket policy that grants s3:GetObject on arn:aws:s3:::prod-loki-chunks/* to every tenant’s IAM role. Symptom: one tenant can list and read every other tenant’s chunks. The audit fails on the first control.
• Compactor missing delete permission. A policy that allows s3:GetObject and s3:PutObject but not s3:DeleteObject. Symptom: the compactor cannot enforce retention. Chunks older than retention_period are not deleted. The bucket grows without bound.

How it works

Five components reach the bucket. Each has a different permission profile.

   Loki deployment
   +----------------+   +----------------+   +----------------+
   | distributor    |   | ingester       |   | compactor      |
   | (no S3)        |   | PUT chunks     |   | GET/DELETE     |
   +----------------+   | GET on cold    |   | marker files   |
                        | tier reads     |   | + chunks       |
                        +----------------+   +----------------+
                              |                    |
                              v                    v
   +----------------+   +----------------+   +----------------+
   | index-gateway  |   | querier        |   | ruler          |
   | GET/PUT index  |   | GET chunks     |   | GET/PUT rules  |
   | files          |   | (cold tier)    |   |                |
   +----------------+   +----------------+   +----------------+
                              |                    |
                              v                    v
   +--------------------------------------------------------+
   | S3 bucket: prod-loki-chunks                            |
   |                                                        |
   | /tenant-a/<fingerprint>/<chunk>      <-- tenant prefix |
   | /tenant-b/<fingerprint>/<chunk>                        |
   | /index/<tenant>/<fingerprint>/<index>                  |
   | /markers/<tenant>/<day>/<marker>                       |
   | /ruler/<tenant>/<rule-group>/<rule>                    |
   +--------------------------------------------------------+

The permission profile per component:

The bucket policy applies at the bucket level. The IAM role’s policy applies to the principal. Both must grant the necessary action on the necessary resource.

How to configure it

A production setup uses IRSA on EKS or an instance profile on EC2. The shape is the same; the binding mechanism differs.

IRSA on EKS

# 1. Trust policy on the IAM role. Allows the EKS OIDC
#    provider to assume the role on behalf of the Loki
#    ServiceAccount in the loki namespace.
# 2. Permissions policy on the role (see below).
# 3. ServiceAccount annotation in the cluster.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: loki
  namespace: loki
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/loki-storage

The trust policy on the IAM role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716EXAMPLE:sub": "system:serviceaccount:loki:loki",
          "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716EXAMPLE:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Permissions policy on the role

The policy grants the actions each component needs, scoped to the Loki bucket and the Loki prefix.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LokiChunksReadWrite",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::prod-loki-chunks/*"
    },
    {
      "Sid": "LokiChunksList",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::prod-loki-chunks",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "",
            "tenant-*/*",
            "index/*",
            "markers/*",
            "ruler/*"
          ]
        }
      }
    }
  ]
}

Three production details:

• s3:DeleteObject is required for the compactor to enforce retention. A policy without it means the compactor cannot delete expired chunks. The bucket grows without bound.
• s3:ListBucket is required for the compactor and the querier to enumerate per-tenant prefixes. A policy without it means the compactor cannot find the chunks to delete.
• The Condition.StringLike on s3:prefix keeps the ListBucket scoped to Loki’s namespace in the bucket. A policy without the condition grants the role the ability to list the whole bucket, including prefixes belonging to other workloads.

Loki configuration

The Loki config binds the storage backend to the bucket. No static credentials are required when IRSA is in effect; the AWS SDK picks them up from the pod metadata.

# /etc/loki/config.yaml
common:
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1
    # access_key_id and secret_access_key are intentionally
    # absent. The AWS SDK reads the IRSA-injected credentials
    # from the pod's projected service account token.

For deployments that must use static keys (a non-AWS bucket, an on-premise MinIO), the credentials live in a Secret and are mounted as environment variables:

# /etc/loki/config.yaml (MinIO / non-AWS)
common:
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1
    access_key_id: ${MINIO_ACCESS_KEY_ID}
    secret_access_key: ${MINIO_SECRET_ACCESS_KEY}
    endpoint: minio.storage.svc.cluster.local:9000
    insecure: true

The static-credential shape is the fallback. The IRSA shape is the production default.

How to validate it

Six checks confirm the IAM shape is wired correctly.

# 1. READ-ONLY: confirm the Loki pod's role. Inside the
# pod, the metadata service returns the assumed role.
kubectl exec -n loki deploy/loki-ingester -- \
  curl -s http://169.254.170.2/v2/credentials \
  | jq -r '.RoleArn'
# expected: the role you configured. A missing or empty
# value means the ServiceAccount annotation is wrong.

# 2. READ-ONLY: confirm the bucket policy by listing
# objects as the Loki role. A successful listing is the
# first sign the policy works.
aws s3api list-objects-v2 \
  --bucket prod-loki-chunks \
  --prefix 'tenant-checkout/' \
  --max-items 1 \
  --query 'Contents[0].Key' \
  --profile loki-storage
# expected: a key under tenant-checkout/. An AccessDenied
# means the bucket policy is wrong.

# 3. READ-ONLY: confirm the compactor can delete. Simulate
# the compactor's request by attempting a DeleteObject on a
# test key.
aws s3api delete-object \
  --bucket prod-loki-chunks \
  --key 'tenant-checkout/test-object' \
  --profile loki-storage
# expected: 204 No Content. An AccessDenied means the role
# lacks s3:DeleteObject; the compactor cannot enforce
# retention.

# 4. READ-ONLY: confirm the ingester can write. Push a
# line and observe the chunk appear.
curl -s -H 'X-Scope-OrgID: tenant-checkout' \
  -X POST http://loki-distributor:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},
    "values":[["1700000000000000000","probe"]]}]}'
sleep 60
aws s3api list-objects-v2 \
  --bucket prod-loki-chunks \
  --prefix 'tenant-checkout/' \
  --query 'Contents[0].Key' \
  --profile loki-storage
# expected: a key under tenant-checkout/ after the ingester
# flushes. A missing key means the ingester cannot write.

# 5. READ-ONLY: confirm the bucket policy by querying the
# effective permissions for the Loki role.
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/loki-storage \
  --action-names s3:GetObject s3:PutObject s3:DeleteObject s3:ListBucket \
  --resource-arns arn:aws:s3:::prod-loki-chunks \
               arn:aws:s3:::prod-loki-chunks/tenant-checkout/test \
  --query 'EvaluationResults[*].EvalDecision'
# expected: allowed for every action and resource. A
# "denied" means the policy is missing an action.

# 6. READ-ONLY: confirm the bucket policy denies cross-
# tenant access. Attempt to read a tenant-a object as
# tenant-b's role (only relevant if you have per-tenant
# roles).
aws s3api get-object \
  --bucket prod-loki-chunks \
  --key 'tenant-a/some-object' \
  /tmp/check \
  --profile loki-tenant-b
# expected: AccessDenied. A successful read means the
# bucket policy is too permissive.

How it can fail

Six failure shapes cover the recurring Loki-S3 incidents.

1. Static access key rotated, pods not restarted. Symptom: 403 Forbidden in the ingester logs. The fix is to restart the pods after a key rotation. The discipline is to use IAM roles so the key is not stored in the cluster.
2. Wildcard IAM policy. Symptom: a compromised Loki pod can read every bucket in the account. The fix is to scope the policy to the Loki bucket and the Loki prefix.
3. Missing s3:DeleteObject. Symptom: the compactor cannot enforce retention. Chunks older than retention_period are not deleted. The bucket grows without bound. The metric loki_compactor_oldest_processed_age_seconds may not reflect the actual retention state because the compactor’s retention sweep is failing.
4. Missing s3:ListBucket. Symptom: the compactor cannot enumerate per-day marker files. The retention sweep stalls. The fix is to add s3:ListBucket to the role.
5. Cross-tenant bucket policy. Symptom: one tenant’s role can read every other tenant’s objects. The fix is prefix scoping in the Resource element.
6. Wrong OIDC audience. Symptom: IRSA stops working after an EKS upgrade or a ServiceAccount change. The pod logs show 403 Forbidden on STS. The fix is to re-verify the trust policy’s aud and sub conditions.

How to troubleshoot it

The diagnostic order for an S3-permission failure:

1. What is the S3 error code? The Loki log entry contains the AWS error code. AccessDenied is a policy problem; NoSuchBucket is a name problem; NoSuchKey is a lifecycle problem.
2. Is the role assumed correctly? curl http://169.254.170.2/v2/credentials inside the pod returns the assumed role. A missing or empty value means the IRSA annotation is wrong.
3. Does the role have the right policy? aws iam get-role-policy --role-name loki-storage --policy-name LokiStorage. Compare with the documented policy.
4. Does the bucket policy grant the action? aws s3api get-bucket-policy --bucket prod-loki-chunks. The Statement element is the source of truth for resource- scoped access.
5. Does the compactor specifically have s3:DeleteObject? The compactor’s retention sweep fails silently when DeleteObject is missing.
6. Is the bucket region correct? aws s3api get-bucket- location --bucket prod-loki-chunks. A wrong region causes 301 redirects and intermittent 403s.

Security implications

The S3 permission shape is a security boundary. The boundary has three faces:

• Loki pod to bucket. The IAM role must be scoped to the Loki bucket and the Loki prefix. A wildcard role lets a compromised Loki pod reach every bucket in the account.
• Tenant to tenant at the bucket layer. The bucket policy must scope each tenant’s IAM role to its own prefix. A wildcard Resource defeats the tenant isolation the API layer enforces.
• Operator to bucket. The bucket policy must not grant human principals wildcard access. Audit access uses scoped read-only roles with s3:GetObject and s3:ListBucket on a defined prefix.

The discipline is least privilege: every action is necessary; every Resource is the smallest possible. The policy is reviewed in the same pull request as the Loki config change.

Performance implications

The S3 permission shape has negligible performance implication. The AWS request signing adds microseconds per request; the bucket policy evaluation adds microseconds. The performance cost of a misconfigured shape is also negligible on the happy path; the cost shows up only when the policy is wrong and the requests fail.

The performance cost of an over-permissive shape is paid by the bucket: a wildcard s3:ListBucket lets the compactor list the entire bucket every cycle, which is many thousands of objects. The fix is the StringLike condition on s3:prefix.

Production guidance

• Use IRSA on EKS, instance profiles on EC2, Workload Identity on GKE. Never store long-lived access keys in the cluster.
• Scope the IAM role policy to the Loki bucket and the Loki prefix. s3:* on arn:aws:s3:::* is a security incident waiting to happen.
• Scope the bucket policy to per-tenant prefixes. The storage-layer tenant isolation depends on the prefix in the Resource element.
• Grant s3:DeleteObject only to the compactor. A separate role for the compactor is the cleanest shape; a single role with the action is acceptable for smaller deployments.
• Monitor loki_request_duration_seconds for status_code="403" on the ingester and the compactor. A non-zero rate is either a misconfigured policy or an attacker.
• Pair the S3 permission shape with the bucket versioning and lifecycle lessons. Versioning is the recovery story from a wrong retention sweep.

Verification

You should now be able to answer:

• Which S3 actions does the compactor require that the ingester does not, and why?
• How does IRSA replace a static access key, and what is the operator’s responsibility on key rotation?
• Why is a wildcard IAM policy a security incident rather than a minor over-permission?
• What is the symptom when the bucket policy grants s3:GetObject on arn:aws:s3:::prod-loki-chunks/* to every tenant role?
• What is the difference between the IAM role’s policy and the bucket’s policy, and why does Loki need both?

Quiz