Tempo Object Storage Security

A Tempo cluster runs for nine months without incident. The cloud team migrates IAM roles from a permissive developer policy to a least-privilege policy. The Tempo pods are still using the previous role ARN; the new policy does not grant s3:PutObject to that principal. The next morning, the ingester has not flushed a block for twelve hours. Distributors are still accepting writes; the in-memory head block has long since reached its limit; tempo_distributor_dropped_spans_total has been rising for two hours. The metric dashboard is green because no alert fires on a slow drain. The data is sitting in the WAL, with no place to go.

Object storage is the only persistent copy of the data. The IAM permissions, the bucket policy, and the encryption posture are the security perimeter, not the configuration knob.

What Tempo object-storage security means

A Tempo deployment writes trace blocks to one bucket and (in deployments with the metrics-generator enabled) service-graph metrics to a second bucket. Each bucket has three sides:

• Tempo side. The storage.trace.s3 (or gcs, or azure) config block. Holds the credentials (or the workload-identity reference) and the bucket coordinates.
• Cloud IAM side. The policy attached to the IAM role, service account, or managed identity Tempo assumes. Grants the specific actions on the bucket and prefix.
• Bucket side. The bucket policy (S3), the IAM policy (GCS), or the storage-account policy (Azure). Denies what is not allowed; refuses non-TLS; enforces the encryption posture.

Each side enforces part of the boundary. A misconfiguration on any single side compromises the bucket.

   Tempo pod
       |
       |  IRSA / Workload Identity / access keys
       v
   +-------------------+      +-------------------+
   |   IAM policy      | ---> |   Bucket policy   |
   | (grants actions)  |      |  (denies others)   |
   +-------------------+      +-------------------+
                                       |
                                       v
                                +-----------------+
                                |   Bucket        |
                                |  trace blocks   |
                                |  KMS-encrypted  |
                                +-----------------+

Why a sysadmin cares

Three operational pains motivate the discipline.

1. The bucket is the only persistent copy of the data. A leaked AWS key with s3:PutObject and s3:GetObject on the Tempo bucket is a complete trace-data breach. The leak is durable, queryable, and reflective of every attribute the application emitted.
2. The IAM migration is the failure shape. Least-privilege migrations are routine. The platform team is usually not on the IAM change-management list. A Tempo cluster that worked for nine months has a brief outage when the IAM policy changes; the on-call engineer needs to know what IAM action to ask for.
3. Multi-tenant isolation runs through IAM. Tempo writes under the prefix, operator, tenant path. A least-privilege policy that scopes s3:PutObject and s3:GetObject to the specific prefix prevents a leaked credential from listing other tenants.

How it works — the permission shape

Tempo’s AWS SDK calls the bucket with the following actions:

• s3:ListBucket — on every search and every compactor pass, to enumerate the relevant prefix.
• s3:GetObject — on every querier lookup, to fetch the block data.
• s3:PutObject — every ingester flush, every compactor pass, every metrics-generator scrape.
• s3:DeleteObject — the compactor’s lifecycle eviction of merged blocks; not used by the ingester or querier.

These four actions are the working set. Anything additional is over-permissioned. Anything missing produces a specific symptom.

How to configure it

S3 on EKS: IRSA

# Service account annotation. Tempo assumes the role
# GrafanaObservability-TempoS3.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tempo
  namespace: observability
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/GrafanaObservability-TempoS3

The IAM trust policy allows Tempo’s service account to assume the role:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLEDOCID"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        ":sub": "system:serviceaccount:observability:tempo"
      }
    }
  }]
}

The IAM policy attached to that role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucketOnPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::tempo-traces-prod",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["blocks/single-tenant/*"]
        }
      }
    },
    {
      "Sid": "GetPutDeleteOnObjects",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::tempo-traces-prod/blocks/single-tenant/*"
    },
    {
      "Sid": "KMSDecryptOnly",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/abcd-...-wxyz"
    }
  ]
}

The Tempo config consumes the IRSA automatically; no static keys are set in YAML:

# /etc/tempo/tempo.yaml
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-prod
      region: us-east-1
      # No access_key / secret_key: the SDK uses the pod's
      # IRSA-resolved session token from the EC2 IMDS.
      # SSE-KMS is enforced by the bucket policy, not the config.

Severity: CONFIGURATION. Restart the ingester, querier, and compactor for new credentials to take effect.

Bucket policy: enforce TLS and deny outside the VPC

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::tempo-traces-prod",
        "arn:aws:s3:::tempo-traces-prod/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    },
    {
      "Sid": "AllowTempoRoleOnly",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/GrafanaObservability-TempoS3"
      },
      "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": [
        "arn:aws:s3:::tempo-traces-prod",
        "arn:aws:s3:::tempo-traces-prod/*"
      ]
    },
    {
      "Sid": "RequireSSEKMS",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::tempo-traces-prod/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        }
      }
    }
  ]
}

The three statements:

• Deny any non-TLS request to the bucket. A misconfigured client sending cleartext S3 hits the deny and gets PermanentRedirect / AccessDenied.
• Allow only the Tempo role. Even with IAM otherwise permissive, the bucket policy restricts who can touch the prefix.
• Deny any PutObject without SSE-KMS. A client that forgets the encryption header fails the upload.

GCS on GKE: Workload Identity

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tempo
  namespace: observability
  annotations:
    iam.gke.io/gcp-service-account: tempo-sa@tempo-prod.iam.gserviceaccount.com

The Tempo config:

storage:
  trace:
    backend: gcs
    gcs:
      bucket_name: tempo-traces-prod
      enable_workload_identity: true

The GCS IAM policy:

bindings:
- members:
  - serviceAccount:tempo-prod.svc.id.goog[tempo]/tempo
  role: roles/storage.objectUser
- members:
  - serviceAccount:tempo-prod.svc.id.goog[tempo]/tempo
  role: roles/storage.legacyBucketReader

Azure Blob on AKS: Workload Identity

storage:
  trace:
    backend: azure
    azure:
      container_name: tempo-traces
      storage_account_name: tempotracesprod
      use_managed_identity: true
      endpoint_suffix: core.windows.net

The AKS federated credential maps the Tempo service account to an Azure managed identity; the managed identity has the Storage Blob Data Contributor role on the storage account.

How to validate it

Severity: READ-ONLY.

# 1. Confirm the Tempo pod is using the IRSA role.
kubectl -n observability exec deploy/tempo -- \
  env | grep -E 'AWS_ROLE|AWS_WEB_IDENTITY'
# AWS_ROLE_ARN=arn:aws:iam::123456789012:role/GrafanaObservability-TempoS3
# AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

# 2. Assume the same role from a CI runner and confirm the policy
# actually grants the four actions.
aws sts assume-role-with-web-identity \
  --role-arn arn:aws:iam::123456789012:role/GrafanaObservability-TempoS3 \
  --role-session-name tempo-test \
  --web-identity-token "$(cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)" \
  > /tmp/tempo-creds.json

AWS_ACCESS_KEY_ID=$(jq -r .Credentials.AccessKeyId /tmp/tempo-creds.json) \
AWS_SECRET_ACCESS_KEY=$(jq -r .Credentials.SecretAccessKey /tmp/tempo-creds.json) \
AWS_SESSION_TOKEN=$(jq -r .Credentials.SessionToken /tmp/tempo-creds.json) \
  aws s3 ls s3://tempo-traces-prod/blocks/single-tenant/ --max-items 1
# 2026-08-12 17:02:14          0 1e0a6b2f-...

# 3. Confirm the deny rules actually fire. Same role, cleartext.
AWS_ACCESS_KEY_ID=$(jq -r .Credentials.AccessKeyId /tmp/tempo-creds.json) \
AWS_SECRET_ACCESS_KEY=$(jq -r .Credentials.SecretAccessKey /tmp/tempo-creds.json) \
AWS_SESSION_TOKEN=$(jq -r .Credentials.SessionToken /tmp/tempo-creds.json) \
  aws --no-ssl s3 ls s3://tempo-traces-prod/ --max-items 1
# PermanentRedirect: bucket policy requires SecureTransport: true.

# 4. Confirm an upload without SSE-KMS is rejected.
echo "test" > /tmp/test-block
AWS_ACCESS_KEY_ID=$(jq -r .Credentials.AccessKeyId /tmp/tempo-creds.json) \
AWS_SECRET_ACCESS_KEY=$(jq -r .Credentials.SecretAccessKey /tmp/tempo-creds.json) \
AWS_SESSION_TOKEN=$(jq -r .Credentials.SessionToken /tmp/tempo-creds.json) \
  aws s3 cp /tmp/test-block s3://tempo-traces-prod/blocks/single-tenant/test
# AccessDenied: PutObject requires SSE-KMS.

# 5. Confirm Tempo can write a block end-to-end (the metric dashboard).
curl -s http://tempo:3200/metrics | \
  grep -E 'tempo_ingester_failed_flushes_total|tempo_ingester_blocks_flushed_total'
# tempo_ingester_failed_flushes_total 0
# tempo_ingester_blocks_flushed_total 47821

A clean validation: the Tempo pod carries the IRSA annotation, the assumed role grants the four actions, the deny rules fire on the right requests, and the ingester’s flush metrics show successful flushes with no failures.

How it can fail

Four recurring shapes from real incidents.

1. An IAM migration strips s3:PutObject. A platform IAM consolidation replaces permissive roles with least-privilege ones; Tempo’s principal loses s3:PutObject. Symptom: tempo_ingester_failed_flushes_total rises; distributor_dropped_spans_total rises once the head block fills. Fix: add the four actions back to the Tempo principal.
2. A KMS key rotation is missed. The bucket policy allows KMS access via the old key ARN; the key is rotated but Tempo’s role retains the old ARN as a Resource. Symptom: KMS access denied on every upload. Fix: update the IAM resource to the new key ARN; the rotation is otherwise transparent.
3. A cross-account trust becomes a public trust. A dev sets the bucket policy to "Principal": "*" for testing and forgets to remove it. Symptom: a public scanner finds the bucket; every trace is enumerable. Fix: the bucket policy should never be Principal: * for production data; restore to the specific role ARN.
4. The metrics-generator bucket shares the trace bucket. A copy-paste config sets storage.metrics.s3.bucket to the same bucket as storage.trace.s3.bucket. Symptom: the metrics-generator writes a small object next to the trace blocks; the compactor sees the foreign object; the bucket-listing scan slows. Fix: the metrics bucket must be a separate bucket with its own policy and its own encryption key.

How to troubleshoot it

The diagnostic order:

1. Is the bucket reachable from the Tempo pod? kubectl exec into the pod and run aws s3 ls s3://... using the IRSA credentials. A failure here is the network or DNS.
2. Are the credentials valid? aws sts get-caller-identity. An expired STS session points at the IRSA webhook; pod restart resolves it.
3. Does the role grant the four actions? Inspect the IAM policy attached to the role. s3:PutObject, s3:GetObject, s3:DeleteObject on the bucket ARN.
4. Is the bucket policy denying an action? A deny rule on aws:SecureTransport or on the missing SSE header fails the request even when IAM allows it.
5. Is the KMS key accessible? aws kms describe-key on the key ARN. A deny here is the root cause of a Put failure.

Security implications

• The bucket is the trace store. The same regulatory rules that apply to the application database apply to the bucket. Encryption at rest, TLS in transit, scoped access.
• Workload identity first. IRSA / Workload Identity over static keys. The IAM session expires hourly.
• Bucket policy layers on top of IAM. The bucket policy denies what IAM does not see: cleartext requests, outside the VPC, without the SSE header.
• Per-prefix scoping. A multi-tenant Tempo bucket scopes the IAM Resource to the prefix; the credentials cannot read another tenant’s blocks.
• Separate metrics bucket. The metrics-generator trust boundary is different from the trace trust boundary; the buckets do not share a key.

Performance implications

• KMS encrypt/decrypt cost. KMS adds roughly 1-2 ms per object; a block upload is a single object. Negligible per span; ~$0.03 per 10 000 PutObject requests at current pricing.
• Cross-region transfer cost. S3 cross-region adds 2c/GiB transfer. A 5 TiB bucket queried from a different region per Grafana click is a five-figure monthly bill.
• Workload identity token refresh. The STS session refresh is transparent; one request per hour is the only cost.

Production guidance

• Use IRSA on EKS, Workload Identity on GKE/AKS. No long-lived AWS keys in a Kubernetes Secret.
• Two buckets, two encryption keys, two IAM roles. The metrics-generator is a separate trust boundary.
• Layer the bucket policy: deny non-TLS, deny outside the VPC endpoint, require SSE-KMS. Even a permissive IAM role is bounded by the bucket policy.
• Right-size the KMS key. A single CMK with rotate-on annual and a kms:GenerateDataKey grant for the Tempo role is enough.
• Audit IAM changes on the runbook. Tempo belongs on the IAM change-management rota because an IAM migration affects Tempo.

Verification

You should now be able to answer:

• Which four IAM actions are the working set for a Tempo ingester, querier, and compactor?
• What is the trust-binding shape of IRSA, and why is it preferable to static AWS keys?
• Why is the metrics-generator bucket a separate trust boundary from the trace bucket?
• What does the bucket policy deny that the IAM policy allows through?
• Which Tempo metric surfaces the bucket-write failure?

Quiz