Tempo Hardening Checklist

A platform team runs an audit for an upcoming PCI-DSS assessment. The cluster has Tempo with auth_enabled: false, a TLS-less OTLP port reachable from the cluster pod network, an IAM role with s3:* on the bucket, an OTel Collector with no attributes/redact processor, no X-Scope-OrgID injection at the proxy, and an S3 bucket that allows Principal: *. The auditors flag six findings in the first hour. The remediation is six weeks of changes coordinated across three teams (platform, ops, security), with the platform taking the lead because the data path runs through Tempo. The cost is six figures in engineering time and a delayed audit cycle.

The list of failures above reads like a “do not ship” list. The lesson is to make it a list to not skip during the deploy. The hardening checklist turns the regulator’s finding into a pre-deployment gate.

What the Tempo hardening checklist is

The checklist is the union of six configuration decisions that must all be set, plus a monitoring baseline that catches a regression. Six surfaces, six “must-have” controls:

1. Authentication. auth_enabled: true plus an OIDC-aware proxy that injects X-Scope-OrgID.
2. Tenancy. Per-tenant rate limits under distributor.limits.*; per-prefix scoping on the bucket.
3. TLS. Certs at the receiver or at the proxy; never plaintext on the OTLP port.
4. Attribute redaction. attributes/redact with an allowlist in the OTel Collector pipeline, before batch.
5. Object-storage IAM. IRSA / Workload Identity; least- privilege policy; SSE-KMS; deny non-TLS at the bucket.
6. Monitoring-of-the-observability. Metrics that fire when any of the above regresses.

The checklist turns six independent decisions into a single artefact. The artefact is committed to the repository; the CI runs the audit before every release.

   +-----------------------------------+
   |  Production security pre-flight  |
   +-----------------------------------+
       |              |               |
   auth_enabled     TLS at          attributes/redact
   true + proxy    receiver         in collector
       |              |               |
       v              v               v
   tenancy         certs            allowlist
   limits          rotated          committed
       |              |               |
       +------+-------+-------+-------+
              |
              v
   +-----------------------------------+
   |  Object storage                   |
   |  IRSA / Workload Identity         |
   |  least-privilege IAM              |
   |  SSE-KMS                          |
   |  bucket policy denials            |
   +-----------------------------------+
              |
              v
   +-----------------------------------+
   |  Monitoring                       |
   |  tempo_distributor_* metrics      |
   |  auth disabled alert              |
   |  cert expiry alert                |
   |  attribute leak audit             |
   +-----------------------------------+

Why a sysadmin cares

Five reasons the checklist matters.

1. Compliance. PCI, HIPAA, GDPR, SOC 2 — every regime that touches trace data requires “appropriate technical measures.” The checklist is the artefact the auditor points to.
2. Blast radius. A misconfiguration on any one of the six surfaces can put the entire fleet’s trace data at risk. Each failure amplifies the others.
3. Onboarding speed. A new service has a single checklist to run; the security team has a single review to do; the audit has a single document to examine.
4. Regression detection. A “we set it and forgot it” Tempo slowly returns to its dev defaults under Kubernetes manifest churn. The checklist is the periodic re-audit.
5. Operational discipline. A team that runs the checklist on every release has fewer surprise pages.

How it works — the surfaces in detail

Surface 1 — authentication and tenancy

# /etc/tempo/tempo.yaml
auth_enabled: true

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 127.0.0.1:4317
          tls:
            cert_file: /etc/tempo/tls/tempo.crt
            key_file:  /etc/tempo/tls/tempo.key
        http:
          endpoint: 127.0.0.1:4318
          tls:
            cert_file: /etc/tempo/tls/tempo.crt
            key_file:  /etc/tempo/tls/tempo.key
  auth_context:
    extractors:
      - name: org-id
        from: header
        key: X-Scope-OrgID
  limits:
    ingestion_rate_limit_bytes: 10485760   # 10 MiB/sec/tenant
    ingestion_burst_size_bytes: 20971520
    max_traces_per_user:         10000
    max_bytes_per_trace:         5242880

The proxy in front injects the tenant from the user’s identity. Tempo loopback-binds the OTLP ports; only the proxy reaches them.

Surface 2 — TLS at the boundary

# See Surface 1 above. Alternative: terminate at the proxy
# with nginx and let Tempo speak plain HTTP on loopback.
# Both shapes are acceptable. Pick one and pin it.

Surface 3 — attribute redaction at the OTel Collector

# /etc/otelcol/config.yaml
processors:
  attributes/redact:
    actions:
      - key: http.route
        action: retain
      - key: http.method
        action: retain
      - key: http.status_code
        action: retain
      - key: service.name
        action: retain
      - key: service.version
        action: retain
      # ... the full allowlist, see lesson 04
      - key: user.email
        action: delete
      - key: request.body
        action: delete
      - key: db.statement
        action: delete
      - key: http.url
        action: hash

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [attributes/redact, transform/policy, batch]
      exporters:  [otlp/tempo]

Surface 4 — object-storage IAM

# Service account annotation for IRSA on EKS.
# tempo-pod assumes GrafanaObservability-TempoS3.
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces-prod
      region: us-east-1
      # access_key / secret_key not present: SDK uses IRSA.

The IAM policy on GrafanaObservability-TempoS3:

{
  "Effect": "Allow",
  "Action": [
    "s3:ListBucket"
  ],
  "Resource": "arn:aws:s3:::tempo-traces-prod",
  "Condition": {
    "StringLike": { "s3:prefix": ["blocks/single-tenant/*"] }
  }
}

Surface 5 — bucket policy

{
  "Sid": "DenyInsecureTransport",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": [
    "arn:aws:s3:::tempo-traces-prod",
    "arn:aws:s3:::tempo-traces-prod/*"
  ],
  "Condition": {
    "Bool": { "aws:SecureTransport": "false" }
  }
}

Surface 6 — monitoring

# Prometheus alerts
groups:
- name: tempo-hardening
  rules:
  - alert: TempoAuthDisabled
    expr: tempo_distributor_reachable_ingesters == 0
    # Indirect signal; combine with a config audit.

  - alert: TempoCertExpiresSoon
    expr: (tempo_tls_cert_not_after - time()) < 86400
    for: 5m
    labels: { severity: page }
    annotations:
      summary: 'Tempo certificate expires in less than one day'

  - alert: TempoEmptyDroppedSpansRising
    expr: rate(tempo_distributor_dropped_spans_total[5m]) > 0
    labels: { severity: warn }

  - alert: TempoIngestFlushesFailing
    expr: rate(tempo_ingester_failed_flushes_total[5m]) > 0
    labels: { severity: page }

  - alert: TempoBucketAccessDenied
    expr: rate(tempo_ingester_failed_flushes_total{reason="AccessDenied"}[5m]) > 0
    labels: { severity: page }

How to configure it — the full audit

A single audit script that confirms every surface:

#!/usr/bin/env bash
# /usr/local/bin/tempo-hardening-audit
# Severity: READ-ONLY.

set -uo pipefail
failures=()

# 1. Authentication.
if curl -fsS http://tempo:3200/api/traces/00000000000000000000000000000000 \
   -o /dev/null 2>&1; then
  failures+=("auth_unauthenticated_returns_2xx")
fi

# 2. TLS at the OTLP receiver.
if ! openssl s_client -connect tempo:4317 \
     -CAfile /etc/tempo/tls/internal-ca.crt </dev/null 2>&1 \
     | grep -q 'Verification: OK'; then
  failures+=("otlp_tls_handshake_failed")
fi

# 3. Per-tenant limits active.
curl -s http://tempo:3200/metrics | grep -q 'tempo_distributor_ingester_spans_received_total' \
  || failures+=("distributor_metric_missing")

# 4. Loopback binding (port must be unreachable from outside).
nc -z -w 2 public-tempo-host 4317 \
  && failures+=("otlp_port_open_to_public")

# 5. Redaction. Probe with a known offender; Tempo must not have it.
trace_id=$(otel-cli span export \
  --endpoint tempo:4317 \
  --tls --ca-file /etc/tempo/tls/internal-ca.crt \
  --service validate-hardening --name "POST /audit" \
  --attrs 'user.email=test@example.com' 2>/dev/null \
  | awk -F= '/trace_id/ {print $2}')
sleep 5
match=$(curl -sG http://tempo:3200/api/search \
  --data-urlencode 'q={ span.user.email =~ "test@example.com" }' \
  --data-urlencode 'limit=10' | jq '.traces | length')
if [ "$match" -gt 0 ]; then
  failures+=("redaction_failed_user_email_in_buckets")
fi

# 6. IRSA assumed.
kubectl -n observability exec deploy/tempo -- \
  printenv AWS_ROLE_ARN 2>/dev/null \
  | grep -q 'role/GrafanaObservability-TempoS3' \
  || failures+=("irsa_role_not_assumed")

# 7. Bucket deny rules. The role's policy is committed.
[ -f /etc/tempo/iam-policy.json ] \
  || failures+=("iam_policy_not_committed")

if [ ${#failures[@]} -eq 0 ]; then
  echo "Tempo hardening audit: PASS"
  exit 0
fi
echo "Tempo hardening audit: FAIL"
printf ' - %s\n' "${failures[@]}"
exit 1

The audit is run on a schedule (a weekly cron against the cluster) and as a pre-deployment gate in CI. The script’s exit code is the pipeline’s gate.

How to validate it — the operational checklist

A pre-deployment gate that goes through every control:

• auth_enabled: true in tempo.yaml; verified by grepping the config.
• OTLP loopback-bound; the port is unreachable from the pod network.
• TLS configured; the cert is valid for at least 14 days.
• attributes/redact allowlist committed to the collector config repo.
• Collector pipeline order: [attributes/redact, ..., batch] — attributes/redact is first, batch is last.
• IRSA / Workload Identity in place on the Tempo service account; static keys not present.
• Bucket policy denies non-TLS.
• KMS key rotation committed; alert on not_after exists.
• Prometheus alerts on the metric set above.
• Audit script exits 0 on the staging cluster.

A clean validation: every item is ticked; the audit script exits 0; the cert has more than 14 days of validity; the collector pipeline order is enforced in CI.

How it can fail

Five shapes from real audits.

1. Half the checklist is applied during a copy-paste of an older Tempo config. A new service uses a tempo.yaml from a previous team; auth_enabled is false. Symptom: the cluster accepts data without tenant identification. Fix: enforce the gate in CI.
2. The pipeline order regresses. A config change that puts batch before attributes/redact. Symptom: wire dumps show raw attributes. Fix: assert the order in CI; the YAML schema rejects a pipeline with batch first.
3. Cert-manager rotates the cert but a stale secret persists. The TLS reload does not happen; Tempo serves the old cert. Symptom: handshakes fail with “certificate has expired” against the new SAN. Fix: reload the Tempo process on every cert-manager rotation; alert on tempo_tls_cert_not_after approaching zero.
4. The metrics-generator bucket shares the trace bucket. Two roles, two keys, one bucket. Symptom: metrics data appears under the trace prefix; the compactor scans it as a foreign object. Fix: separate bucket; separate IAM role; separate KMS key.
5. The audit script runs against staging, not production. Production drifts from staging. Symptom: the audit passes on staging but production has the original bug. Fix: the audit runs against production through the same proxy.

How to troubleshoot it

The diagnostic order for “is this Tempo cluster hardened?”:

1. Run the audit script. Its output names the failures.
2. Check the metrics. Each control maps to a metric.
3. Inspect the config with kubectl get configmap -o yaml; the diff against the committed baseline surfaces drift.
4. Confirm the proxy is up. Many of the controls depend on the fronting proxy (auth, X-Scope-OrgID injection). The proxy’s logs are the first place to look.

Security implications

• Every control maps to a regulatory requirement. Authentication → access control. TLS → encrypt in transit. Redaction → data minimisation. IAM → least-privilege. Monitoring → incident response.
• The checklist is the artefact the auditor reads. Not the test cases, not the runbooks — the checklist, with signatures and dates.
• The CI gate is the strongest form of the checklist. Documentation is recall; CI is enforcement.

Performance implications

The security controls impose a small overhead:

• TLS adds 5-10% CPU; negligible against the bucket write cost.
• attributes/redact adds ~10 microseconds per trace.
• Per-tenant rate limits add a single atomic increment per span.
• IRSA adds an STS call at pod startup; the session persists for an hour.

None of these change the capacity plan materially.

Production guidance

• Apply the checklist in CI, not in review. The CI gate is the strongest version; the review is the weakest.
• Run the audit on a schedule. A weekly cron is the minimum; a pre-deployment run is the standard.
• Fail the deploy on any checklist failure. A missing metric is a finding; a missing alert is a finding; a commented-out attributes/redact block is a finding.
• Keep the checklist short. Six surfaces, six controls, one artefact. The longer the list, the less it gets run.
• Pair each control with a metric. Configurations without observability drift silently; configurations with metrics drift loudly.

Verification

You should now be able to answer:

• Which six configuration surfaces are the focus of the hardening checklist?
• Which two surfaces belong together (TLS at the receiver and auth at the proxy) and why are they a pair?
• How do you make a hardening checklist resistant to human drift?
• Which metric family surfaces a per-tenant limit exhaustion?
• Why is monitoring-of-the-observability part of the checklist rather than a separate concern?

Quiz