Loki Hardening Checklist

A platform team inherits a Loki deployment from a previous team. The Grafana dashboards work. The agents are pushing. The audit arrives. The auditor asks the first question: “Show me how you authenticate callers to Loki.” The team opens the config and finds auth_enabled: false. The second question: “How do you scope S3 access per tenant?” The team opens the bucket policy and finds s3:GetObject on arn:aws:s3:::*. The third question: “How do you redact PII before it reaches Loki?” The team opens the agent config and finds no labeldrop rule for email. The fourth question: “How do you prevent log injection?” The team opens the application and finds a format-string logger that interpolates raw user input. The audit fails on the first control.

This is the failure mode of a Loki deployment that is operationally fine and security-broken. The hardening checklist is the discipline that closes the gap. The checklist is not optional; the auditor will walk it on the first visit, and the platform will be measured against it.

What it is

Loki hardening is the set of configurations and operational disciplines that close the security boundary of a multi-tenant Loki deployment. There are six layers, each with its own configuration surface:

1. API authentication. auth_enabled: true, the X-Scope-OrgID header, a reverse proxy that authenticates the caller.
2. TLS termination. TLS at the Loki HTTP API, TLS at the proxy, TLS at the agent-to-proxy path, TLS at the Loki-to-S3 path.
3. S3 permissions. IAM role with IRSA / instance profile, bucket policy with prefix-scoped per-tenant principals.
4. PII redaction. Application-level structured logging, agent-level labeldrop and replace rules, server-side max_label_value_length and max_label_values_per_label as backstop.
5. Log-injection defence. Structured (JSON) logger at the source, agent-level json parser and replace safety net.
6. Operational disciplines. CI audit for PII labels, runbook for incident response, alerting on loki_discarded_samples_total, bucket versioning enabled, secrets stored in a vault not in environment variables.

The checklist is the union of the previous five lessons plus the operational disciplines that bind them together. The lesson is the summary; the runbook is the implementation.

Why a sysadmin cares

A sysadmin cares because the checklist is the operational discipline that turns a working Loki into a hardened Loki. Six failure modes are specific to missing hardening:

1. No auth, on a shared network. auth_enabled: false on a pod reachable from a wide NetworkPolicy. Symptom: any caller can push and query.
2. TLS missing on the agent-to-proxy path. Plain HTTP between the agent and the reverse proxy. Symptom: the X-Scope-OrgID header is visible on the wire; an attacker on the same network can sniff the tenant ID.
3. Wildcard S3 policy. s3:GetObject on *. Symptom: a compromised Loki pod reaches every bucket in the account.
4. No PII redaction. A label called email reaches the index. Symptom: a right-to-erasure request triggers a six-week incident.
5. No log-injection defence. A format-string logger. Symptom: the audit trail is corrupted by attacker- supplied newlines.
6. No operational disciplines. No CI audit, no runbook, no alerting. Symptom: the on-call engineer at 03:00 has no playbook.

How it works

The six layers compose into a single production shape. Each layer is a defence; missing one weakens the others.

   +-------------------+
   | application       |  layer 5: structured JSON logger
   | (whitelist fields)|
   +---------+---------+
             |
             v
   +-------------------+
   | agent             |  layer 4: labeldrop, replace
   | (redact, parse)   |  layer 5: json stage, regex safety net
   +---------+---------+
             |
             v
   +-------------------+
   | reverse proxy     |  layer 1: authenticate, inject X-Scope-OrgID
   | (JWT / OIDC /     |  layer 2: TLS termination
   |  mTLS)            |
   +---------+---------+
             |
             v
   +-------------------+
   | Loki distributor  |  layer 1: auth_enabled: true
   | Loki ingester     |  layer 2: TLS to S3
   | Loki querier      |  layer 3: S3 IAM role (IRSA)
   | Loki compactor    |  layer 6: per-tenant runtime overrides
   +---------+---------+
             |
             v
   +-------------------+
   | S3 bucket         |  layer 3: prefix-scoped bucket policy
   | (versioned)       |  layer 6: bucket versioning, lifecycle
   +-------------------+

The data flow is the same as in the previous lessons. The hardening layer adds the boundary that turns each hop into a trusted hop.

How to configure it

A production deployment that satisfies every layer. The config is a union of the previous lessons; this section shows the whole shape together.

Loki common config

# /etc/loki/config.yaml
auth_enabled: true                # layer 1: API authentication

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  http_tls_config:                 # layer 2: TLS termination
    cert_file: /etc/loki/tls/loki.crt
    key_file:  /etc/loki/tls/loki.key

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-distributor-0.loki-distributor-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1
    # access_key_id and secret_access_key absent: IRSA on EKS,
    # instance profile on EC2, Workload Identity on GKE.
    # layer 3: S3 permissions via IAM role

limits_config:
  # layer 4 (backstop): PII defence at the server
  max_label_value_length: 256
  max_label_values_per_label: 200
  max_streams_per_user: 100000

  # layer 6: per-tenant overrides
  per_tenant_override_config: /etc/loki/overrides.yaml
  per_tenant_override_period: 10s
  retention_period: 2160h        # 90 days

compactor:
  working_directory: /var/lib/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h

# layer 2: TLS to S3 (S3 endpoint is HTTPS by default;
# explicit insecure: false is the production default).

Reverse proxy

# /etc/nginx/conf.d/loki.conf
upstream loki_backend {
  server loki-distributor:3100;
  keepalive 32;
}

server {
  listen 8443 ssl;                # layer 2: TLS termination
  server_name loki.example.internal;

  ssl_certificate     /etc/nginx/certs/loki.crt;
  ssl_certificate_key /etc/nginx/certs/loki.key;
  ssl_protocols       TLSv1.2 TLSv1.3;
  ssl_ciphers         HIGH:!aNULL:!MD5;

  # layer 1: authenticate the caller (JWT, OIDC, mTLS)
  location / {
    auth_jwt "loki";
    auth_jwt_key_file /etc/nginx/jwt/jwks.json;
    set $tenant $jwt_tenant_claim;
    proxy_set_header X-Scope-OrgID $tenant;  # layer 1: inject tenant
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass http://loki_backend;
  }

  location /loki/api/v1/push {
    auth_jwt "loki";
    auth_jwt_key_file /etc/nginx/jwt/jwks.json;
    set $tenant $jwt_tenant_claim;
    proxy_set_header X-Scope-OrgID $tenant;
    client_max_body_size 16m;
    proxy_pass http://loki_backend;
  }
}

Agent

// /etc/alloy/config.alloy
loki.process "hardened" {
  forward_to = loki.write.local.receiver

  // layer 5: parse JSON, ensure structured output.
  // `source` is omitted so the stage parses the log line
  // itself; naming a key that is not already in the extracted
  // map makes the stage a silent no-op.
  stage.json {
    expressions = {
      "level"      = "",
      "msg"        = "",
      "trace_id"   = "",
      "request_id" = "",
    }
  }

  // layer 4: drop PII labels, even if the application
  // accidentally emits them. The block is stage.label_drop,
  // with an underscore, and `values` is a list of literal
  // label names -- not a regex, so name every variant.
  stage.label_drop {
    values = [
      "email",
      "user_email",
      "customer_email",
      "phone",
      "ssn",
      "national_id",
      "ip",
    ]
  }

  // layer 4: move per-line fields to structured metadata,
  // not to labels.
  stage.structured_metadata {
    values = {
      "request_id" = "",
      "trace_id"   = "",
    }
  }

  // layer 5: safety net for any control character that
  // slipped through.
  stage.replace {
    expression = "([\\x00-\\x1F\\x7F])"
    replace    = "?"
  }
}

loki.write "default" {
  endpoint {
    url = "https://loki.example.internal/loki/api/v1/push"
    headers = {
      "X-Scope-OrgID" = "team-checkout",
    }
  }
}

Application

# layer 5: structured JSON logger that whitelists fields.
# No format-string interpolation of user input.
import logging, json

class JsonLineFormatter(logging.Formatter):
    ALLOWED_EXTRA = {"request_id", "trace_id", "level"}

    def format(self, record):
        payload = {
            "ts": self.formatTime(record),
            "level": record.levelname,
            "msg": record.getMessage(),
            **{
                k: v for k, v in record.__dict__.items()
                if k in self.ALLOWED_EXTRA
            },
        }
        return json.dumps(payload, ensure_ascii=False)

S3 bucket policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TenantCheckoutWrite",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/loki-storage"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::prod-loki-chunks/tenant-checkout/*"
    },
    {
      "Sid": "TenantCheckoutList",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/loki-storage"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::prod-loki-chunks",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["tenant-checkout/*", "tenant-checkout/"]
        }
      }
    }
  ]
}

One statement per tenant, scoped to the tenant prefix. The bucket policy is the storage-layer enforcement of tenant isolation.

How to validate it

The validation is the union of the previous lessons. The on-call engineer runs the full checklist every release.

# layer 1: auth_enabled is on.
curl -s http://loki-distributor:3100/config | jq '.auth_enabled'
# expected: true

# layer 1: a request without the header is rejected.
curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST http://loki-distributor:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},
    "values":[["1700000000000000000","probe"]]}]}'
# expected: 401

# layer 2: TLS is in effect.
openssl s_client -connect loki.example.internal:8443 \
  </dev/null 2>&1 | grep -E 'Protocol|Cipher'
# expected: TLSv1.2 or TLSv1.3 and a strong cipher.

# layer 3: S3 IAM role is assumed correctly.
kubectl exec -n loki deploy/loki-ingester -- \
  curl -s http://169.254.170.2/v2/credentials | jq -r '.RoleArn'
# expected: the role you configured.

# layer 4: PII labels are dropped at the agent.
logcli series --analyzer-ingester --since=24h '{job=~".+"}' \
  | awk -F'{' '{print $2}' | awk -F'}' '{print $1}' \
  | tr ',' '\n' | awk -F'=' '{print $1}' \
  | sort | uniq -c | sort -rn | head -10
# expected: bounded labels (job, namespace, instance,
# level, env). No PII label at the top.

# layer 5: the application emits JSON.
# APP_CONTAINER is the name shown by `docker ps` for the
# application whose logs Promtail/Alloy is tailing.
APP_CONTAINER=webapp-app-1
docker logs "$APP_CONTAINER" 2>&1 | tail -5 | jq -e . > /dev/null \
  && echo "JSON OK" || echo "NOT JSON"
# expected: JSON OK.

# layer 6: per-tenant overrides are loaded.
curl -s http://loki-distributor:3100/runtime-config \
  | jq '.overrides | keys'
# expected: the tenant IDs you declared.

The seven checks above are the on-call engineer’s playbook. A CI job that runs them on every release is the production discipline.

How it can fail

Six failure shapes recur when the checklist is incomplete.

1. Auth on, TLS off. The deployment rejects unauthenticated pushes but the agent-to-proxy path is plain HTTP. Symptom: an attacker on the same network can sniff the X-Scope-OrgID header and push as the right tenant.
2. TLS on, auth off. The deployment encrypts traffic but accepts every push. Symptom: any caller can push; the encrypted traffic is a polite veneer.
3. Auth on, PII redaction off. The deployment scopes pushes by tenant but a tenant’s application emits PII labels. Symptom: a right-to-erasure request finds the PII in the index.
4. PII redaction on, log-injection defence off. The agent drops PII labels but the application’s format string injects newlines. Symptom: the audit trail is corrupted by attacker-supplied newlines.
5. Bucket policy wildcard. The IAM role is scoped correctly but the bucket policy grants s3:GetObject on arn:aws:s3:::*. Symptom: a compromised Loki pod reaches every bucket.
6. No CI audit. The deployment passes every manual check but a new release re-introduces the request_id label. Symptom: the cardinality bomb returns three weeks after the last audit.

How to troubleshoot it

The diagnostic order is the union of the previous lessons. The on-call engineer walks the checklist in order:

1. Layer 1 — API authentication. curl /config | jq .auth_enabled. A 401 on a headerless request confirms the layer.
2. Layer 2 — TLS termination. Run openssl s_client against the proxy port. A handshake that fails means the certificate or the protocol is wrong.
3. Layer 3 — S3 permissions. Inspect the IAM role policy and the bucket policy. The two must compose.
4. Layer 4 — PII redaction. Run the cardinality audit. A PII label at the top of the list is the failure shape.
5. Layer 5 — Log-injection defence. docker logs | jq -e .. A non-JSON line means the application is still emitting a format string.
6. Layer 6 — Operational disciplines. Confirm the CI audit runs on every release, the runbook is current, and the alerting is wired.

The walk takes ten minutes. The walk is the on-call playbook.

Security implications

The hardening checklist is the security boundary of the Loki platform. Every layer is a defence; missing one weakens every other.

• Layer 1 missing. The platform is single-tenant in practice. Every push shares one ingester state; every query returns one bucket’s data. The audit fails.
• Layer 2 missing. The X-Scope-OrgID header is sniffable on the network. An attacker can spoof the tenant.
• Layer 3 missing. A compromised Loki pod reaches every bucket in the account. The blast radius is total.
• Layer 4 missing. A PII label triggers a compliance incident.
• Layer 5 missing. The audit trail is corrupted by attacker-supplied newlines.
• Layer 6 missing. A regression re-introduces a vulnerability that was fixed six months ago. The on- call engineer at 03:00 has no playbook.

Performance implications

The hardening layers have negligible performance implication on the happy path. TLS adds microseconds per request; the regex evaluation at the agent adds microseconds per line; the structured logger adds microseconds per line; the IAM role assumption adds microseconds per S3 request.

The performance cost of a missing layer is significant. A missing PII redaction layer is the cardinality bomb; the platform’s ingester exhausts memory. A missing authentication layer is the noisy-tenant denial of service; the platform’s distributor saturates. The hardening layers are also the performance ceiling; without them, the platform scales with the worst tenant, not with the sum of the per-tenant limits.

Production guidance

• Walk the full checklist on every release. The walk is ten minutes; the CI job runs it on every commit.
• Pair every configuration change with a corresponding validation command. The validation is the proof that the change is live.
• Document the on-call playbook. The 03:00 incident does not wait for the engineer to read the docs.
• Enable bucket versioning. The recovery story from a wrong retention sweep depends on it.
• Monitor loki_discarded_samples_total{reason, tenant} and loki_request_duration_seconds_count{status_code} in the same dashboard. The metrics are the first signal of a regression.
• Review the IAM role policy and the bucket policy in the same pull request as the Loki config. The two compose; reviewing them separately misses the gaps.
• Pair the agent rules with the application whitelist. The application must not emit PII; the agent must not pass PII through.
• Pair the structured logger at the source with the agent’s JSON parser. The application encodes the value; the agent parses and re-emits; the line stays as one entry.

Verification

You should now be able to answer:

• What are the six layers of the Loki hardening checklist?
• Why is every layer necessary, and what is the failure shape when one layer is missing?
• Which layer enforces tenant isolation at the storage layer, and which at the API layer?
• Where should PII redaction happen, and why is the application the primary defence?
• What is the on-call playbook for a 03:00 Loki incident, and how does the CI audit complement it?

Quiz