Tenant Audit · Observability

A former employee’s name appears in a data-protection request. The question is precise: between 1 April and 30 June, which principals read log data from the team-hr tenant, and what did they query? The platform team has Grafana, three months of retention, and no answer. Loki’s query log was never enabled, the gateway access log rotates after seven days, and the incident-federated datasource has been used by whoever happened to be in the on-call Grafana team at the time.

Nothing was breached. The platform simply cannot describe its own access, which for a regulated data class is itself the finding.

What it is

Tenant audit is the practice of retaining and reviewing evidence of who did what to each tenant. Four distinct trails, each answering a different question:

Query log - who read this tenant’s data, when, and with what query. Loki log_queries_longer_than plus the query-frontend log; Mimir’s activity tracker.
Access log - which authenticated principal reached the gateway, for which tenant, and what the gateway resolved the tenant scope to. This is the trail that survives a compromised or mis-scoped client.
Alert and notification log - which rules fired for which tenant, who was notified, and who silenced what. A silence is a change to the monitoring posture and belongs in the audit.
Configuration change log - changes to limits, retention, credentials, datasource scope and the tenant register itself. This is git history if you followed lesson three, and a mystery if you did not.

The quarterly review is the human process that consumes those trails: reconcile the register, remove stale tenants, remove stale access, and re-confirm retention against what was promised.

Why a sysadmin cares

You will be asked the access question. For any tenant holding personal, financial or health data, “who read this” is asked eventually, and it cannot be answered retrospectively.
Access accumulates and never decays on its own. Grafana team memberships, service-account tokens, break-glass grants and gateway credentials all outlive their purpose by default.
Tenants outlive their owners. Teams reorganise; the tenant keeps ingesting. Without reconciliation, a meaningful share of your bill and your risk belongs to nobody.
Silences hide outages. A silence created during an incident and never removed is the most common cause of “why did nothing page us” six months later.

How it works

  WRITE / READ TRAFFIC
        |
        +--> gateway access log  --------+
        |    principal, resolved tenant  |
        |    scope, path, status         |
        |                                |
  Loki / Mimir query path                |
        +--> query log ------------------+
        |    org_id, query, duration,    |
        |    bytes processed             |
        |                                |
  Alertmanager / Grafana Alerting        |
        +--> notification + silence log -+
        |                                |
  Git (tenants/, runtime.yaml,           |
       price-book, gateway maps)         |
        +--> change log -----------------+
                                         |
                                         v
                        +----------------------------+
                        | audit tenant (write-only   |
                        | for the platform, readable |
                        | by security, longer        |
                        | retention: 400 days)       |
                        +-------------+--------------+
                                      |
                          quarterly review process
                                      |
        +-----------------+-----------+------------+
        |                 |                        |
   reconcile         revoke stale             confirm
   register          access                   retention

The design rule: audit data lives in its own tenant with its own retention, and platform operators can write it but not delete it. Storing team-hr access logs inside team-hr means the people whose access is being audited control the evidence.

Under the hood

Loki does not have a dedicated audit log. What it has is the query-frontend and querier request log, which includes the resolved org_id, the LogQL expression, the time range, the duration and bytes processed. Setting frontend.log_queries_longer_than: 0s logs every query, which is what you want for audit and which does add log volume proportional to query rate.

Those log lines go to the process’s stdout or journal. To make them queryable you must ship them like any other log - typically an Alloy instance on the Loki hosts scraping the journal and pushing to a dedicated audit tenant. This produces a deliberate circularity: Loki logging its own queries into Loki. That is acceptable as long as the audit tenant is separate, because a failure in a customer tenant then cannot destroy the evidence.

Mimir provides activity_tracker.filepath, which records in-flight requests to a file so that a crash leaves a trace of what was running. It is a crash-forensics tool, not an audit log; per-tenant query attribution in Mimir comes from the query-frontend log and the cortex_query_frontend_queries_total{user} counter.

Grafana Enterprise emits structured audit events for logins, datasource queries and permission changes. On open-source Grafana the equivalent signals are the grafana logger at info level plus the HTTP access log, which record user, org and the datasource proxy path - enough to attribute a query to a Grafana user, but you must correlate the Grafana user with the gateway principal yourself, because the gateway sees only the Grafana service credential.

How to configure it

Log every query, with the tenant, on the read path:

# /etc/loki/loki.yaml (excerpt)
frontend:
  log_queries_longer_than: 0s     # 0 = log every query. Audit needs
                                  # completeness; expect roughly
                                  # 1 log line per query
querier:
  engine:
    max_look_back_period: 0s

limits_config:
  # attach the query to a principal by requiring the header end to end
  max_query_length: 721h

server:
  log_level: info
  log_format: logfmt              # logfmt parses cleanly in LogQL

Give the gateway a log format that records the resolved scope, not just the request:

# /etc/nginx/conf.d/audit-log.conf
log_format tenant_audit escape=json
  '{"ts":"$time_iso8601",'
  '"principal":"$remote_user",'
  '"requested_tenant":"$http_x_scope_orgid",'
  '"resolved_tenant":"$sent_http_x_resolved_orgid",'
  '"method":"$request_method",'
  '"path":"$uri",'
  '"status":$status,'
  '"bytes":$body_bytes_sent,'
  '"duration":$request_time,'
  '"src":"$remote_addr"}';

access_log /var/log/nginx/tenant-audit.log tenant_audit;

Recording both requested_tenant and resolved_tenant is the point. A request that asked for four tenants and was narrowed to one is the evidence that your ACL worked; a request where they are equal and wide is the evidence that it did not.

Ship all of it to the audit tenant:

// /etc/alloy/audit.alloy
local.file_match "audit" {
  path_targets = [
    {__path__ = "/var/log/nginx/tenant-audit.log", log_type = "gateway"},
  ]
}

loki.source.file "audit" {
  targets    = local.file_match.audit.targets
  forward_to = [loki.process.audit.receiver]
}

loki.source.journal "loki_queries" {
  matches    = "_SYSTEMD_UNIT=loki-query-frontend.service"
  forward_to = [loki.process.audit.receiver]
  labels     = {log_type = "query"}
}

loki.process "audit" {
  // keep the tenant as a label so per-tenant audit queries are cheap
  stage.logfmt {
    mapping = {org_id = "", query = "", duration = ""}
  }
  stage.labels {
    values = {audited_tenant = "org_id"}
  }
  forward_to = [loki.write.audit.receiver]
}

loki.write "audit" {
  endpoint {
    url       = "https://logs.example.internal/loki/api/v1/push"
    tenant_id = "audit"          // separate tenant, longer retention,
                                 // platform cannot delete from it
    basic_auth {
      username      = "alloy-audit"
      password_file = "/etc/alloy/audit.password"
    }
  }
}

Retention for the audit tenant is a policy decision, and it is longer than everything else:

# /etc/loki/runtime.yaml (excerpt)
overrides:
  audit:
    retention_period: 9600h            # 400 days: covers a full year
                                       # plus the review cycle
    ingestion_rate_mb: 4
    max_global_streams_per_user: 2000
    # deliberately NOT granting delete permissions; the compactor
    # delete API is disabled for this tenant at the gateway

Alerting audit, so silences are visible:

# alertmanager.yml (excerpt)
route:
  group_by: ['tenant', 'alertname']
  routes:
    - matchers: [tenant=~".+"]
      receiver: tenant-webhook

receivers:
  - name: tenant-webhook
    webhook_configs:
      # a small service that writes every notification to the audit
      # tenant before forwarding to the team channel
      - url: http://audit-forwarder:8080/notify
        send_resolved: true

How to validate it

Prove that a query you just ran is attributable. Run a marked query:

logcli --addr=https://logs.example.internal --org-id=team-hr \
  query '{namespace="hr-prod"} |= "audit-probe-8812"' --since=5m --limit=1

Then find it in the audit tenant:

logcli --org-id=audit query \
  '{log_type="query", audited_tenant="team-hr"} |= "audit-probe-8812"' \
  --since=10m --limit=5

2026-08-13T16:04:22Z {log_type="query", audited_tenant="team-hr"}
  level=info org_id=team-hr traceID=4f21a0 latency=fast
  query='{namespace="hr-prod"} |= "audit-probe-8812"'
  query_type=filter range_type=range length=5m0s
  duration=41.2ms status=200 total_bytes=2.1MB

That single line contains tenant, query, time and cost. Now confirm the gateway attributed a principal:

logcli --org-id=audit query \
  '{log_type="gateway"} | json | resolved_tenant="team-hr"' \
  --since=10m --limit=3

{"ts":"2026-08-13T16:04:22+00:00","principal":"grafana-view-hr",
 "requested_tenant":"team-hr","resolved_tenant":"team-hr",
 "method":"GET","path":"/loki/api/v1/query_range","status":200,
 "bytes":41221,"duration":0.048,"src":"10.30.4.19"}

Answer the quarterly question - who read this tenant, ranked:

logcli --org-id=audit instant-query \
  'topk(10, sum by (principal) (count_over_time({log_type="gateway"} | json | resolved_tenant="team-hr" [90d])))'

{principal="grafana-view-hr"}    184213
{principal="oncall-sre"}            412
{principal="compliance-jsmith"}      31

Three principals over a quarter, two of them low-volume and worth explaining. That is a reviewable answer.

Reconcile the register against reality. Every discrepancy is a finding:

# READ-ONLY - tenants in storage with no register entry
comm -13 \
  <(ls tenants/*.yaml | xargs -n1 basename | sed 's/\.yaml$//' | sort) \
  <(aws s3 ls s3://logs-prod-eu-west-1/ | awk '{print $2}' | tr -d / | sort)

fake
team-billing-prod
team-search-old

Three findings: a legacy fake prefix from an auth_enabled: false period, a typo tenant from an agent misconfiguration, and a tenant whose team renamed itself and left the old one behind.

Check credentials without a tenant, and tenants without a credential:

comm -3 \
  <(awk -F: '{print $1}' /etc/nginx/loki.htpasswd | sed 's/^alloy-//' | sort) \
  <(ls tenants/*.yaml | xargs -n1 basename | sed 's/\.yaml$//' | sort)

	audit
team-intern-2025

team-intern-2025 has a live write credential and no tenant definition. That is a revocation, today.

Check for silences that outlived their incident:

amtool --alertmanager.url=http://alertmanager:9093 silence query \
  --within 720h -o simple | awk '$0 !~ /^ID/ {print}' | head

d41c8f2a  tenant=team-payments  2026-03-11  jsmith  "deploy window"

A silence from March labelled “deploy window” is not a deploy window.

How it can fail

1. Query logging never enabled. Symptom: the access question is unanswerable. Cause: log_queries_longer_than defaults to a non-zero value, so only slow queries are recorded, which is the opposite of an audit sample.

2. Audit logs stored in the audited tenant. Symptom: an audit trail that disappears exactly when it matters, or a tenant whose retention change silently shortens its own evidence. Cause: convenience.

3. Audit tenant retention shorter than the review cycle. Symptom: a quarterly review that can see six weeks. Cause: the audit tenant inherited the 31-day default because no override was written.

4. Grafana user not correlatable to a gateway principal. Symptom: the gateway log says grafana-view-hr for every query, so a query is attributable to Grafana but not to a person. Cause: Grafana proxies datasource queries with one service credential. Mitigation: keep the Grafana access log, and correlate by timestamp and datasource UID.

5. Reconciliation never run. Symptom: unowned tenants, live credentials for departed teams, and Grafana teams with members who changed roles two years ago. Cause: the review is documented as quarterly and has happened once.

6. Audit volume becomes the largest tenant. Symptom: the audit tenant is the top line on the cost report. Cause: logging every query at a high query rate, with the full query text, at 400-day retention. This is a real trade-off - reduce by dropping high-frequency alerting queries from the audit stream while keeping all human-originated queries.

You are asked who read a tenant three months ago and query logging was off

You cannot answer from telemetry, and you should say so plainly rather than offering inference. What you can reconstruct is weaker evidence: gateway access logs if they survived rotation, Grafana’s own logs for datasource proxy calls, the list of principals that had access during the window from git history of the ACL and register, and the Grafana team memberships at the time from the same history.

Report that as what it is: a list of who could have read the data, not who did. Then close the gap - enable full query logging, set the audit tenant retention to cover at least one review cycle plus a year, and record the date the gap closed so the next request has a clear boundary between “we cannot say” and “here is the evidence”.

How to troubleshoot it

Confirm the audit stream is arriving at all. An audit pipeline that silently stopped is worse than none, because you will trust it:
```
logcli --org-id=audit instant-query \
  'sum by (log_type) (count_over_time({log_type=~".+"}[1h]))'
```
```
{log_type="gateway"} 41822
{log_type="query"} 3914
```
A log_type missing from this output is a broken shipper.

Alert on audit-stream absence. This is the one alert that must exist, because the failure is invisible:

- alert: AuditStreamMissing
  expr: sum(count_over_time({log_type="gateway"}[30m])) == 0
  for: 15m
  labels:
    severity: critical
  annotations:
    summary: 'Gateway audit stream has stopped; access is unattributable'

If a query is not attributable, work backwards through the hops. Is org_id present in the Loki log line? Is the gateway log format recording resolved_tenant? Is the shipper labelling audited_tenant? One of the three is missing.
If the register and reality disagree, do not delete first. An unexpected prefix may hold weeks of real data from a typo. Identify the writer via the gateway log before removing anything:
```
logcli --org-id=audit query \
  '{log_type="gateway"} | json | resolved_tenant="team-billing-prod"' \
  --since=720h --limit=5
```
For a stale-access finding, check last use before revoking. Revoking an unused credential is safe; revoking an actively used one during business hours is an outage you caused.
Record every finding and its disposition. The review is only evidence if its output is durable. A file per quarter in the same repository as tenants/ is sufficient and is itself audited by git.

Security implications

The audit tenant must be append-only from the platform’s perspective. Block the compactor delete API for it at the gateway, and if the object store supports it, apply an object-lock or retention policy on the audit prefix. An operator who can delete audit data has no audit.
Audit logs contain sensitive content. A LogQL query text can include a line filter with a customer identifier or a token fragment. Treat the audit tenant as at least as sensitive as the most sensitive tenant it observes, and restrict read access to security and platform leadership.
Separate write and read paths for audit. The platform writes; a distinct set of principals reads. Same-credential access to both is the shortcut that makes the trail deniable.
Reviews are a control, so record the negative results too. “No stale credentials found” with a date and a command output is evidence. A review with no output is indistinguishable from a review that never happened.

Performance implications

Full query logging is proportional to query rate, not data volume. A platform with heavy alerting evaluation can generate more audit lines than a platform with heavy log ingest. Measure before sizing.
Keep the tenant as a label, keep the query text in the body. audited_tenant is low cardinality and makes per-tenant audit queries cheap. Promoting principal to a label is usually acceptable too; promoting query or traceID would be a cardinality disaster.
400-day retention on the audit tenant is the dominant audit cost. Audit logs compress well - they are highly repetitive logfmt or JSON - so the real figure is often lower than teams fear. Measure the ratio as in the cost lesson rather than guessing.
Reconciliation queries are heavy. The 90-day count_over_time used above should be run during the review, not put on a dashboard that refreshes every thirty seconds.

Verification

You should now be able to answer:

Which four audit trails does a multi-tenant observability platform need, and what does each one uniquely answer?
Which Loki setting causes every query to be logged, and why is the default unsuitable for audit?
Why must audit data live in its own tenant rather than in the tenant it describes?
Why does the gateway need to log both the requested and the resolved tenant scope?
What is the correct first action on discovering a tenant with no owner, and why is it not deletion?