Tenant Onboarding · Observability

The fourteenth tenant on a shared Loki and Mimir platform takes four minutes to onboard. The fifteenth takes three days, because the person who onboarded the first fourteen did it by hand, in a different order each time, and left no record of which credential belongs to which team. Two tenants are writing under one shared password. One tenant’s Grafana datasource has no tenant header and has been silently reading another tenant’s data since March.

Onboarding is where multi-tenancy either becomes an operable system or an accumulation of special cases. The engineering content is trivial; the discipline is the whole lesson.

What it is

Tenant onboarding is the repeatable procedure that turns an agreed request into a fully provisioned tenant. A tenant is not provisioned until all six artefacts exist:

A tenant ID - stable, lower-case, never reused.
A write credential mapped to that tenant at the gateway.
A limits override in runtime_config sized to the declared volume.
A read path - a Grafana datasource carrying the tenant header, with datasource permissions restricted to the owning team.
A retention decision recorded, because it is usually a legal commitment.
A register entry - owner, cost centre, data classification, review date.

Miss any one and you have a partially provisioned tenant, which is the state that produces the incidents in the opening paragraph.

Why a sysadmin cares

Onboarding drift is the root cause behind most multi-tenancy operational debt. Three specific costs:

Unattributable data. A tenant with no register entry has no owner. When it produces 40% of the bill or an OOM, there is nobody to talk to.
Uncontrolled credentials. Shared or undocumented write credentials cannot be rotated, because nobody knows what breaks.
Silent misconfiguration. A datasource without the tenant header either fails closed (visible, fine) or reads the wrong tenant (invisible, a data incident).

The fix is not documentation. It is making the correct path the only path: a declarative repository, a request form that collects the facts you need, and a verification step that fails loudly.

How it works

  Request form (issue template)
     tenant slug, owner, cost centre,
     data class, GB/day, retention,
     signals needed
              |
              v
  Pull request to the platform repo
     tenants/team-billing.yaml
              |
        review + merge
              |
              v
  +-----------------------------------------+
  | Apply pipeline (idempotent)             |
  | 1. render runtime.yaml overrides        |
  | 2. render gateway credential map        |
  | 3. create Grafana folder + datasource   |
  |    via provisioning or HTTP API         |
  | 4. write credential into secret store   |
  | 5. append to tenant register            |
  +-----------------------------------------+
              |
              v
  Verification probes
     write probe -> 204
     forged-tenant probe -> 401/403
     read probe -> own labels only
              |
              v
  Hand the credential reference to the team

The pipeline must be idempotent, because you will run it again for every limit change and every credential rotation. That property is what turns onboarding and day-two operations into the same procedure.

Under the hood

Nothing in Loki, Mimir or Tempo needs to be told a tenant exists. There is no create-tenant API and no registration step. A tenant comes into existence the first time a write arrives with a new X-Scope-OrgID value, and the storage prefix is created lazily on the first chunk flush. Until then, GET /runtime_config will show your override for a tenant that has no data at all, which is expected and harmless.

The corollary is that a typo also creates a tenant. There is no validation against a list of known tenants, because there is no list. Your register is the only authority on which tenant IDs are legitimate, and reconciling the register against the storage prefixes is the only way to detect strays.

On the Grafana side, provisioned datasources are read from /etc/grafana/provisioning/datasources/*.yaml at startup and on SIGHUP. A datasource created by provisioning cannot be edited in the UI, which is exactly what you want: the tenant header cannot be changed by a curious Editor. Datasources created through the HTTP API are editable, so if you provision via API, lock them down with datasource permissions and set editable: false where the API supports it.

How to configure it

Start with the request. This is the issue template - the fields exist because each one is needed by a later step, not for bureaucracy.

# .github/ISSUE_TEMPLATE/tenant-request.yml (fields, abridged)
tenant_slug:        team-billing      # lower-case, stable, never reused
display_name:       Billing Platform
owner_team:         billing
owner_contact:      billing-oncall@example.com
cost_centre:        CC-4471
data_classification: internal          # internal | confidential | pci
signals:            [logs, metrics]    # traces not requested yet
estimated_gb_day:   12
retention_days:     31
peak_streams:       4000
justification_for_deviation: none

Then the declarative tenant definition, one file per tenant:

# tenants/team-billing.yaml
tenant: team-billing
owner: billing
contact: billing-oncall@example.com
cost_centre: CC-4471
data_classification: internal
created: 2026-08-13
review_due: 2027-02-13          # every tenant expires into a review

limits:
  # 12 GB/day is ~0.14 MB/s average; provision for a 4x peak plus
  # headroom, not for the average
  ingestion_rate_mb: 2
  ingestion_burst_size_mb: 4
  max_global_streams_per_user: 6000     # 1.5x declared peak
  retention_period: 744h                # 31 days

grafana:
  folder: Billing
  datasources: [loki, mimir]
  viewer_team: billing

The rendered output for Loki - additive, generated, never hand-edited:

# /etc/loki/runtime.yaml  (generated from tenants/*.yaml)
overrides:
  team-billing:
    ingestion_rate_mb: 2
    ingestion_burst_size_mb: 4
    max_global_streams_per_user: 6000
    retention_period: 744h

The credential. Generate it, never choose it, and store only the hash at the gateway:

# CONFIGURATION - generates a write credential for one tenant
TENANT=team-billing
TOKEN="$(openssl rand -base64 32 | tr -d '/+=' | cut -c1-40)"

# gateway side: bcrypt hash only
htpasswd -B -b -n "alloy-${TENANT}" "$TOKEN" \
  >> /etc/nginx/loki.htpasswd

# team side: the plaintext goes to the secret store and nowhere else
vault kv put "secret/observability/tenants/${TENANT}/write" \
  username="alloy-${TENANT}" password="$TOKEN"

unset TOKEN

Add the credential-to-tenant mapping at the gateway. This is the line that makes the credential mean something:

# /etc/nginx/conf.d/loki-tenants.map
"alloy-team-billing"   "team-billing";

Grafana provisioning, one datasource per tenant per signal:

# /etc/grafana/provisioning/datasources/team-billing.yaml
apiVersion: 1
datasources:
  - name: Loki - Billing
    uid: loki-team-billing
    type: loki
    access: proxy
    url: https://logs.example.internal
    editable: false            # the tenant header must not be
                               # changeable from the UI
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      timeout: 120
      maxLines: 5000
    secureJsonData:
      httpHeaderValue1: team-billing

  - name: Mimir - Billing
    uid: mimir-team-billing
    type: prometheus
    access: proxy
    url: https://metrics.example.internal/prometheus
    editable: false
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      httpMethod: POST
    secureJsonData:
      httpHeaderValue1: team-billing

Provisioning alone does not restrict who can use the datasource. Bind it to the owning team:

# CONFIGURATION - restrict the datasource to the owning team
DS_UID=loki-team-billing
curl -s -X POST \
  -H "Authorization: Bearer ${GRAFANA_SA_TOKEN}" \
  -H 'Content-Type: application/json' \
  "https://grafana.example.internal/api/access-control/datasources/${DS_UID}/teams/7" \
  --data '{"permission":"Query"}'

How to validate it

Never hand over a tenant that has not passed all four probes.

Probe 1 - the credential writes to the right tenant.

TENANT=team-billing
read -rs -p 'token: ' TOKEN

curl -s -o /dev/null -w '%{http_code}\n' \
  -u "alloy-${TENANT}:${TOKEN}" \
  -H 'Content-Type: application/json' \
  -X POST https://logs.example.internal/loki/api/v1/push \
  --data "{\"streams\":[{\"stream\":{\"job\":\"onboard-probe\"},\"values\":[[\"$(date +%s)000000000\",\"tenant onboarding probe\"]]}]}"

Probe 2 - the credential cannot write elsewhere. Send a forged tenant header with the same credential:

curl -s -o /dev/null -w '%{http_code}\n' \
  -u "alloy-${TENANT}:${TOKEN}" \
  -H 'X-Scope-OrgID: team-payments' \
  -H 'Content-Type: application/json' \
  -X POST https://logs.example.internal/loki/api/v1/push \
  --data '{"streams":[{"stream":{"job":"forged"},"values":[["1755000000000000000","x"]]}]}'

A 204 here is a finding, not a pass: the gateway is forwarding the client header. The probe data has landed in team-payments. Expected result is the write succeeding into team-billing because the gateway overwrote the header - confirm with probe 3, not by the status code alone.

Probe 3 - the probe line is visible to the right tenant only.

logcli --addr=https://logs.example.internal --org-id=team-billing \
  query '{job="onboard-probe"}' --since=10m --limit=5

2026-08-13T14:22:07Z {job="onboard-probe"} tenant onboarding probe

Then the same query as another tenant must return nothing:

logcli --addr=https://logs.example.internal --org-id=team-payments \
  query '{job="onboard-probe"}' --since=10m --limit=5

level=info msg="no logs returned"

Probe 4 - the limits override is live.

curl -s http://127.0.0.1:3100/runtime_config \
  | yq '.overrides."team-billing"'

ingestion_rate_mb: 2
ingestion_burst_size_mb: 4
max_global_streams_per_user: 6000
retention_period: 31d

And confirm Grafana sees its datasources:

curl -s -H "Authorization: Bearer ${GRAFANA_SA_TOKEN}" \
  https://grafana.example.internal/api/datasources \
  | jq -r '.[] | select(.uid|startswith("loki-team-billing")) | "\(.uid) \(.type)"'

loki-team-billing loki

How it can fail

1. Datasource provisioned without the tenant header. Symptom: either 401 on every panel (harmless, obvious) or - if auth_enabled is false anywhere in the path - results from the wrong tenant. Cause: httpHeaderName1 present but httpHeaderValue1 missing from secureJsonData, which Grafana accepts silently.

2. Credential issued, gateway map not reloaded. Symptom: the team reports 401 while you can see the entry in the htpasswd file. Cause: systemctl reload nginx was never run, or the map file is included but the map block was not regenerated.

3. Tenant ID reuse. Symptom: a newly onboarded team sees log lines from a decommissioned service. Cause: the slug was reused within the retention window of the previous occupant. Tenant IDs must be retired permanently, not recycled.

4. Limits sized from the average. Symptom: 429 every deploy, or every nightly batch window. Cause: 12 GB/day was divided by 86,400 and provisioned as the ceiling. Real traffic is bursty; size for peak with headroom and use the burst allowance for deploys.

5. Retention never recorded. Symptom: an auditor asks how long billing logs are kept and there are three different answers. Cause: the retention decision lived in a chat message; the override carries a number nobody can justify.

6. Onboarding done by hand for “just this one”. Symptom: the next credential rotation misses that tenant, whose agents start failing 90 days later when the old credential is revoked. Cause: the tenant exists in the gateway but not in tenants/.

How to troubleshoot it

Confirm which of the six artefacts exists. Run through them explicitly rather than assuming the pipeline completed:

TENANT=team-billing
grep -c "alloy-${TENANT}" /etc/nginx/loki.htpasswd
grep -c "\"${TENANT}\"" /etc/nginx/conf.d/loki-tenants.map
curl -s localhost:3100/runtime_config | yq ".overrides.\"${TENANT}\" != null"
ls "tenants/${TENANT}.yaml"

1
1
true
tenants/team-billing.yaml

Separate authentication from tenancy. A 401 is the gateway. A 204 with no data visible is a tenant-name mismatch between write and read. Check both sides of the mapping before touching Loki.

Check whether the data arrived under a different name.

aws s3 ls s3://logs-prod-eu-west-1/ | grep -i billing

                           PRE team-billing-prod/
                           PRE team-billing/

Two prefixes is the diagnosis: the agent config and the gateway map disagree.

Check the Grafana side last, because it is the easiest to misread. Use the datasource health endpoint rather than a dashboard:

curl -s -H "Authorization: Bearer ${GRAFANA_SA_TOKEN}" \
  https://grafana.example.internal/api/datasources/uid/loki-team-billing/health | jq -r '.status, .message'

OK
Data source successfully connected.

Reconcile the register. Any storage prefix without a tenants/*.yaml file is an unowned tenant. Investigate before deleting - it may be a typo with a week of real data in it.

Security implications

The write credential is the whole boundary on the ingest side. Three rules:

One credential per tenant per environment. Shared credentials cannot be rotated and cannot be revoked without collateral damage.
Plaintext exists once. Generate, hash at the gateway, store in the secret manager, and never place the plaintext in the tenant file, a ticket, or a chat message. Everything in tenants/*.yaml is reviewable in a pull request, which is exactly why secrets do not go there.
Rotation is dual-credential, not big-bang. Add the new credential alongside the old, both mapped to the same tenant; let the team roll their agents; confirm zero requests on the old credential; then remove it.

# READ-ONLY - confirm the old credential is unused before revoking
awk '$0 ~ /alloy-team-billing-2025/ {c++} END {print c+0}' \
  /var/log/nginx/loki-access.log

Read scope deserves equal attention. Datasource permissions bound to a Grafana team are the control; a shared “Observability” datasource with a tenant header defeats the entire model, because any Viewer in the org can query that tenant.

Performance implications

Onboarding decisions set the cost profile for the tenant’s whole lifetime:

Provisioned limits become the cluster’s capacity commitment. The sum of per-tenant ceilings should exceed observed aggregate peak with headroom, but not by so much that no limit is reachable before the ingesters are.
Retention dominates storage cost. Ninety days rather than thirty-one is roughly three times the object storage and three times the compactor work for that tenant. Ask for the justification at onboarding, when it is cheap to say no.
Low-volume tenants are not free. Each one carries ingester overhead and flushes small chunks on the idle timer. Onboarding one tenant per microservice looks organised and costs real money; one tenant per team is the shape that performs.

Verification

You should now be able to answer:

Which six artefacts must exist before a tenant is considered provisioned?
Why is there no create-tenant API in Loki or Mimir, and what does that imply about typos?
Why must the write credential and the Grafana datasource header be generated from the same source of truth?
What does a 204 on a forged-tenant-header probe tell you?
Why is provisioning limits from the daily average the wrong sizing method?