New High-Volume Service

A team deploys a new analytics service on a Friday afternoon. The service is not in the observability stack; it ships its logs to stdout and lets the agent pick them up. By Monday morning the analytics service is ingesting 80 percent of the cluster’s total bytes. The streams-per-tenant ceiling is hit on two unrelated tenants whose labels happened to multiply. The ingester memory is climbing. The on-call is paged for a spike they cannot attribute to any known service.

This is the new-service incident. It is the most preventable of the three spike shapes because the cause is known in advance: a service is about to start emitting logs that nobody sized the platform for. The prevention is an onboarding checklist; the recovery is the same as for any spike plus a conversation with the team that owns the new service.

What it is

A new high-volume service is a service whose steady-state log output exceeds what the platform was sized for, or whose label cardinality exceeds what the index can absorb. The service may be genuinely new (a fresh deployment) or newly connected (a service that existed but did not log to Loki before). Both shapes have the same signature: a service label that did not exist yesterday appears at the top of the per-service byte ranking.

The risk is not only volume. The risk is cardinality. A new service often comes with a new label set, and the label set may include values that the existing agents drop or that the existing rate limits do not bound. The audit before onboarding is the cheapest control.

Why a sysadmin cares

A new-service incident is the spike shape that the rest of the fleet pays for. Three operational consequences recur:

• The rate limit is set per-tenant. A new tenant that exceeds its rate limit at the distributor is rejected; a new tenant that exceeds it at the agent fills the agent’s forwarder queue and drops locally. Either way, the new tenant’s logs are not the only ones at risk; the agent’s queue may also be holding legitimate logs from other tenants.
• The index fan-out is global. Loki’s index is one logical store across tenants. A new tenant with high cardinality raises the per-query lookup cost for every other tenant. A dashboard that used to scan 100,000 streams now scans 10,000,000.
• The retention budget is shared. Loki retention is sized for a steady-state ingest rate plus a margin. A new tenant that triples the steady-state rate consumes the retention budget three times faster; the oldest logs of every tenant are evicted to make room for the new tenant’s output.

The cost is paid by everyone; the cause is owned by one team. The platform team is the one that pages.

How it works

The mechanism is a missing rate of change in the per-service ingest panel. A new service that emits at high volume produces a sharp step in the metric; the step is visible within an hour if the panel is built, invisible if it is not.

Per-service ingest rate (bytes per second, last 7 days)
            ^
            |                            +-- analytics-svc
            |                         +--+        (new Tuesday)
            |                      +--+
            |                   +--+
            |               +---+
            |            +--+
            |         +--+
            |      +--+
            |   +--+
            |+--+
            +------------------------------------------> time
            checkout-svc          inventory-svc
            (old, steady)         (old, steady)

The new tenant is the rising line. The line is not noisy because the new service is steady; it is noisy because the other services are not, and the panel cannot separate the two without a per-service breakdown. The top-N query is the breakdown.

How to configure it

The configuration has three layers: the agent pipeline, the platform limits, and the onboarding ticket. Each layer has a role.

The agent pipeline is where the new service’s labels are scrubbed and its rate is capped. The pipeline is the same shape as for any other service; the values are tuned for the new service’s expected volume.

# /etc/alloy/config.alloy
# Per-service agent pipeline. Each new tenant gets its own
# component file; the rate limit is set from the onboarding
# audit.
loki.relabel "analytics_svc_scrub" {
  forward_to = loki.write.analytics.receiver

  # Drop labels that the service emits but that are unbounded.
  # The list comes from the cardinality audit.
  rule {
    action        = "labeldrop"
    regex         = "(request_id|correlation_id|session_id|user_id)"
  }

  # Cap the rate at the agent. The cap is the per-tenant rate
  # from the onboarding ticket; this is the second line of
  # defence (the platform limit is the first).
  rule {
    action        = "sampling"
    source_labels = ["service_name"]
    regex         = "analytics-svc"
    sample_rate   = 10
  }
}

loki.write "analytics" {
  endpoint {
    url = "http://loki-distributor:3100/loki/api/v1/push"

    # Per-tenant batch parameters. The batch_wait is the time
    # the agent holds lines before flushing; the batch_size is
    # the maximum batch size. Larger batches are cheaper; the
    # trade-off is the delay before lines reach Loki.
    batch_wait = "1s"
    batch_size = "1MB"
    min_backoff = "100ms"
    max_backoff = "5s"
  }
}

The platform limits are the per-tenant caps in Loki. These are the values the platform team sets when the onboarding ticket is approved.

# /etc/loki/config.yaml (Loki 3.x)
limits_config:
  # Per-tenant ingestion rate, in MB/s. Sized at 1.5x the
  # expected steady-state rate from the audit. The new tenant
  # is rejected at the distributor when it exceeds this.
  ingestion_rate_mb: 20

  # Per-tenant burst window, in seconds. A short burst is
  # legitimate (startup, restart); a sustained burst is a spike.
  ingestion_burst_size_mb: 40

  # Hard ceiling on active streams per tenant. A safety net that
  # fires before the ingester exhausts memory.
  max_streams_per_user: 10000

  # Per-tenant retention. The window is set by the platform
  # team's policy and the tenant's compliance requirements.
  retention_period: 744h

The onboarding ticket is the human-side control. The ticket is the conversation between the platform team and the service team that owns the new service.

# onboarding ticket fields
fields:
  service_name: analytics-svc
  team: growth
  expected_request_rate_rps: 5000
  expected_lines_per_request: 8
  expected_avg_line_bytes: 256
  expected_steady_state_ingest_mb_s: 10
  label_set_audited: true
  label_cardinality_max_per_hour: 5000
  retention_days: 31
  rate_limit_mb_s: 15
  alert_on_label_cardinality_spike: true

How to validate it

The validation is three queries. The first confirms the service is in the cluster; the second confirms its rate is within the budget; the third confirms its cardinality is within the budget.

# 1. Confirm the service appears in the per-service ranking.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=1h \
  '{service_name=~".+"}' \
  | grep -oE 'service_name="[^"]+"' | sort | uniq -c | sort -rn | head -20

# 2. Confirm the rate is within the per-tenant budget. The
# steady-state rate should be below the limit; the burst should
# be below the burst cap.
# Severity: READ-ONLY
logcli query --since=15m \
  'sum(rate({service_name="analytics-svc"}[5m]))'

# 3. Confirm the cardinality is within the budget. The number
# of distinct streams per hour is the cardinality tell.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=1h \
  '{service_name="analytics-svc"}' | wc -l

Expected: the stream count is below max_streams_per_user and the rate is below ingestion_rate_mb. A service that exceeds either budget is a release blocker; the rate limit and the stream cap are the gates.

How it can fail

Five failure shapes recur at the new-service incident.

1. The unannounced service. A team deploys a service that ships logs to Loki without coordinating with the platform team. The first indicator is a service label that did not exist yesterday appearing in the top-N query.
2. The mis-sized service. A team deploys a service whose steady-state rate is 10x what they estimated. The first indicator is the service’s rate hitting the per-tenant cap within hours of deployment.
3. The high-cardinality label. A team deploys a service with a label that scales with the request rate (a request id, a session id). The first indicator is the stream count climbing toward max_streams_per_user.
4. The shared agent. A new service shares an agent pipeline with an existing service, and the agent’s batch size is tuned for the existing service’s volume. The first indicator is the agent’s forwarder queue growing and the new service’s lines arriving at Loki with high latency.
5. The wrong tenant id. A service is configured with the wrong tenant id at the agent, and its logs are billed to another tenant. The first indicator is the bill.

How to troubleshoot it

1. Confirm the new service (query 1 above)
        |
        v
2. Measure its rate (query 2 above)
        |
        v
3. Measure its cardinality (query 3 above)
        |
        v
4. Apply the right response for each finding:
        |
        +----> rate too high -> lower the agent sampling rate;
        |                       raise the per-tenant cap if the
        |                       audit justifies it
        |
        +----> cardinality too high -> add a labeldrop rule;
        |                            move the field to
        |                            structured metadata
        |
        +----> unannounced service -> notify the team; negotiate
        |                            the rate limit and retention
        |
        v
5. Validate (queries 1-3 again)
        |
        v
6. Add the service to the per-service rate panel so future
   growth is visible

Security implications

The wrong-tenant-id shape is the security dimension. A service whose logs are billed to the wrong tenant leak one tenant’s data into another’s index. The leak is invisible at query time (the labels look right); the leak is visible at audit time (the wrong tenant’s logs appear in the right tenant’s bill). The fix is the same as for any cross-tenant leak: notify both tenants, identify the exposure window, and accelerate retention on the affected tenant. The application-level fix is to never configure the tenant id manually; the platform-level defence is a tenant-id allowlist at the distributor.

Performance implications

The performance cost of a new service is paid in two places. The ingest path pays in CPU and memory. The query path pays in index fan-out and chunk-store IOPS. Both costs are bounded by the per-tenant rate limit and the per-tenant stream cap; both are visible only across the fleet, not to the offending tenant in isolation.

Verification

You should now be able to answer:

• What three checks should the onboarding audit cover before a new service ships logs to Loki?
• What is the per-tenant rate limit, and how is its value chosen from the audit?
• What is the cardinality risk of a new service, and where in the pipeline is it bounded?
• What is the correct response to a service team that refuses to negotiate a rate limit?

Quiz