Good Loki Labels

A platform team inherits a Loki cluster from a contractor. The streams look like this: \{job="nginx-prod-eu-west-2a-pod-7f9c", level="INFO", pod_ip="10.4.2.18", container_id="a3b1..."\}. Every log line in production is its own stream. The platform team spends a week rebuilding the relabel rules. This is what good Loki labels look like before the rebuild, not after.

What it is

Good Loki labels are a small, fixed, bounded set of label names that every log line in the cluster carries, with values that are known in advance by the team writing the pipeline. The set is not arbitrary; it is the intersection of three properties:

• Bounded. Each label has a known maximum cardinality that does not depend on traffic.
• Semantic. Each label answers a question an operator asks during an incident: which application, which environment, which host, which container, what severity.
• Stable. Each label is stamped at a fixed point in the pipeline, not derived from log content that varies per line.

The canonical set for a Kubernetes-based stack is small:

job          application name (e.g. "checkout-svc")
namespace    Kubernetes namespace (e.g. "payments")
app          label from the pod (often equals job)
env          environment (prod / staging / dev)
cluster      Kubernetes cluster name (in multi-cluster setups)
instance     pod or node identifier (the unique unit of work)
level        severity after parsing (info / warn / error)
component    sub-component within an application

That is eight labels. None of them grow with traffic. None of them carry user data. Every one of them is asked about during an incident.

Why a sysadmin cares

The label set is the index. Every query, every dashboard, every alert that touches Loki begins with a label selector. When the label set is canonical, the on-call engineer can compose a query without thinking: {namespace="payments", level="error"} is a query that works today, next year, and across clusters. When the label set is bespoke, every team reinvents the same answer in six different ways, and the on-call engineer spends ten minutes figuring out which app_name the contractor used this time.

The label set is also the contract between the application owner and the platform team. The application owner decides what to emit; the platform team decides what to keep. The good label set is the small list both sides agree on.

How it works

The good labels are stamped once, by the agent, from sources that are stable: the pod metadata, the node metadata, the container metadata. They are not extracted from log line content. A label whose value can change per line is, by construction, unbounded.

  Kubernetes pod                  Alloy / Promtail
  +-------------------+           +-------------------------+
  | namespace=payments|  ------>  | relabel: keep these,    |
  | app=checkout-svc  |           |          drop the rest  |
  | pod=checkout-7f9c |           |                         |
  +-------------------+           | structured metadata:    |
                                  |   level, status_code,   |
                                  |   latency_ms            |
                                  +-------------------------+
                                            |
                                            v
                                  +-------------------------+
                                  | Loki stream label set:  |
                                  |   job, namespace, app,  |
                                  |   env, instance, level  |
                                  +-------------------------+

The label set is the same for every line the agent emits from a given pod, so the stream is stable. The line content carries everything else, attached as structured metadata so it can be queried at log-parse time without joining the index.

How to configure it

The agent is the right place to stamp the canonical set. Below is a production Alloy pipeline that turns a Kubernetes pod log into a stream with eight bounded labels and the rest in structured metadata.

# /etc/alloy/config.alloy
loki.source.kubernetes "pods" {
  cluster_name = "prod-eu-west-2"
  forward_to   = loki.relabel.canonical.receiver
}

loki.relabel "canonical" {
  forward_to = loki.process.enrich.receiver

  # Keep only the labels we have agreed on. Anything else from
  # the source is dropped at the boundary.
  rule {
    action        = "labelkeep"
    regex         = "namespace|pod|container|app|cluster"
  }

  # Rename pod -> instance so the label name is stable across
  # container and VM workloads. instance is the canonical
  # "where the work runs" label.
  rule {
    action        = "labelmap"
    regex         = "pod"
    replacement   = "instance"
  }
}

loki.process "enrich" {
  forward_to = loki.write.local.receiver

  # Pull the static metadata out of the pod once, not per line.
  stage.static_labels {
    values = {
      job       = "checkout-svc",
      component = "checkout-api",
      env       = "prod",
    }
  }

  # Parse the log line. What is structural becomes structured
  # metadata. What is bounded becomes a label.
  stage.regex {
    expression = "^(?P<level>INFO|WARN|ERROR|DEBUG) "
  }
  stage.labels {
    values = { "level" = "" }
  }
  stage.structured_metadata {
    values = { "status_code" = "" }
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki-distributor:3100/loki/api/v1/push"
  }
}

The same outcome from a VM-based workload, where there is no Kubernetes metadata:

# /etc/alloy/config.alloy (VM host)
loki.source.file "syslog" {
  targets    = local.file_match("/var/log/syslog")
  forward_to = loki.relabel.host.receiver
}

loki.relabel "host" {
  forward_to = loki.process.parse.receiver

  # Keep the canonical set: job (the application), instance (the
  # host), env (the environment), level (after parse).
  rule { action = "labeldrop", regex = "" }   # no-op; explicit
}

loki.process "parse" {
  forward_to = loki.write.local.receiver

  stage.regex {
    expression = "^(?P<level>INFO|WARN|ERROR|DEBUG) "
  }
  stage.labels {
    values = { "level" = "" }
  }
}

How to validate it

Three checks confirm the canonical set is in place. They are the same three checks for any label rule, applied with the canonical names.

# 1. The actual stream label set. Should match the canonical set.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=1h \
  '{job="checkout-svc"}' | head -3

Expected output: each line is \{app="checkout", cluster="...", component="checkout-api", env="prod", instance="checkout-7f9c", job="checkout-svc", level="INFO", namespace="payments"\} and nothing else. Any additional key in the set is a violation.

# 2. The cardinality of each canonical label. Should be flat and
# bounded.
# Severity: READ-ONLY
logcli labels --since=1h job
logcli labels --since=1h level
logcli labels --since=1h namespace

Expected output: job returns the application list (tens of entries); level returns the severity scale (five to ten); namespace returns the namespace list (tens). If job returns hundreds or thousands of entries, a misconfiguration is renaming pods or containers into the job field.

# 3. The relabel rules on the agent. Should show the canonical
# set being stamped.
# Severity: READ-ONLY
curl -s http://alloy:12345/api/v1/status/config \
  | jq '.components[].arguments' | grep -A2 labelkeep

How it can fail

Five failure shapes recur with the canonical set in production.

1. Renaming drift. A pod label is renamed in the deployment manifest but the agent relabel rule still references the old name. Symptom: app shows empty values; streams collapse into a single “empty app” stream; query results become inconsistent.

2. Adding “just one more” label. A new label is added to the canonical set without a cardinality review. Symptom: stream count rises; the audit (lesson 06) catches it after the damage.

3. Component count explosion. component is meant to be a handful of values per application; an over-eager refactor stamps one component per HTTP handler. Symptom: stream count rises linearly with traffic.

4. Environment label churn. A staging environment gets promoted to a new env value every quarter. Symptom: the cardinality of env creeps up over months and is only noticed when the audit finds a hundred environments.

5. Instance cardinality explosion. instance is supposed to be bounded by replica count, but a misconfigured autoscaler stamps the request ID into instance instead. Symptom: stream count rises linearly with traffic; max_label_values_per_label rejections appear.

How to troubleshoot it

1. Identify the unexpected label (logcli labels --since=1h)
       |
       v
2. Find the agent pipeline stamping it (grep -R in /etc/alloy)
       |
       v
3. Decide: rename it (if the value is bounded), drop it (if it
   is not), or move to structured metadata (if it is per-line)
       |
       v
4. Roll the fix; verify with logcli series that the label is gone
       |
       v
5. Update the platform label contract documentation

Security implications

Good labels are bounded, and bounded labels are auditable. The canonical set is small enough that an audit can read the entire list of values and confirm none of them are PII. A label set that grows beyond the canonical is no longer auditable by hand, which is why the label audit (lesson 06) exists.

Performance implications

Eight bounded labels keep the cross product in the operating envelope. Every additional label the team adds is a multiplier. The performance ceiling of a Loki cluster is set by the team that designed the label set, not by the team that runs the ingesters.

Verification

You should now be able to answer:

• What is the canonical bounded label set for a production Loki stack?
• Why is the label set a contract between application owner and platform team?
• Where in the agent pipeline are the canonical labels stamped?
• What is the cardinality budget per label, and why does the product matter?

Quiz