Log cardinality and capacity - sizing and tuning

A log store is only as good as its cardinality discipline and capacity planning. This lesson covers both.

Cardinality

Cardinality is the number of unique values for a label. Low cardinality is good for indexing; high cardinality explodes index size.

That table is a Loki table, and the distinction matters because the two backends fail in completely different ways.

Loki. Labels define streams. Every unique combination of label values is a separate stream with its own index entry and its own chunk set. A request_id label with a million values is a million streams, and it will take the install down. Keep labels to a handful of low-cardinality dimensions — host, job, service, env, level — and put everything else in the log line, where | json retrieves it at query time.

Elasticsearch. High-cardinality values are fine, and are frequently the point: a keyword field holding millions of distinct request_id or trace_id values costs very little, and those are exactly the fields that make correlation across services possible. Stripping them buys nothing and destroys the investigation. ES has cardinality risks, but they are different ones:

• Mapping explosion — too many distinct field names, not values, usually from unbounded dynamic mapping. Control it with explicit mappings plus dynamic: strict, or park unpredictable structures under a flattened field.
• Heap pressure from aggregations — a terms aggregation over a high-cardinality field builds a large in-memory structure. That is a query-shape problem, fixed with composite aggregations or by not running it, and not a reason to drop the field at ingest.

Reduce cardinality (Loki)

Labels are declared on the sink, not built up in a remap transform. Writing .labels.host in a transform creates an ordinary nested event field called labels and does nothing to the Loki label set:

# Vector
sinks:
  loki:
    type: loki
    inputs: ['parse']
    endpoint: 'http://loki:3100'
    labels:
      host: '{{ host }}'
      service: '{{ service }}'
      level: '{{ level }}'

Everything not named there travels in the line and stays queryable. Reach for del(.request_id) only when you have decided the field has no investigative value at all — for Loki, leaving it in the message body is almost always the better trade.

Capacity planning

Estimate log volume:

events_per_host_per_day = 100,000
bytes_per_event = 500
volume_per_host_per_day = 50 MB
hosts = 100
total_per_day = 5 GB
retention = 30 days
total_storage = 150 GB

For 1,000 hosts: 5 TB. For 10,000: 50 TB. Storage costs scale with log volume and retention.

Retention tiers

Different retention for different tiers:

For most production, 30 days hot + 90 days cold is enough. For compliance, longer archive.

Plan for spikes

Log volume can spike:

• A burst of errors (incident).
• A misbehaving application logging too much.
• An attacker flooding logs (log noise attack).

Plan capacity for 5-10x normal volume. Configure alerts on log volume:

There is no node_log_messages_total; node_exporter does not count log lines. The counter has to come from whatever is actually handling the logs. Pick the one that matches your pipeline:

# Loki: lines accepted by the distributor, per tenant. Works whatever
# shipper is in front of it, which makes it the best single signal.
- alert: LogVolumeSpike
  expr: |
    sum by (tenant) (rate(loki_distributor_lines_received_total[5m]))
      > 2 * sum by (tenant) (rate(loki_distributor_lines_received_total[1h] offset 1d))
  for: 15m

# Vector: events out of each host's source, when you want per-host attribution
- alert: HostLogVolumeSpike
  expr: |
    sum by (instance) (rate(vector_component_sent_events_total{component_kind="source"}[5m]))
      > 2 * sum by (instance) (rate(vector_component_sent_events_total{component_kind="source"}[1h] offset 1d))
  for: 15m

Two details that decide whether the alert is useful. offset 1d compares against the same hour yesterday rather than against a fixed number, so a nightly batch job does not page anybody. And for: 15m is what stops a single burst — a deploy, a restart — from firing; a genuine log flood does not stop after five minutes.

Alert on the drop as well:

- alert: LogVolumeCollapsed
  expr: sum(rate(loki_distributor_lines_received_total[10m])) == 0
  for: 10m

Logs stopping is the more dangerous failure of the two: the dashboard goes quiet, everything looks calm, and the audit trail you would need has a hole in it.

Avoid common pitfalls

• High-cardinality labels: explode index size. Use sparingly.
• No retention policy: store grows without bound. Always set retention.
• Unstructured logs: harder to search. Parse at shipper.
• Verbose debugging in production: add a level filter before shipping.

Knowledge check