Log aggregation — the cluster-wide pipeline

Log aggregation is the cluster-wide pipeline. The multi-tenant considerations, the log retention, and the log-based metrics are the inputs. This lesson walks the pipeline, the multi-tenancy, the retention, and the production patterns.

The cluster-wide pipeline

The cluster-wide pipeline:

flowchart LR
    A[Container logs] --> B[Promtail DaemonSet]
    B --> C[Loki]
    C --> D[Hot storage: 30 days]
    C --> E[Cold storage: S3 1 year]
    C --> F[Grafana]
    C --> G[Alertmanager]

The pipeline is the cluster-wide log flow.

The multi-tenant considerations

The multi-tenant considerations:

flowchart LR
    A[Team A] --> B[Promtail]
    C[Team B] --> B
    D[Team C] --> B
    B --> E[Loki]
    E --> F{Grafana}
    E --> G{Alertmanager}
    F --> A
    F --> C
    F --> D
    G --> A
    G --> C
    G --> D

The multi-tenant considerations:

• Labels per namespace: {namespace="team-a"}.
• Tenant labels: {team="team-a"}.
• Per-tenant retention: different retention per tenant.
• Per-tenant RBAC: the Grafana RBAC.

The Loki multi-tenancy

The Loki multi-tenancy:

auth_enabled: true

tenants:
  - name: team-a
    id: team-a
  - name: team-b
    id: team-b

server:
  http_listen_port: 3100

The Loki multi-tenancy is via the tenant ID.

The Promtail multi-tenancy

The Promtail multi-tenancy:

clients:
  - url: http://loki:3100/loki/api/v1/push
    tenant_id: team-a

The Promtail pushes the logs with the tenant ID.

The log retention

The log retention:

# Loki compactor config
compactor:
  working_directory: /data/compactor
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: filesystem

# Per-tenant retention
limits_config:
  retention_period: 720h  # 30 days

The retention is per tenant.

The hot vs cold storage

The hot vs cold storage:

flowchart LR
    A[Logs 0-30 days] --> B[Hot storage: local SSD]
    C[Logs 30-365 days] --> D[Cold storage: S3]

The hot storage is fast; the cold storage is cheap.

The log-based metrics

The log-based metrics:

# Loki ruler
groups:
  - name: log-metrics
    rules:
      - record: nginx:error_rate:rate5m
        expr: sum(rate({service="nginx"} |= "error" [5m]))

The log-based metrics are the input for the alerts.

The Loki ruler

The Loki ruler:

apiVersion: monitoring.coreos.com/v1
kind: LokiRuler
metadata:
  name: my-loki-ruler
spec:
  rules:
    groups:
    - name: log-alerts
      rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({service="nginx"} |= "error" [5m])) > 1
        for: 5m
        labels:
          severity: warning

The Loki ruler evaluates the rules against the logs.

The production patterns

The production patterns:

# The Loki architecture
helm install loki grafana/loki-stack \
  --namespace monitoring \
  --set prometheus.enabled=true \
  --set grafana.enabled=true \
  --set loki.persistence.enabled=true \
  --set loki.persistence.size=100Gi

The patterns are the production deployment.

The cost optimization

The cost optimization:

Hot storage (30 days): 100 GB at $0.10/GB/month = $10/month
Cold storage (1 year): 1 TB at $0.023/GB/month = $23/month
Total: $33/month for 1 year of logs

The cost optimization is the retention strategy.

The cross-course references

The Observability course covers the logs in detail.

• The Loki course covers the log storage.
• The Grafana course covers the dashboards.
• The Prometheus course covers the metrics.

Quiz

Production discipline

• Design the cluster-wide pipeline. Per cluster.
• Configure the multi-tenancy. Per tenant.
• Configure the retention. Hot and cold.
• Use the log-based metrics. The Loki ruler.
• Monitor the Loki metrics. The bucket, the query.
• Document the aggregation. The pipeline, the retention.

The cluster-wide log aggregation is the production pattern. Operating it well is the pipeline, the multi-tenancy, the retention, and the log-based metrics.