Loki and Promtail — the log collection and storage

Loki is the log storage; Promtail is the log collector. The Promtail tails the /var/log/pods and pushes to Loki. The LogQL queries the logs. The Grafana integrates the Loki. This lesson walks the deployment, the Promtail configuration, the LogQL queries, and the cross-reference to the Observability course.

The Loki architecture

The Loki architecture:

flowchart LR
    A[Container] --> B[Promtail]
    B --> C[Loki distributor]
    C --> D[Loki ingester]
    D --> E[Chunks]
    E --> F[S3]
    E --> G[Loki querier]
    G --> H[Grafana]

The Loki architecture is the log path.

The Loki deployment

The Loki deployment:

helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki \
  --namespace monitoring \
  --set deploymentMode=SingleBinary

The Helm chart deploys the Loki single-binary mode.

The Loki single-binary mode

The Loki single-binary mode:

flowchart LR
    A[Promtail] --> B[Loki distributor]
    B --> C[Loki ingester]
    C --> D[Loki querier]
    D --> E[Chunks]
    E --> F[S3]

The single-binary mode is for small to medium clusters.

The Loki distributed mode

The Loki distributed mode:

flowchart LR
    A[Promtail] --> B[Distributor]
    B --> C[Ingester]
    C --> D[Store]
    D --> E[S3]
    F[Querier] --> D
    A --> F

The distributed mode is for large clusters.

The Promtail configuration

The Promtail configuration:

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: kubernetes-pods
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      target_label: app
    - source_labels: [__meta_kubernetes_pod_log_stream]
      target_label: stream
    - source_labels: [__meta_kubernetes_pod_label_team]
      target_label: team

The Promtail tails the /var/log/pods and pushes to Loki.

The Promtail deployment

The Promtail deployment:

helm install promtail grafana/promtail \
  --namespace monitoring

The Helm chart deploys the Promtail as a DaemonSet.

The LogQL queries

The LogQL queries:

# All logs from the nginx service
{service="nginx"}

# Logs with error
{service="nginx"} |= "error"

# Logs with regex
{service="nginx"} |~ "ERROR|WARN"

# Logs with metric
rate({service="nginx"} |= "error" [5m])

# Logs with line filter
{service="nginx"} != "healthcheck"

The LogQL is the input for the dashboards and alerts.

The Grafana integration

The Grafana integration:

flowchart LR
    A[Grafana] --> B[Loki datasource]
    B --> C[Loki]
    C --> D[Logs]

The Grafana connects to Loki via the Loki datasource.

The log-based alerts

The log-based alerts:

# Prometheus alert (via Loki ruler)
groups:
  - name: log-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({service="nginx"} |= "error" [5m]))
          > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate from nginx logs"

The log-based alerts are the input for the Alertmanager.

The long-term storage

The long-term storage:

# Loki storage config
storage:
  type: s3
  s3:
    bucketnames: my-loki-bucket
    endpoint: s3.amazonaws.com
    access_key_id: <key>
    secret_access_key: <key>

The Loki stores the chunks in S3 for long-term retention.

The production patterns

The production patterns:

flowchart LR
    A[Promtail] --> B[Loki]
    B --> C[S3]
    B --> D[Grafana]
    B --> E[Alertmanager]

The log collection and storage are integrated.

The cross-course references

The Observability course covers the logs in detail.

• The Loki course covers the log storage.
• The Promtail course covers the collection.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Deploy Loki and Promtail. The canonical stack.
• Use the Loki single-binary mode. For small/medium clusters.
• Use the Loki distributed mode. For large clusters.
• Use the LogQL queries. The dashboards and alerts.
• Configure the long-term storage. S3 or GCS.
• Document the deployment. The Helm values, the queries.

The Loki and Promtail are the cluster’s log stack. Operating it well is the deployment, the LogQL queries, and the long-term storage.