Loki and Elasticsearch - the modern log store

Loki and Elasticsearch are the two most common log stores in production. Loki is label-based and Grafana-friendly; Elasticsearch is full-text and mature. This lesson covers both for production.

Loki

Loki is Grafana’s log store. It indexes labels (not full text), making it cheaper than Elasticsearch. Queries use LogQL.

Deploy

# Download from grafana.com - pin the version, do not glob a URL
LOKI_VERSION=3.3.2
wget "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/loki-linux-amd64.zip"
unzip loki-linux-amd64.zip
sudo install -m 0755 loki-linux-amd64 /usr/local/bin/loki

# Configure
cat > /etc/loki/loki.yaml <<EOF
# auth_enabled: false means Loki performs NO authentication and treats
# every request as the single tenant "fake". It is the correct setting
# for a single-tenant deployment ONLY when the port is unreachable from
# anywhere except the shippers and Grafana - see the callout below.
auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /var/lib/loki

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /var/lib/loki/tsdb-index
    cache_location: /var/lib/loki/tsdb-cache
  filesystem:
    directory: /var/lib/loki/chunks

compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true
  delete_request_store: filesystem
  compaction_interval: 10m
  retention_delete_delay: 2h

limits_config:
  retention_period: 720h
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
EOF

loki -config.file=/etc/loki/loki.yaml

Retention needs the compactor

retention_period on its own does nothing. In Loki, deletion is performed exclusively by the compactor component, and only when it is running with retention_enabled: true. Without the compactor: block above, Loki starts cleanly, accepts the setting, reports no error - and never deletes a chunk.

# WRONG - accepted, ignored, store grows for ever
limits_config:
  retention_period: 30d

This is the worst shape a defect can take: the operator has followed the “always set retention” rule, the config looks right in review, and the store grows until the filesystem fills. At 5 GB/day for 100 hosts, a volume sized for a 150 GB 30-day working set is exhausted on day 30 and every log after that is dropped - during whatever incident happens to be running.

Two details in the corrected block matter:

• retention_period must be a multiple of 24h, so write it as 720h rather than 30d and there is no ambiguity.
• delete_request_store is required on recent Loki versions for the delete path to initialise; set it to the same backend as your chunks.

Verify that retention is actually running rather than assuming it:

# The compactor exports its own metrics - no series here means no retention
curl -s localhost:3100/metrics | grep loki_compactor_

# And watch the store's own filesystem
df -h /var/lib/loki

Query with LogQL

{job="syslog"} |= "error"
{job="syslog"} | json | level="error"
sum by (host) (count_over_time({job="syslog"}[5m]))

Loki is fully integrated with Grafana; use Grafana’s Explore to query.

Elasticsearch

Elasticsearch is the most mature log store. Full-text search, aggregations, and Kibana for visualisation.

Deploy

Elasticsearch is not in the Debian, Ubuntu or RHEL archives — apt install elasticsearch fails with “Unable to locate package”. Elastic ships its own repository, and that is the supported path on a Linux host:

# Elastic's signing key and repository (Debian/Ubuntu)
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch \
  | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update && sudo apt install elasticsearch

The installer prints a generated password for the elastic user and enables TLS. Keep both. From 8.0 the defaults are secure, and every “quickstart” that turns them off is a lab shortcut, not a deployment.

# LAB ONLY - authentication disabled. Throwaway VM, never a routable network.
docker run -d --name elasticsearch \
    -p 127.0.0.1:9200:9200 -p 127.0.0.1:9300:9300 \
    -e "discovery.type=single-node" \
    -e "xpack.security.enabled=false" \
    docker.elastic.co/elasticsearch/elasticsearch:8.10.0

The port mappings are pinned to 127.0.0.1 deliberately. A bare -p 9200:9200 publishes on every interface and, on a default Docker install, writes a DNAT rule that bypasses the host firewall — so the “local” lab store is reachable from the network. Containers are the Docker course’s subject; here they are only a convenient way to get a disposable Elasticsearch.

Index lifecycle

For logs, use ILM (index lifecycle management):

• Hot phase (1-7 days): active writes, fast search.
• Warm phase (8-30 days): no writes, slower search.
• Cold phase (31-90 days): less resources, slower.
• Frozen phase (optional): mounted as a searchable snapshot from object storage. Cheapest to keep, slowest to query - this is what makes multi-year compliance retention affordable.
• Delete phase (after 90 days): automatic removal.

Unlike Loki’s retention, ILM is driven by Elasticsearch itself and needs no separate component. It still needs verifying: GET logs-*/_ilm/explain shows the phase each index is actually in and any error that stalled it.

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {"actions": {}, "min_age": "0ms"},
      "warm": {"min_age": "7d", "actions": {}},
      "cold": {"min_age": "30d", "actions": {}},
      "delete": {"min_age": "90d", "actions": {"delete": {}}}
    }
  }
}

Query

GET logs-*/_search
{
  "query": {
    "match": {"message": "error"}
  }
}

Choose between Loki and Elasticsearch

For a Grafana-only stack, Loki is the natural choice. For a general-purpose log analysis stack, Elasticsearch.

Knowledge check