Lab: Deploy Loki

Objective

By the end of this lab you will have a running Loki that you can say four true things about: the process is up, its endpoints answer, a log line pushed into it comes back out and lands on disk, and its retention and limits are the ones the written policy says they should be. You will also have started the same Loki wrongly, on purpose, and seen it look completely healthy while accepting nothing.

That last part is the reason this lab exists. Loki is easy to start and hard to declare healthy. A misconfigured deployment binds its port, serves /metrics, reports Running to Docker, and returns 204 to a client that is losing every line. “The container is up” is a status check. What follows is a validation.

Architecture

One container, one config file, one named volume. Everything a production Loki does is present here — distributor, ingester, querier, compactor — just inside one process rather than a fleet.

     curl (the client, the operator, and the smoke test)
        |
        | POST /loki/api/v1/push        GET /loki/api/v1/query_range
        | GET  /ready /services /config /metrics
        v
  +---------------------------------------------+
  |  loki 3.3, -target=all (single binary)      |
  |                                             |
  |  distributor -> ingester -> WAL             |
  |                      |                      |
  |                      | flush                |
  |                      v                      |
  |  querier <------ /loki/chunks (filesystem)  |
  |  compactor ----> index + retention          |
  +---------------------+-----------------------+
                        |
                        v
              docker volume: loki-data
                /loki/wal      write-ahead log
                /loki/chunks   flushed chunks, per tenant
                /loki/compactor working directory

The volume is where the lab’s two durability proofs live. A chunk in the ingester’s memory is a log line you have not stored yet; a chunk under /loki/chunks is one you have. The distance between those two states is measured in minutes, and knowing that number for your own configuration is most of what “operating Loki” means.

Requirements

• A disposable Linux host with Docker Engine 28.x and the Compose v2 plugin. The lab binds two loopback ports (3100 and, briefly, 3101), creates one named volume and one directory under your home.
• curl and jq on the host.
• GNU date. The lab builds nanosecond timestamps with date +%s and arithmetic, and one step deliberately builds a timestamp eight days in the past with date -u -d '8 days ago'.
• About 1 GB of free disk and 1 GB of RAM headroom. The corpus in this lab is a handful of lines; the disk is for the image.
• No out-of-band access requirement. Nothing here touches the host network configuration, the firewall, or SSH.

Scenario

Your team is about to point forty hosts at a new log store. The last one was declared ready on a Friday because the container was running and the port answered; on Monday the dashboards were empty and nobody could say when the logs had stopped arriving, because nothing had ever proved they were arriving in the first place.

You have been asked to stand this one up and to hand over evidence rather than an assertion. The deliverable is not the running Loki. It is the script that says whether the running Loki works, and the written policy it checks against.

Tasks

Task 1: Decide the mode, and write down why

Loki 3.x ships three deployment modes out of one binary, selected by the -target flag: all (single binary), the read / write / backend split (simple scalable), and one process per component (microservices). The choice is not about ambition, it is about volume and about how many people are going to carry the pager.

LABDIR="$HOME/rb-obs-loki-deploy"
mkdir -p "$LABDIR"
cd "$LABDIR"

cat > "$LABDIR/mode-decision.md" <<'DOC'
# Loki deployment mode decision

Estimated ingest:   40 hosts x ~25 MB/day = ~1 GB/day
Peak allowance:     4x average, so size for ~4 GB/day
Mode chosen:        single binary (-target=all)
Rationale:          well under the ~50 GB/day point at which one process stops
                    being the right shape; one config file, one restart, one
                    thing to reason about at 03:00.
Storage:            filesystem in this lab; object storage before production.
Trigger to revisit: sustained ingest above 20 GB/day, OR the first time an
                    ingest spike makes the querier unavailable, whichever
                    comes first.
DOC

cat "$LABDIR/mode-decision.md"

Write your own numbers into that file rather than keeping mine. The value is not the answer, it is the recorded trigger: a mode chosen with a written revisit condition is a decision, and a mode chosen because the demo worked is a one-way door you will meet again during an incident.

Task 2: Write the configuration

Every section below is one of the top-level blocks from the configuration lesson. Read the comments — the failure mode of each section is named next to it.

# loki-config.yaml
# Single binary. -target defaults to `all`, so every component runs in this
# one process and this one file is the union of every per-target section.

# Multi-tenant when true, in which case every request must carry an
# X-Scope-OrgID header. False here means single-tenant, and every write lands
# under the default tenant id, which you will see on disk in Task 4.
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      # One process, so the ring has one member and needs no external KV
      # store. A simple-scalable or microservices deployment replaces this
      # with consul or etcd, and a replica that cannot reach it never joins
      # the ring and never appears at all.
      store: inmemory

# The chain of schemas this cluster has used. Append-only: removing an entry
# breaks reads of everything written under it, because Loki picks the schema
# by the timestamp of the data.
schema_config:
  configs:
    - from: 2024-04-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  # Retention is enforced by the compactor, globally, from this value. It is
  # the one number in this file whose wrong value is a compliance event
  # rather than an outage: logs older than it are deleted, quietly and on
  # schedule.
  retention_period: 744h          # 31 days

  # Enforced by the distributor as a token bucket, per tenant. A limit that
  # never fires is a comment; this one is deliberately small enough that a
  # misbehaving client meets it.
  ingestion_rate_mb: 8
  ingestion_burst_size_mb: 16

  # Per-stream, so one noisy stream cannot consume the tenant's whole budget.
  per_stream_rate_limit: 4MB
  per_stream_burst_size: 8MB

  # Rejects lines whose timestamp is too far in the past: clock-skewed hosts
  # and replay pipelines. Task 7 trips this one on purpose.
  reject_old_samples: true
  reject_old_samples_max_age: 168h

  max_entries_limit_per_query: 5000

ingester:
  # How long a stream may sit idle, and how old a chunk may get, before the
  # ingester flushes it to storage. Short values here so the flush is
  # observable inside the lab window; production values are longer, and the
  # gap between "pushed" and "durable" is exactly this long.
  chunk_idle_period: 2m
  max_chunk_age: 10m
  wal:
    # The difference between a restart that loses the last few minutes of
    # every stream and one that replays them. On by default in Loki 3.x;
    # written out here because it is worth being explicit about.
    enabled: true
    dir: /loki/wal

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  # Retention only happens when this is true. With it false, the
  # retention_period above is documentation.
  retention_enabled: true
  retention_delete_delay: 2h
  # Required whenever retention is enabled. Leaving it out is the misconfig
  # in Task 8, and Loki refuses to start rather than run without it.
  delete_request_store: filesystem

analytics:
  reporting_enabled: false

compose.yaml:

# compose.yaml
name: rb-obs-loki-deploy

services:
  loki:
    image: grafana/loki:3.3.0
    container_name: rb-loki
    command: ['-config.file=/etc/loki/loki-config.yaml']
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
      - loki-data:/loki
    ports:
      - '127.0.0.1:3100:3100'

volumes:
  loki-data:

And the policy the deployment is supposed to satisfy. This file is what makes the fourth validation class possible: without it, “is the retention right?” has no answer.

cat > "$LABDIR/policy.env" <<'POLICY'
# What this deployment is supposed to be. The smoke test compares the running
# config against these values; a difference is a finding, not a preference.
POLICY_RETENTION=744h
POLICY_INGESTION_RATE_MB=8
POLICY_REJECT_OLD=true
POLICY_TENANCY=single
POLICY

cat "$LABDIR/policy.env"

Task 3: Start it, and run the first two validation classes

$ cd ~/rb-obs-loki-deploy && docker compose up -d

Class 1, process status. The weakest check in the set, and the one most deployments stop at:

cd "$LABDIR"
docker compose ps

Class 2, endpoint status. Three endpoints, each answering a different question:

# Is every component ready? The ingester is only ready after WAL replay, so a
# 503 for the first few seconds is the endpoint working.
for i in $(seq 1 30); do
  curl -sf http://127.0.0.1:3100/ready >/dev/null && break
  sleep 2
done
curl -s http://127.0.0.1:3100/ready

# Which components did this process actually start? This is the direct answer
# to "did the -target flag do what I meant", and it is the check Task 6 turns
# into a diagnosis.
curl -s http://127.0.0.1:3100/services

# What configuration is the running process using? Not the file on disk - the
# resolved config, with every default filled in.
curl -s http://127.0.0.1:3100/config | head -40

Look at the /services output and find distributor, ingester, querier and compactor in it. On a single binary they are all there. On a simple-scalable write target you would see the first two and not the last two, and that difference is the whole diagnostic in Task 6.

Task 4: Class 3, functional status — push, query, and find the bytes

A push that returns 204 has been accepted by the distributor. It has not necessarily been stored, and it has certainly not been proved queryable. Do all three:

NOW_NS=$(( $(date -u +%s) * 1000000000 ))
MARKER="smoke-$NOW_NS"

jq -nc --arg ts "$NOW_NS" --arg line "$MARKER" \
  '{streams:[{stream:{job:"smoke", env:"lab", level:"info"},
              values:[[$ts, $line]]}]}' \
| curl -sf -X POST http://127.0.0.1:3100/loki/api/v1/push \
    -H 'Content-Type: application/json' --data-binary @- \
    -w 'push: HTTP %{http_code}\n'

Query it straight back. It will be there, and where it comes from is the interesting part:

curl -sG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="smoke"}' \
  --data-urlencode 'since=15m' \
| jq -r '.data.result[0].values[][1]'

Now look at storage. The line just came back, so it is tempting to conclude it is stored:

docker compose exec loki find /loki -maxdepth 2 -type d
docker compose exec loki find /loki/chunks -type f | head
docker compose exec loki find /loki/wal -type f | head

There is a write-ahead log and there are probably no chunks. The querier answered from the ingester’s memory, which it does for recent data, and the chunk will not be written until chunk_idle_period or max_chunk_age fires. Force it and look again:

curl -sf -X POST http://127.0.0.1:3100/flush -w 'flush: HTTP %{http_code}\n'
sleep 5
docker compose exec loki find /loki/chunks -type f | head

Now there are files, under a directory named for the tenant. With auth_enabled: false every write lands under Loki’s default tenant id, which is why a single-tenant cluster still has a tenant directory on disk — and why turning auth_enabled on later, without a proxy that supplies the same id, makes the existing data invisible rather than deleted.

Task 5: Prove the restart is survivable

Push a marker into a new stream, do not flush it, and count the chunk files first so that afterwards you can say which mechanism saved it:

NOW_NS=$(( $(date -u +%s) * 1000000000 ))
WAL_MARKER="wal-$NOW_NS"
echo "$WAL_MARKER" > "$LABDIR/wal-marker.txt"

jq -nc --arg ts "$NOW_NS" --arg line "$WAL_MARKER" \
  '{streams:[{stream:{job:"wal-test", env:"lab", level:"info"},
              values:[[$ts, $line]]}]}' \
| curl -sf -X POST http://127.0.0.1:3100/loki/api/v1/push \
    -H 'Content-Type: application/json' --data-binary @- \
    -w 'push: HTTP %{http_code}\n'

BEFORE=$(docker compose exec -T loki find /loki/chunks -type f | wc -l)
echo "chunk files before the restart: $BEFORE"

$ cd ~/rb-obs-loki-deploy && docker compose restart loki

for i in $(seq 1 30); do
  curl -sf http://127.0.0.1:3100/ready >/dev/null && break
  sleep 2
done

curl -sG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="wal-test"}' \
  --data-urlencode 'since=30m' \
| jq -r '.data.result[0].values[][1]' | grep -F "$(cat "$LABDIR/wal-marker.txt")" \
  && echo "MARKER SURVIVED THE RESTART"

AFTER=$(docker compose exec -T loki find /loki/chunks -type f | wc -l)
echo "chunk files after the restart: $AFTER (before: $BEFORE)"

The marker came back. Now work out how, because the two answers have different failure modes. If the chunk count is unchanged, nothing reached storage across the restart and the line was replayed out of the write-ahead log. If the count grew, the ingester flushed on its way down and you are reading a chunk.

Both are durability, and only one of them survives a process that is not asked politely. A docker compose restart sends a signal and waits; an OOM kill, a docker kill, or a host that loses power do not. That is the case the WAL exists for, and it is why “we restart cleanly” is not an argument against having one.

Task 6: The wrong-target start

This is the failure the lesson calls a silent ingestion outage. Start a second Loki from the same config file, with one flag changed, on its own port:

docker run -d --name rb-loki-wrongtarget \
  -v "$LABDIR/loki-config.yaml:/etc/loki/loki-config.yaml:ro" \
  -p 127.0.0.1:3101:3100 \
  grafana/loki:3.3.0 \
  -config.file=/etc/loki/loki-config.yaml -target=ingester

sleep 10
docker ps --filter name=rb-loki-wrongtarget --format '{{.Names}} {{.Status}}'

Run every check an operator would think to run, and note that they pass:

curl -s -o /dev/null -w 'metrics: HTTP %{http_code}\n' http://127.0.0.1:3101/metrics
curl -s -o /dev/null -w 'config:  HTTP %{http_code}\n' http://127.0.0.1:3101/config
curl -s http://127.0.0.1:3101/services

The process is up, the port is bound, /metrics serves, /config is correct — it is the same file. /services is the one output that differs, and it names the problem: the components this process started are not the components a client needs. Now push at it and record what comes back:

NOW_NS=$(( $(date -u +%s) * 1000000000 ))
jq -nc --arg ts "$NOW_NS" \
  '{streams:[{stream:{job:"wrong-target"}, values:[[$ts, "does this land?"]]}]}' \
| curl -s -o /dev/null -X POST http://127.0.0.1:3101/loki/api/v1/push \
    -H 'Content-Type: application/json' --data-binary @- \
    -w 'push to wrong target: HTTP %{http_code}\n'

Write the status code into your notes. Whatever it is, the operational shape is the same: an agent pointed at this endpoint is not storing logs, and nothing in the container’s status, its port, or its metrics endpoint says so. The check that catches it is /services, and the check that catches it without anyone looking is the functional class from Task 4 — push a line, read it back.

docker rm -f rb-loki-wrongtarget

Task 7: Trip a limit on purpose and read the evidence

Limits are enforced at the distributor. Confirm the ones you set are actually loaded, then break one:

curl -s http://127.0.0.1:3100/config | tr -d ' "' \
| grep -E 'retention_period:|ingestion_rate_mb:|reject_old_samples:|reject_old_samples_max_age:'

Now push a line dated eight days ago, which is outside reject_old_samples_max_age. This is what a host with a badly skewed clock, or a replay of an old file, looks like from Loki’s side:

OLD_NS=$(( $(date -u -d '8 days ago' +%s) * 1000000000 ))

jq -nc --arg ts "$OLD_NS" \
  '{streams:[{stream:{job:"skewed-clock"}, values:[[$ts, "eight days late"]]}]}' \
| curl -s -X POST http://127.0.0.1:3100/loki/api/v1/push \
    -H 'Content-Type: application/json' --data-binary @- \
    -w '\nold sample push: HTTP %{http_code}\n'

The response body names the reason, and the server counts it:

curl -s http://127.0.0.1:3100/metrics | grep '^loki_discarded_samples_total'

Read the reason label rather than assuming it: the same metric family reports rate limiting, oversized labels and old samples under different reasons, and the reason is the whole diagnosis. A client that sees this in production is not misbehaving in one way — it is misbehaving in a specific way that this label names for you.

Task 8: Retention, the compactor, and the misconfiguration that stops it

Retention is not a property of the storage backend; it is work the compactor does. Confirm the component is running and find out what it reports about itself:

curl -s http://127.0.0.1:3100/services | grep -i compactor
curl -s http://127.0.0.1:3100/metrics | grep -i '^loki_compactor' | head -20

Enumerate rather than guess at a metric name — the set differs between Loki versions, and the one you want is whichever counter or timestamp advances on each compaction_interval. Note two or three of them in your handover; they are what “is retention running?” looks like at 03:00.

Now break it in the way most people break it. Comment out delete_request_store and restart:

cd "$LABDIR"
cp loki-config.yaml loki-config.yaml.good
sed -i 's/^  delete_request_store:/#  delete_request_store:/' loki-config.yaml
grep -n 'retention_enabled\|delete_request_store' loki-config.yaml

docker compose restart loki
sleep 8
docker compose ps
docker compose logs --no-log-prefix --tail=15 loki

Loki 3.x validates this pair at startup: retention is enabled and there is nowhere to record delete requests, so it stops and names the key. Read the log line rather than skimming the exit code — it is the entire diagnosis, and this is the good kind of failure: loud, immediate, and specific. Put the file back:

mv loki-config.yaml.good loki-config.yaml
docker compose restart loki
for i in $(seq 1 30); do
  curl -sf http://127.0.0.1:3100/ready >/dev/null && break
  sleep 2
done
curl -s http://127.0.0.1:3100/ready

Retention deletion itself is not observable inside a 90-minute lab: the compactor works on index tables at its compaction_interval, and deletion is further delayed by retention_delete_delay. What is observable, and what belongs in the handover, is that retention is enabled, that the compactor is running, and that the configured period matches the policy. Leave the lab with those three facts checked and a note of which metric you would watch tomorrow.

Task 9: Assemble the smoke test

Everything above becomes one script with an exit code. This is the deliverable — the thing you would run after every config change, every upgrade and every maintenance window, and the thing that would have caught the wrong-target start before forty hosts were pointed at it.

cat > "$LABDIR/smoke-loki.sh" <<'SMOKE'
#!/usr/bin/env bash
# Four-class validation for a single-binary Loki.
#   1 process   2 endpoints   3 functional round trip   4 semantic vs policy
# Non-zero exit = do not point a fleet at this cluster.
set -euo pipefail

cd "$(dirname "$0")"
# shellcheck disable=SC1091
. ./policy.env

LOKI=${LOKI:-http://127.0.0.1:3100}
fail() { echo "FAIL: $1"; exit 1; }

echo "[1/4] process"
docker compose ps --status running --format '{{.Service}}' | grep -q '^loki$' \
  || fail "the loki container is not running"

echo "[2/4] endpoints"
curl -sf "$LOKI/ready" >/dev/null || fail "/ready is not answering 2xx"
SERVICES=$(curl -sf "$LOKI/services") || fail "/services is not answering"
for COMPONENT in distributor ingester querier compactor; do
  echo "$SERVICES" | grep -q "$COMPONENT" \
    || fail "$COMPONENT is not running in this process - check the -target flag"
done

echo "[3/4] functional round trip"
NOW_NS=$(( $(date -u +%s) * 1000000000 ))
MARKER="smoke-$NOW_NS"
jq -nc --arg ts "$NOW_NS" --arg line "$MARKER" \
  '{streams:[{stream:{job:"smoke", env:"lab", level:"info"},
              values:[[$ts, $line]]}]}' \
| curl -sf -X POST "$LOKI/loki/api/v1/push" \
    -H 'Content-Type: application/json' --data-binary @- \
  || fail "the push was rejected"

sleep 2
curl -sfG "$LOKI/loki/api/v1/query_range" \
  --data-urlencode 'query={job="smoke"}' --data-urlencode 'since=5m' \
| grep -qF "$MARKER" || fail "the line pushed a moment ago did not come back"

curl -sf -X POST "$LOKI/flush" >/dev/null || fail "the flush endpoint refused"
sleep 5
docker compose exec -T loki find /loki/chunks -type f | grep -q . \
  || fail "no chunk files on disk after a flush - storage is not durable"

echo "[4/4] semantic"
# /config answers YAML, so read it defensively rather than with a parser that
# assumes a format.
cfg() { curl -sf "$LOKI/config" | tr -d ' "' | sed -n "s/^.*$1:\([^,}]*\).*$/\1/p" | head -1; }
[ "$(cfg retention_period)" = "$POLICY_RETENTION" ] \
  || fail "retention is $(cfg retention_period), policy says $POLICY_RETENTION"
[ "$(cfg ingestion_rate_mb)" = "$POLICY_INGESTION_RATE_MB" ] \
  || fail "ingestion rate is $(cfg ingestion_rate_mb), policy says $POLICY_INGESTION_RATE_MB"
[ "$(cfg reject_old_samples)" = "$POLICY_REJECT_OLD" ] \
  || fail "reject_old_samples is $(cfg reject_old_samples), policy says $POLICY_REJECT_OLD"

echo "PASS: process, endpoints, round trip and policy all check out"
SMOKE

chmod +x "$LABDIR/smoke-loki.sh"
"$LABDIR/smoke-loki.sh"

Validation

The smoke test is the validation, and it is worth understanding what each class buys before trusting the PASS.

1. Run it against the healthy cluster. It should print PASS and exit 0:

"$LABDIR/smoke-loki.sh"; echo "exit=$?"

2. Run it against a cluster you have broken, and confirm it fails. This is the step people skip, and a check that has never failed is a check nobody has tested:

docker compose stop loki
"$LABDIR/smoke-loki.sh"; echo "exit=$?"
docker compose start loki
sleep 10

3. Confirm the round-trip class is the one carrying the weight. Point the script at the wrong-target Loki from Task 6 and watch which class fails first — the endpoint class, on the missing distributor:

docker run -d --name rb-loki-wrongtarget \
  -v "$LABDIR/loki-config.yaml:/etc/loki/loki-config.yaml:ro" \
  -p 127.0.0.1:3101:3100 \
  grafana/loki:3.3.0 \
  -config.file=/etc/loki/loki-config.yaml -target=ingester
sleep 10
LOKI=http://127.0.0.1:3101 "$LABDIR/smoke-loki.sh"; echo "exit=$?"
docker rm -f rb-loki-wrongtarget

4. Confirm the policy class is real. Change one value in policy.env, run the script, and watch it report the difference rather than the failure of a service:

sed -i 's/^POLICY_RETENTION=.*/POLICY_RETENTION=2160h/' "$LABDIR/policy.env"
"$LABDIR/smoke-loki.sh"; echo "exit=$?"
sed -i 's/^POLICY_RETENTION=.*/POLICY_RETENTION=744h/' "$LABDIR/policy.env"

Expected Outcome

• One Loki container running, /ready answering, and /services listing the distributor, ingester, querier and compactor.
• A chunk file under /loki/chunks in a tenant-named directory, and a write-ahead log under /loki/wal.
• A marker line that survived a container restart it was never flushed for, and a recorded chunk count before and after that says which mechanism saved it.
• Notes recording the HTTP status a push receives from a -target=ingester process, and the reason label on the discard counter after the old-sample push.
• A Loki that refused to start with retention enabled and no delete_request_store, and then started cleanly once the key was restored.
• smoke-loki.sh printing PASS on a healthy cluster and a named, specific failure on each of the three broken ones.

Troubleshooting

The container exits immediately. Read the log rather than restarting it: docker compose logs --no-log-prefix loki. Loki parses its configuration at startup and names the offending key and line. A restart loop with no log output at all is almost always a bind-mount path that does not exist on the host, which makes Docker create a directory where you meant to mount a file.

A config edit does not take effect after a restart. sed -i writes a new file and renames it, so the path now points at a different inode. If the restarted container still reports the old configuration on /config, recreate it rather than restarting it: docker compose up -d --force-recreate loki. This is the single-file bind mount’s one sharp edge, and it is why a directory mount is the safer shape for anything a script rewrites.

/ready returns 503 for longer than a minute. Read what it says: the endpoint is component-aware and names the component that is not ready. An ingester stuck in WAL replay after an unclean stop is normal and finishes; an ingester that never becomes ready usually cannot write to its WAL directory, which is a volume permission problem rather than a Loki problem.

The push returns 400. The body names the field. A timestamp in seconds rather than nanoseconds is the most common cause — Loki reads it as a moment in 1970 and rejects it as too old, which is the same rejection Task 7 triggers on purpose. A stream label outside [a-zA-Z_][a-zA-Z0-9_]* is the second.

The push returns 429. A rate limit engaged. Check the values are the ones you set with the /config grep from Task 7, and remember that the limit is per-tenant at the distributor: on a multi-replica deployment the effective aggregate is the per-replica value times the replica count.

The query returns nothing but the push returned 204. Widen the window first — since=1h — because the default is short and the marker may be outside it. If it is still absent, the ingester accepted it and the querier is looking somewhere else, which on this single-binary deployment means the schema or the storage path was changed between the push and the query.

No chunk files after POST /flush. Confirm the flush returned 2xx, wait longer than five seconds, and check you are looking inside the container rather than on the host — the chunks live in the named volume, mounted at /loki. If the image has no find, docker compose exec loki ls -R /loki/chunks | head answers the same question.

docker compose exec fails inside the smoke test. The script uses -T to disable TTY allocation, which is required when it runs from cron or CI. If you copied the command without -T, that is the difference.

Cleanup

$ cd ~/rb-obs-loki-deploy && docker compose config --volumes && docker compose down -v

docker rm -f rb-loki-wrongtarget 2>/dev/null || echo "no stray container"
docker volume ls | grep rb-obs-loki-deploy || echo "volume gone"
cd "$HOME"
rm -rf "$HOME/rb-obs-loki-deploy"

The image stays in the local cache. Remove it with docker image rm grafana/loki:3.3.0 if you want the disk back.

Production notes

Take the smoke test to production before you take the cluster there. The script in Task 9 is not a lab artefact. Schedule it, alert on a non-zero exit, and keep its output — the validation history is the honest answer to “when did this cluster last work”, which is the first question asked in every logging incident and the one nobody can usually answer.

Change the storage before you change the topology. Everything in this lab runs on a filesystem backend, which is right for a lab and wrong for anything whose loss would matter. Moving to object storage is a common.storage change plus a new schema_config entry with a future from date; it is append-only and reversible, and it is far cheaper to do at 1 GB/day than during the migration to simple scalable that a growing estate eventually forces.

Treat schema_config as an audit log. Every entry is a dated statement about how data written in that period must be read. Add entries with a from date in the near future, never in the past, and put the change request number in a comment next to each one. A cluster that cannot read its own history is usually a cluster where somebody tidied this block.

Set limits from a measurement and alert on the rejections. The values in this lab are deliberately small. In production, measure the largest client’s peak push rate, add headroom, and put an alert on any non-zero rate of loki_discarded_samples_total. A limit that never fires is a comment; a limit that fires without an alert is a silent data-loss channel with a metric nobody reads.

Write the retention number down in two places and diff them. The policy file in this lab exists because retention is the one setting whose wrong value is discovered by an auditor rather than by an engineer. The compactor enforces it on schedule, deletion is not reversible, and the gap between “what the compliance document says” and “what limits_config says” is invisible until somebody asks for a year-old log.

What You Learned

• A container status is not a health check. The wrong-target Loki passed process status, bound its port, served metrics and returned a correct /config. Only /services and a functional round trip caught it.
• A 204 is an acceptance, not a durability guarantee. You watched a query answer from ingester memory before any chunk existed, forced a flush, and found the file. Those are three different states of the same log line.
• A clean restart hides which durability mechanism you actually have. The marker survived, and the chunk count told you whether the WAL replayed it or the ingester flushed on shutdown. Only the first of those survives a process that is killed rather than asked.
• Validation has four classes and the first two are the cheap ones. Process and endpoint status are necessary and insufficient; the round trip proves the cluster works, and the policy comparison proves it is the cluster you meant to build.
• The loud failures are the kind ones. Loki refusing to start without delete_request_store cost you a minute. The silent equivalents — a rejected old sample, an ingester-only target, a WAL on an ephemeral volume — cost whatever the gap in the logs turns out to have contained.