Lab: Build a central log pipeline with Vector and Loki

This lab builds a central log pipeline: Vector on a host ships logs to a local Loki, accessed through Grafana. By the end you will have an end-to-end pipeline.

Tasks

Task 1: Install Loki

# Download from grafana.com
wget https://github.com/grafana/loki/releases/download/v*/loki-*.zip
unzip loki-*.zip
sudo cp loki-*/loki /usr/local/bin/

# Minimal config
sudo mkdir -p /etc/loki /var/lib/loki
sudo tee /etc/loki/loki.yaml <<EOF
server:
  http_listen_port: 3100
common:
  ring:
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /var/lib/loki
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
storage_config:
  tsdb_shipper:
    active_index_directory: /var/lib/loki/tsdb-index
    cache_location: /var/lib/loki/tsdb-cache
  filesystem:
    directory: /var/lib/loki/chunks
limits_config:
  retention_period: 30d
EOF

# Run
loki -config.file=/etc/loki/loki.yaml &

Task 2: Install Vector

curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | sh

Task 3: Configure Vector to ship to Loki

# /etc/vector/vector.yaml
sources:
  syslog:
    type: file
    include:
      - /var/log/syslog
    read_from: beginning

  journald:
    type: journald

transforms:
  add_host:
    type: remap
    inputs:
      - syslog
      - journald
    source: |
      .host = get_hostname!()

sinks:
  loki:
    type: loki
    inputs:
      - add_host
    endpoint: http://localhost:3100
    encoding:
      codec: text
    labels:
      host: "{{ host }}"
      job: vector
    buffer:
      type: disk
      max_size: 268435488
      when_full: block

Validate the configuration before restarting. Vector refuses to start on an unknown field, and finding that out from systemctl status is slower than being told directly:

vector validate --config /etc/vector/vector.yaml
sudo systemctl restart vector

Why not the generic http sink

A tempting first attempt is to point the generic http sink at http://localhost:3100/loki/api/v1/push with codec: json. It does not work, for two separate reasons, and both are worth understanding because they recur whenever you wire a shipper to a structured ingest API.

First, /loki/api/v1/push does not accept arbitrary JSON. It accepts one specific envelope:

{"streams":[{"stream":{"job":"vector"},"values":[["1723377600000000000","the log line"]]}]}

The http sink emits a plain array of events, so Loki answers HTTP 400 and every line is dropped. The dedicated loki sink exists precisely to build that envelope, split the events into streams by label, and order the values by nanosecond timestamp.

Second, labels: is a parameter of the loki sink, not the http sink. Vector fails configuration validation with an unknown-field error, so it never starts at all.

Note also that ${HOSTNAME} is a shell variable, not an exported environment variable. It is empty in the environment of a systemd unit, so Vector’s interpolation has nothing to read. Use get_hostname!() in a remap transform, as above, and reference the resulting field with "{{ host }}".

Task 4: Generate log events

logger "Lab test message"
journalctl -n 1

Task 5: Verify Loki is receiving

Two checks. First confirm the labels exist at all, then pull actual lines back:

# Which label values has Loki seen?
curl -s http://localhost:3100/loki/api/v1/label/job/values | jq

# Retrieve log lines
curl -sG http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="vector"}' \
  --data-urlencode 'limit=5' | jq '.data.result | length'

The second command must print a number greater than 0. If it prints 0, nothing has been ingested; go back to journalctl -u vector and look for HTTP 400 responses from Loki.

Verify with a query that can fail loudly

Two details in that command matter more than they look.

Use /loki/api/v1/query_range, not /loki/api/v1/query. The plain query endpoint is the instant metric query endpoint; a log-line selector sent to it returns an empty result rather than an error, so a completely dead pipeline looks the same as a quiet one.

Pass the selector through curl -sG --data-urlencode. A LogQL selector contains {, }, " and =. Written inline as ?query={host="web1"} the shell performs brace expansion and strips the quotes, so Loki receives a different query from the one you typed.

The habit to take away: a verification step that returns an empty result on failure is not a verification step. Count the rows and assert the count, so an empty pipeline fails the check instead of passing it quietly. This is exactly how real log pipelines end up dropping everything while the dashboard looks healthy.

Task 6: Install Grafana

sudo apt install grafana
sudo systemctl enable --now grafana-server

Open http://localhost:3000.

Task 7: Add Loki data source

Configuration > Data sources > Add data source.
Type: Loki.
URL: http://localhost:3100.
Save and test.

Task 8: Explore logs

Explore > Loki data source.
Query: {job="vector"} or {job="vector"} |= "sshd".
Narrow to one machine with {job="vector", host="web1"}.
See live log events.

Task 9: Document

LOKI LAB
========
Host: <host>
Date: 2026-08-09

Components:
- Loki: http://localhost:3100
- Vector: /etc/vector/vector.yaml
- Grafana: http://localhost:3000

Sources:
- /var/log/syslog
- systemd journal

Destination: Loki (dedicated `loki` sink, disk buffer)
Labels: job=vector, host=<host>

Verification:
- vector validate --config /etc/vector/vector.yaml passes
- query_range for {job="vector"} returns a non-zero count
- Grafana queries return events
- Loki stopped for 2 minutes: no lines lost, buffer replayed

Validation

Run these from the lab host.

# Both units are enabled, so the pipeline survives a reboot.
systemctl is-enabled loki vector grafana-server
systemctl is-active  loki vector grafana-server

# The configuration Vector is actually running is valid.
vector validate --config /etc/vector/vector.yaml

# Vector is not silently erroring on every send.
sudo journalctl -u vector --since '5 minutes ago' | grep -iE 'error|400|refused' \
  || echo "no send errors"

# Loki has seen the labels.
curl -s http://localhost:3100/loki/api/v1/label/job/values | jq -r '.data[]'
# expect: vector

# Log lines come back. Assert the count - an empty result is a failure,
# not a quiet success.
curl -sG http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="vector"}' \
  --data-urlencode 'limit=5' | jq '.data.result | length'
# expect: greater than 0

Prove the disk buffer survives an outage

The buffer: type: disk / when_full: block setting in Task 3 is the difference between a log pipeline and a log lossy pipeline, and it is worth nothing until you have seen it work. Take the sink away and confirm nothing is lost:

# Emit a marker we can count precisely, then take Loki down.
MARKER="buffer-test-$(date +%s)"
sudo systemctl stop loki

# 60 lines over two minutes, with Loki unreachable the whole time.
for i in $(seq 1 60); do logger "$MARKER seq=$i"; sleep 2; done

# Vector should be buffering to disk, not discarding. The buffer grows:
sudo du -sh /var/lib/vector/*
sudo journalctl -u vector --since '3 minutes ago' | tail -5

# Bring Loki back and give Vector time to drain.
sudo systemctl start loki
sleep 60

# All 60 lines must be present. Fewer means the buffer was not doing its job.
curl -sG http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode "query={job=\"vector\"} |= \"$MARKER\"" \
  --data-urlencode 'limit=200' \
  | jq '[.data.result[].values[]] | length'
# expect: 60

A count below 60 means the buffer filled and dropped, or the sink was configured with when_full: drop_newest. Check max_size against your real log rate and the longest outage you intend to survive: 256 MB of buffer at 1 MB/s of logs buys you roughly four minutes, not the hour you may be assuming.

Expected outcome

Loki, Vector and Grafana run under systemd and are enabled.
vector validate passes against the running configuration.
Loki reports job=vector as a known label value.
A query_range for {job="vector"} returns a non-zero count of real log lines, from both /var/log/syslog and the journal.
Grafana’s Explore view returns live events for {job="vector"} and narrows correctly by host.
With Loki stopped for two minutes, Vector buffers to disk and replays every line on recovery — verified by an exact count, not by the dashboard looking populated.

Cleanup

# Stop and disable the pipeline.
sudo systemctl disable --now grafana-server vector loki

# Remove the lab configuration and the on-disk buffer.
sudo rm -f /etc/vector/vector.yaml
sudo rm -rf /var/lib/vector

# Remove packages if this host was not meant to keep them.
sudo apt-get remove --purge -y grafana loki

# Confirm nothing is still listening on the lab ports.
ss -ltnp | grep -E ':(3000|3100)\b' || echo "all lab ports closed"

Loki and Grafana ship with no authentication on these ports. If you keep the stack, bind it to localhost or put it behind the host firewall before you walk away from it — a central log store is, by construction, the most sensitive thing on the host.

Objectives

Prerequisites