Central log collection — Fluent Bit, Vector, Loki · Docker & Containers

Per-host logs answer “what did this container do”. Central logs answer “why did the order fail”, which is the question that spans six services and three hosts and is the only one anybody asks during an incident.

Getting there is not one architecture. It is three, and they differ in exactly the property that matters when things break: what happens to your application when the log destination is unavailable.

The three topologies

flowchart LR
  subgraph A["A - driver push"]
    C1[Container] --> D1[Daemon + remote driver] --> S1[(Store)]
  end
  subgraph B["B - agent tails files"]
    C2[Container] --> D2[Daemon json-file] --> F2["/var/lib/docker/containers"] --> AG2[Fluent Bit] --> S2[(Store)]
  end
  subgraph C["C - agent reads the API"]
    C3[Container] --> D3[Daemon any driver] --> API["/containers/id/logs"] --> AG3[Vector] --> S3[(Store)]
  end

	A: driver push	B: agent tails files	C: agent reads API
Driver required	`fluentd`, `gelf`, `syslog`, …	`json-file` only	any readable driver
Store outage blocks the app	Yes, by default	No	No
Local copy retained	dual-logging cache	the json files	the driver store
Extra process per host	none	one agent	one agent
Metadata available	driver-attached	parsed from path	full container object
Survives agent restart	n/a	with a position DB	with the driver store

A — the driver pushes

The daemon’s logging driver connects directly to the aggregator. No agent, no files, fewest moving parts.

It is also the topology that makes your log aggregator a synchronous dependency of every container on the host. Delivery mode defaults to blocking, so when the aggregator is unreachable the container’s write to stdout does not complete and the application stalls. mode=non-blocking with a bounded max-buffer-size mitigates it by dropping lines instead.

Choose A when you have very tight host resource constraints and you have consciously accepted the coupling and set mode=non-blocking.

B — an agent tails the JSON files

The daemon writes json-file as usual. A Fluent Bit or Vector process on the host reads /var/lib/docker/containers/*/*.log, parses, enriches, and ships.

The container never touches the network path. If the store is down, the agent buffers or stalls and the application does not notice. This is the default recommendation for a reason.

Topology B silently requires the json-file driver

The whole design depends on the daemon writing a text file the agent can parse. Switch the host to the local driver — which is otherwise the better host-side choice, because it rotates by default and costs less per line — and the files become an internal binary format the daemon reserves for itself. The tail plugin finds nothing matching its glob, or finds bytes it cannot parse.

Nothing errors loudly. Fluent Bit reports zero records for the input; Docker reports nothing at all; the store simply stops receiving from that host. You discover it the next time you search for a log and the results end at the date somebody tightened up the daemon configuration.

If you run topology B, "log-driver": "json-file" is a load-bearing setting and belongs in configuration management with a comment saying so.

C — an agent reads the Docker API

Vector’s docker_logs source retrieves logs through the Docker API rather than reading files directly. That makes it driver-agnostic: it works with json-file, local, journald, and — via the dual-logging cache — with remote drivers too.

It also gets the full container object for free, so container_name, image, and container labels arrive as structured fields rather than being reconstructed from a filesystem path. The cost is a socket dependency: the agent needs access to the Docker API.

Fluent Bit against a Docker host

The keys below are the ones that make the difference between a demo and something that survives a restart:

[SERVICE]
    Flush             5
    Daemon            off
    Log_Level         info
    storage.path      /var/lib/fluent-bit/storage
    storage.sync      normal
    storage.backlog.mem_limit  64M

[INPUT]
    Name              tail
    Path              /var/lib/docker/containers/*/*-json.log
    Parser            docker
    Tag               docker.*
    DB                /var/lib/fluent-bit/tail.db
    Docker_Mode       On
    Skip_Long_Lines   On
    Refresh_Interval  5
    Rotate_Wait       30
    Mem_Buf_Limit     32M
    storage.type      filesystem

[FILTER]
    Name              record_modifier
    Match             docker.*
    Record            host ${HOSTNAME}

[OUTPUT]
    Name              loki
    Match             *
    Host              loki.example.com
    Port              3100
    labels            job=docker, host=${HOSTNAME}
    label_keys        $container_name
    line_format       json

Four of those keys are the ones people leave out and then have an incident about:

DB persists the inode-to-offset mapping in a SQLite file. Without it, an agent restart re-reads every log file from the beginning, duplicating weeks of records into the store, or from the end, losing everything written while it was down. Neither is acceptable and the default is no DB at all.
Rotate_Wait (default 5 seconds) is how long the agent keeps watching a file after it has been rotated, to drain what is still buffered. Five seconds is not enough on a busy host; see the race below.
Docker_Mode (default false) reassembles log lines that Docker split because they exceeded its internal read buffer. Without it, a long JSON log line arrives as several fragments, none of which parses.
Skip_Long_Lines (default false) makes the agent skip a line longer than its buffer instead of stopping. Off means one pathological line can wedge the input.

Note what is not in that config: there is no kubernetes filter. That filter enriches records by querying a Kubernetes API server for pod metadata, and on a plain Docker host there is no API server to query. It is a common copy-paste from a Kubernetes example and it does nothing useful here.

Vector against a Docker host

[sources.docker]
type = "docker_logs"
auto_partial_merge = true
exclude_containers = ["vector"]

[transforms.parse]
type = "remap"
inputs = ["docker"]
source = '''
. = parse_json(string!(.message)) ?? { "message": .message }
.container_name = .container_name
'''

[sinks.loki]
type = "loki"
inputs = ["parse"]
endpoint = "http://loki.example.com:3100"
labels = { job = "docker", host = "{{ host }}", container = "{{ container_name }}" }

[sinks.loki.encoding]
codec = "json"

[sinks.loki.buffer]
type = "disk"
max_size = 536870912
when_full = "block"

exclude_containers keeps the agent from ingesting its own output, which is otherwise a genuine feedback loop: the agent logs a delivery failure, reads its own failure message, tries to deliver it, fails, logs that. On a store outage this produces exponential growth of purely self-referential records.

auto_partial_merge (default true) is Vector’s equivalent of Fluent Bit’s Docker_Mode: Docker splits log messages that exceed 16 KB, and this reassembles them.

The rotation race

This is the failure specific to topology B, and it is the reason lines go missing from the store while every component reports healthy.

Inodes, offsets, and the window in which lines die

Fluent Bit’s tail input tracks each file by inode, not by name, and stores its byte offset in the DB file. When the daemon rotates a log, it renames -json.log to -json.log.1 — same inode, new name — and creates a new file at the old name. The agent follows the inode, drains the remainder, then picks up the new file. That part works.

The failure is a race against max-file. Suppose max-size=10m, max-file=3, and the agent has fallen behind by 25 MB because the store was briefly slow. The container is writing at 5 MB/s, so a rotation happens every two seconds and the oldest file is unlinked every six. The agent is trying to read a file the driver has already deleted.

Because the agent holds an open descriptor, it can still read what it has — unlink does not invalidate a descriptor — but only for Rotate_Wait seconds, after which it closes the file and moves on. Anything not read by then is gone. Permanently. No error is emitted, because from the agent’s point of view it finished with a rotated file, which is the normal case.

Two mitigations, and you want both:

Widen the window. Rotate_Wait 30 instead of the default 5. It costs a few open descriptors and buys six times the grace period.

Widen the buffer. The real defence is a log ceiling large enough that the agent can fall behind for as long as your worst realistic store outage. max-size times max-file divided by peak bytes per second is the number of seconds of agent downtime you can absorb. At 5 MB/s with a 30 MB ceiling, that is six seconds. Six seconds is not an outage budget. Either raise the ceiling or accept that a store hiccup costs you logs.

Read-only / Safemeasure your own budget— How many seconds of shipper downtime the current configuration can absorb for the busiest container.

CONTAINER=api
CID=$(docker inspect --format '{{.Id}}' "$CONTAINER")
LOGFILE="/var/lib/docker/containers/$CID/$CID-json.log"

A=$(sudo stat -c %s "$LOGFILE"); sleep 10; B=$(sudo stat -c %s "$LOGFILE")
RATE=$(( (B - A) / 10 ))

SIZE_MB=$(docker inspect --format '{{index .HostConfig.LogConfig.Config "max-size"}}' "$CONTAINER")
FILES=$(docker inspect --format '{{index .HostConfig.LogConfig.Config "max-file"}}' "$CONTAINER")
echo "rate=$RATE bytes/s  max-size=$SIZE_MB  max-file=$FILES"
echo "buffer seconds = (max-size bytes * max-file) / $RATE"

Choosing the store

Store	Indexing model	Strength	Watch out for
Loki	Labels only; log body unindexed	Cheapest at volume; native Grafana	Label cardinality
Elasticsearch / OpenSearch	Full text	Arbitrary queries, aggregations	Cost and cluster operations
Splunk	Full text	Compliance tooling, mature RBAC	Licensing by ingest volume
CloudWatch Logs	Limited	Zero operations on AWS	Query ergonomics, per-GB cost
Datadog / SaaS	Full text	No infrastructure	Ingest-based pricing surprises

For a team standing something up today: Loki and Grafana. It is free, it operates simply, and its cost model matches container logging — you pay for bytes stored, not for indexing every field of every line.

Loki cardinality: the label that takes the cluster down

Loki builds one stream per unique combination of label values, and each stream carries its own index entry and chunk. That is cheap for {job="docker", host="web-01"} — a handful of streams per host.

It is catastrophic for {container_id="..."}. Container IDs are unique per container, and every deploy replaces every container. A fleet doing ten deploys a day across forty services generates hundreds of new streams daily and never reuses one. Ingestion slows, the index grows without bound, queries that touch a time range spanning many deploys time out, and the failure looks like “Loki is slow” rather than “we chose a bad label”.

Labels are for the small, bounded, stable dimensions you filter on: job, host, service, env, stream. Everything high-cardinality — container ID, request ID, user ID, trace ID — belongs in the log line, which Loki stores and can grep at query time without indexing.

This is one of very few configuration mistakes in the logging stack that is genuinely hard to undo, because the streams already exist.

Verification that can fail

Configuration audits prove nothing about a pipeline. Emit a unique marker on the host and confirm it arrives at the far end.

Read-only / Safeend-to-end probe— One marker, three checkpoints. Whichever one is missing tells you which hop broke.

MARKER="probe-$(hostname -s)-$(date +%s)"

docker run --rm alpine:3.20 echo "$MARKER"

# Checkpoint 1: did the daemon capture it? (json-file / local / journald)
sudo grep -rl --include='*-json.log' -F "$MARKER" /var/lib/docker/containers/ \
&& echo 'OK: daemon captured the line' \
|| echo 'FAIL: driver did not write it locally'

# Checkpoint 2: did the shipper read it?
docker logs --since 2m fluent-bit 2>&1 | tail -20

# Checkpoint 3: query the store. Adjust for your Loki endpoint.
LOKI=http://loki.example.com:3100
curl -sG "$LOKI/loki/api/v1/query_range" \
--data-urlencode 'query={job="docker"}' \
--data-urlencode "start=$(( $(date +%s) - 300 ))000000000" \
| grep -q -F "$MARKER" \
&& echo 'PASS: marker reached the store' \
|| echo 'FAIL: marker not in the store'

Run it from cron on one host per rack and alert when it fails. A logging pipeline that nobody probes is a pipeline you discover is broken on the day you need it — and the evidence you needed was never collected, so there is no recovering it retrospectively.

Also watch the shipper’s own metrics, because “it is running” is not “it is delivering”:

Read-only / Safeshipper health— Fluent Bit exposes per-plugin counters on its HTTP endpoint when HTTP_Server is on.

$ curl -s http://127.0.0.1:2020/api/v1/metrics | python3 -m json.tool

{
  "input": {
      "tail.0": { "records": 184203, "bytes": 71223104, "files_opened": 62, "files_rotated": 41 }
  },
  "output": {
      "loki.0": { "proc_records": 183940, "errors": 0, "retries": 12, "retries_failed": 0 }
  }
}

Illustrative output

The number to alert on is retries_failed. Nonzero means records were dropped after exhausting the retry budget: they existed, the agent read them, and they are now gone. records minus proc_records growing steadily means the agent is falling behind, which is the leading indicator of the rotation race above.

What happens when the central store goes down for an hour?

It depends entirely on which topology you chose, which is the practical reason to care about the distinction.

Topology A, default settings. Containers block on stdout writes. The application outage begins minutes into the store outage. Recovery requires restoring the store or recreating every container with a different driver.

Topology A, mode=non-blocking. The ring buffer absorbs max-buffer-size (default 1 MB) and then drops. The application is fine; an hour of logs is gone.

Topology B. The agent stops delivering and applies backpressure to itself. With storage.type filesystem and a storage.path it spools to disk and delivers the backlog when the store returns, so you lose nothing — provided the spool directory has room, which is a partition you must size. Without filesystem storage it buffers in memory up to Mem_Buf_Limit and then pauses the input, at which point the rotation race decides how much you lose.

Topology C. Same as B if you configured a disk buffer, which for Vector means [sinks.loki.buffer] type = "disk" and when_full = "block". With the default memory buffer, you lose the overflow.

The pattern: the local ceiling — spool size in B and C, ring buffer in A — is the only thing standing between a store outage and permanent data loss. Size it against the outage you actually expect, not the one you hope for.

Knowledge check

Why this matters

The mistake that costs most is not choosing the wrong store. Stores are replaceable; the shipping layer in front of them is what you actually operate.

The mistake that costs most is failing to notice which component you have made load-bearing. Topology A makes the aggregator load-bearing for application availability. Topology B makes "log-driver": "json-file" load-bearing, in a file that looks like a tuning setting and gets changed by someone tidying up. Topology C makes the Docker socket, and therefore root on the host, part of the logging agent’s threat surface.

None of those is wrong. All of them are invisible until they fail, and each one fails in a way that points somewhere else — the application is slow, the store is empty, the agent looks healthy. Write the dependency down where the next engineer will see it, probe the pipeline end to end on a schedule, and alert on retries_failed rather than on whether the process is running.