Fluent Bit and Vector - modern log shippers

Fluent Bit and Vector are modern log shippers. They handle structured logs, parse, enrich, and route to multiple destinations. This lesson covers the basics.

Fluent Bit

Fluent Bit is the CNCF-graduated log processor. It reads from sources, parses, and ships to destinations.

Install

# Debian/Ubuntu
sudo apt install fluent-bit

# Or from the repo
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Configuration

# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush 5
    Daemon off
    Log_Level info
    # Filesystem buffering: without it, records held in memory are lost
    # when the aggregator is unreachable and fluent-bit is restarted.
    storage.path /var/lib/fluent-bit/storage
    storage.sync normal
    storage.backlog.mem_limit 64M

[INPUT]
    Name tail
    Path /var/log/syslog
    Tag syslog
    storage.type filesystem
    # Without a DB, a restart re-reads every file from the beginning
    DB /var/lib/fluent-bit/tail.db

[INPUT]
    Name systemd
    Tag journal.*
    storage.type filesystem

# The two inputs produce DIFFERENT field names, so they need
# different filters. See the callout below.
[FILTER]
    Name grep
    Match syslog
    Regex log sshd

[FILTER]
    Name grep
    Match journal.*
    Regex _SYSTEMD_UNIT ssh

[OUTPUT]
    Name forward
    Match *
    Host logs.example.com
    Port 24224
    tls On
    tls.verify On

This reads syslog files and the systemd journal, filters each for sshd events using the field name that input actually emits, and forwards to a Fluentd/Vector aggregator over TLS.

Vector

Vector is a newer, Rust-based log shipper. It has a similar feature set to Fluent Bit but is faster and uses less memory.

Install

curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | sh

Configuration (YAML)

# /etc/vector/vector.yaml
sources:
  syslog:
    type: file
    include:
      - /var/log/syslog
    read_from: beginning

  journald:
    type: journald

transforms:
  sshd_only:
    type: filter
    inputs:
      - syslog
      - journald
    condition:
      type: vrl
      source: 'includes(string!(.message), "sshd")'

sinks:
  central:
    type: http
    inputs:
      - sshd_only
    uri: https://logs.example.com/v1/logs
    encoding:
      codec: json

This reads from syslog files and journald, filters for sshd events, and ships to a central HTTP endpoint.

Choose between Fluent Bit and Vector

• Fluent Bit: more mature, larger ecosystem, more plugins.
• Vector: faster (Rust), less memory, simpler config (YAML vs Fluent Bit’s custom syntax).

For new deployments, Vector is the more modern choice. For existing Fluent Bit stacks, stay with Fluent Bit.

Common patterns

Parse JSON logs

# Vector
transforms:
  parse_json:
    type: remap
    inputs:
      - journald
    source: |
      . = parse_json!(.message)

# Fluent Bit
[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser json

Add metadata

# Vector
transforms:
  enrich:
    type: remap
    inputs:
      - syslog
    source: |
      .host = "${HOSTNAME}"
      .env = "production"

Route to multiple destinations

# Vector
sinks:
  central_logs:
    type: http
    inputs: [enrich]
    uri: https://logs.example.com/

  local_archive:
    type: file
    inputs: [enrich]
    path: /var/log/archived/{{ timestamp }}.log

Send to central and keep a local archive for compliance.

Knowledge check