Fluentd and Fluent Bit — the alternative log collector

Fluent Bit is the alternative log collector to Promtail. It is lightweight, supports parsers and filters, and ships the logs to multiple outputs. This lesson walks the architecture, the deployment, the configuration, and the production patterns.

The Fluent Bit architecture

The Fluent Bit architecture:

flowchart LR
    A[Container logs] --> B[Fluent Bit input]
    B --> C[Fluent Bit parser]
    C --> D[Fluent Bit filter]
    D --> E[Fluent Bit output]
    E --> F[Loki]
    E --> G[Elasticsearch]
    E --> H[S3]

The Fluent Bit architecture is the pipeline.

The Fluent Bit vs Promtail

The Fluent Bit vs Promtail:

The choice is per cluster.

The Fluent Bit deployment

The Fluent Bit deployment:

helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit \
  --namespace monitoring \
  --values fluent-bit-values.yaml

The Helm chart deploys the Fluent Bit as a DaemonSet.

The Fluent Bit configuration

The Fluent Bit configuration:

# fluent-bit-values.yaml
config:
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/pods/*/*/*.log
        Parser docker
        Tag kube.*
        Refresh_Interval 5

  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_Tag_Prefix kube.var.log.pods.
        Kube_URL https://kubernetes.default.svc:443
        Merge_Log On
        Keep_Log On

  outputs: |
    [OUTPUT]
        Name loki
        Match kube.*
        Host loki.monitoring.svc
        Port 3100
        Labels job=fluent-bit,cluster=production

The configuration defines the pipeline.

The parsers

The parsers:

[PARSER]
    Name json
    Format json
    Time_Key @timestamp
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
    Name docker
    Format json
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ

The parsers transform the logs to JSON.

The filters

The filters enrich the logs:

[FILTER]
    Name kubernetes
    Match kube.*
    Kube_Tag_Prefix kube.var.log.pods.
    Kube_URL https://kubernetes.default.svc:443
    Merge_Log On
    Keep_Log On

The filter adds the Kubernetes metadata (pod, namespace, container, etc.).

The outputs

The outputs:

[OUTPUT]
    Name loki
    Match kube.*
    Host loki.monitoring.svc
    Port 3100

[OUTPUT]
    Name es
    Match kube.*
    Host elasticsearch.monitoring.svc
    Port 9200

The outputs ship the logs to the storage.

The Kubernetes metadata

The Kubernetes metadata:

{
  "kubernetes": {
    "namespace_name": "default",
    "pod_name": "nginx-1-abc",
    "container_name": "nginx",
    "docker_id": "abc123",
    "labels": {
      "app": "nginx",
      "team": "team-a"
    }
  }
}

The metadata is added by the kubernetes filter.

The Fluent Bit operations

The Fluent Bit operations:

flowchart LR
    A[Tail] --> B[Parse]
    B --> C[Filter]
    C --> D[Output]
    D --> E[Loki]
    D --> F[ES]
    D --> G[S3]

The operations are the pipeline.

The Elasticsearch integration

The Elasticsearch integration:

[OUTPUT]
    Name es
    Match kube.*
    Host elasticsearch.monitoring.svc
    Port 9200
    Index kube-logs
    Type _doc

The Elasticsearch output is the alternative to Loki.

The production patterns

The production patterns:

# Fluent Bit metrics
curl http://fluent-bit:2020/api/v1/metrics

# Fluent Bit health
curl http://fluent-bit:2020/api/v1/health

The metrics and health are the observability.

The cross-course references

• The Loki course covers the Loki integration.
• The Elasticsearch course covers the Elasticsearch integration.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Choose Fluent Bit or Promtail. Per cluster.
• Configure the parsers. JSON for structured logs.
• Configure the filters. Kubernetes metadata.
• Configure the outputs. Loki, Elasticsearch, S3.
• Use the Fluent Bit metrics. The observability.
• Document the configuration. The parsers, the filters, the outputs.

The Fluent Bit is the alternative log collector. Operating it well is the deployment, the parsers, the filters, and the outputs, with the Kubernetes metadata enrichment.