Event retention — the long-term storage for events

Kubernetes events have a 1-hour default TTL. The event-exporter ships the events to long-term storage. The retention policies are configurable. This lesson walks the event retention, the long-term storage, the retention policies, and the production patterns.

The 1-hour TTL

The 1-hour TTL:

flowchart LR
    A[Event created] --> B[etcd: 1 hour]
    B --> C[Event evicted]
    D[event-exporter] --> E[Long-term storage]

The TTL is the default in etcd.

The event retention policy

The event retention policy:

- Hot storage: 30 days (Loki, Elasticsearch)
- Cold storage: 1 year (S3, GCS)
- Archive: 5 years (Glacier, Coldline)

The retention is configurable.

The event-exporter destinations

The destinations:

receivers:
  - name: loki
    loki:
      url: http://loki:3100/loki/api/v1/push
  - name: s3
    s3:
      bucket: my-events-bucket
      region: us-east-1
  - name: elasticsearch
    elasticsearch:
      url: http://elasticsearch:9200

The destinations are the storage.

The hot storage with Loki

The hot storage with Loki:

receivers:
  - name: loki
    loki:
      url: http://loki:3100/loki/api/v1/push
      labels:
        job: event-exporter
        cluster: production

The Loki is the hot storage.

The cold storage with S3

The cold storage with S3:

receivers:
  - name: s3
    s3:
      bucket: my-events-bucket
      region: us-east-1
      path: events/

The S3 is the cold storage.

The Grafana integration

The Grafana integration:

{service="event-exporter"} |= "FailedScheduling"

The Grafana queries the events via Loki.

The retention policies

The retention policies:

# Loki retention
limits_config:
  retention_period: 720h  # 30 days

# S3 lifecycle policy
LifecycleConfiguration:
  Rules:
  - Id: events-retention
    Status: Enabled
    Expiration:
      Days: 365  # 1 year

The retention policies are per storage.

The event-based alerting

The event-based alerting:

- alert: KubeEventWarning
  expr: |
    sum(rate(kube_events_total{type="Warning"}[5m])) > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Warning events detected"

The events are the input for the alerts.

The event-based metrics

The event-based metrics:

# Warning events per namespace
sum(rate(kube_events_total{type="Warning"}[5m])) by (namespace)

# Reasons
sum(rate(kube_events_total{type="Warning"}[5m])) by (reason)

The metrics are the input for the dashboards.

The production patterns

The production patterns:

flowchart LR
    A[Events 0-30 days] --> B[Hot storage: Loki]
    C[Events 30-365 days] --> D[Cold storage: S3]
    E[Events 1-5 years] --> F[Archive: Glacier]

The pattern is the production flow.

The cost optimization

The cost optimization:

Hot storage: 100 GB at $0.10/GB/month = $10/month
Cold storage: 1 TB at $0.023/GB/month = $23/month
Archive: 5 TB at $0.004/GB/month = $20/month
Total: $53/month for 5 years of events

The cost optimization is the retention strategy.

The cross-course references

• The Loki course covers the log storage.
• The S3 course covers the object storage.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Configure hot and cold storage. The tiered retention.
• Use the event-exporter. The long-term storage.
• Configure the retention policies. Per storage.
• Use the event-based alerts. The Prometheus alerts.
• Use the event-based metrics. The dashboards.
• Document the retention. The events, the policies.

The event retention is the cluster’s event history. Operating it well is the hot + cold storage, with the event-exporter, and the retention policies.