Event rate limiting — the throttle on the event flood

Kubernetes event rate limiting is the throttle on the event flood. The eventratelimit admission controller is the controller. The configuration is via a ConfigMap. This lesson walks the admission controller, the configuration, the per-namespace limits, and the production patterns.

The eventratelimit admission controller

The eventratelimit admission controller:

flowchart LR
    A[Event] --> B[eventratelimit admission]
    B --> C{Within limit?}
    C -->|yes| D[Accept]
    C -->|no| E[Reject]

The admission controller is the gate.

The enablement

The enablement:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
apiServer:
  extraArgs:
    enable-admission-plugins: EventRateLimit
  extraVolumes:
  - name: event-ratelimit-config
    hostPath: /etc/kubernetes/event-ratelimit-config.yaml
    mountPath: /etc/kubernetes/event-ratelimit-config.yaml
    readOnly: true
    pathType: File

The admission controller is enabled on the API server.

The configuration

The configuration:

apiVersion: eventratelimit.admission.k8s.io/v1alpha1
kind: Configuration
limits:
- type: Server
  qps: 100
  burst: 200
- type: User
  qps: 50
  burst: 100

The configuration is the limits.

The limit types

The limit types:

The types are the granularity.

The per-namespace limits

The per-namespace limits:

apiVersion: eventratelimit.admission.k8s.io/v1alpha1
kind: Configuration
limits:
- type: Namespace
  qps: 50
  burst: 100
  cacheSize: 1000

The Namespace limits are per namespace.

The per-object limits

The per-object limits:

apiVersion: eventratelimit.admission.k8s.io/v1alpha1
kind: Configuration
limits:
- type: Object
  qps: 10
  burst: 20

The Object limits are per object.

The conflicting events

The conflicting events:

EventCount: 5 events with the same reason and message
The events are merged into a single event with count=5

The conflicting events are merged; the event count increases.

involvedObject:
  kind: Pod
  name: nginx-1-abc
reason: FailedScheduling
count: 5

The rate limiting in the recorder

The rate limiting in the recorder:

import (
    "k8s.io/client-go/util/flowcontrol"
)

rateLimiter := flowcontrol.NewTokenBucketRateLimiter(10, 20)

The recorder’s rate limiter is the local throttle.

The event flood

The event flood:

A pod fails to schedule 100 times per second.
The API server generates 100 events per second.
The event API is flooded.
The etcd is overwhelmed.

The event flood is the failure mode.

The production patterns

The production patterns:

apiVersion: eventratelimit.admission.k8s.io/v1alpha1
kind: Configuration
limits:
- type: Server
  qps: 100
  burst: 200
- type: User
  qps: 50
  burst: 100
- type: Namespace
  qps: 50
  burst: 100
  cacheSize: 1000
- type: Object
  qps: 10
  burst: 20

The production patterns cover all types.

The cross-course references

• The Admission Controller course covers the admission.
• The API Server course covers the configuration.
• The Observability course covers the events.

Quiz

Production discipline

• Enable the eventratelimit admission controller. The safety net.
• Configure the limits. The Server, User, Namespace, Object.
• Use the per-namespace limits. The multi-tenancy.
• Use the per-object limits. The default.
• Monitor the event rate. The Prometheus metrics.
• Document the configuration. The admission, the limits.

The event rate limiting is the cluster’s event throttle. Operating it well is the admission controller, with the limits, and the per-namespace and per-object limits.