Event recorder — emitting events from controllers

The client-go event recorder is the API for emitting events. The event broadcaster is the central hub. The recorder sink is the destination. This lesson walks the recorder, the broadcaster, the sink, and the production patterns.

The event recorder

The event recorder:

import (
    "k8s.io/client-go/tools/record"
    "k8s.io/client-go/kubernetes"
)

func newRecorder(clientset kubernetes.Interface) record.EventRecorder {
    eventBroadcaster := record.NewBroadcaster()
    eventBroadcaster.StartStructuredLogging(5)
    eventBroadcaster.StartRecordingToSink(&typedcorev1.EventSinkImpl{
        Interface: clientset.CoreV1().Events(""),
    })
    return eventBroadcaster.NewRecorder(scheme.Scheme, corev1.EventSource{
        Component: "my-controller",
    })
}

The recorder is the API for emitting events.

The event broadcaster

The event broadcaster:

flowchart LR
    A[Recorder] --> B[Broadcaster]
    B --> C[Sink 1: API server]
    B --> D[Sink 2: Logging]
    B --> E[Sink 3: Webhook]

The broadcaster is the central hub.

The recorder methods

The recorder methods:

// Normal event
recorder.Event(object, EventTypeNormal, "Scheduled", "Pod is scheduled")

// Warning event
recorder.Event(object, EventTypeWarning, "Failed", "Failed to schedule")

// With annotations
recorder.AnnotatedEventf(object, map[string]string{
    "key": "value",
}, EventTypeNormal, "Reason", "Message %s", arg)

The recorder methods are the API.

The controller-runtime integration

The controller-runtime integration:

import (
    "sigs.k8s.io/controller-runtime/pkg/manager"
    "sigs.k8s.io/controller-runtime/pkg/recorder"
)

func main() {
    mgr, _ := manager.New(cfg, manager.Options{})
    rec := mgr.GetEventRecorderFor("my-controller")
    
    // Use the recorder in the reconciler
    rec.Event(pod, corev1.EventTypeNormal, "Scheduled", "Pod is scheduled")
}

The controller-runtime provides the recorder.

The event sink

The event sink:

type EventSinkImpl struct {
    Interface EventInterface
}

func (e *EventSinkImpl) Create(event *v1.Event) (*v1.Event, error) {
    return e.Interface.Events(event.Namespace).Create(context.TODO(), event, metav1.CreateOptions{})
}

The sink is the destination.

The structured logging

The structured logging:

import (
    "k8s.io/klog/v2"
)

eventBroadcaster.StartStructuredLogging(5)

The structured logging is the alternative sink.

The rate limit

The rate limit:

import (
    "k8s.io/client-go/util/flowcontrol"
)

eventBroadcaster = record.NewBroadcaster(
    record.WithContext(ctx),
    record.WithQueueDepth(1000),
    record.WithSleepDuration(time.Second),
    record.WithUpstreamLimit(10),  // 10 events per second
)

The rate limit is the throttle.

The custom recorder

The custom recorder:

type RecordingRule struct {
    Recorder record.EventRecorder
}

func (r *RecordingRule) RecordEvent(object runtime.Object, eventType, reason, message string) {
    r.Recorder.Event(object, eventType, reason, message)
}

The custom recorder is the abstraction.

The event in the reconciler

The event in the reconciler:

func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    pod := &corev1.Pod{}
    if err := r.Get(ctx, req.NamespacedName, pod); err != nil {
        return ctrl.Result{}, err
    }
    
    r.Recorder.Event(pod, corev1.EventTypeNormal, "Reconciled",
        fmt.Sprintf("Reconciled pod %s", pod.Name))
    
    return ctrl.Result{}, nil
}

The event is part of the reconciliation.

The production patterns

The production patterns:

flowchart LR
    A[Reconciler] --> B[Recorder]
    B --> C[Event Normal]
    B --> D[Event Warning]
    C --> E[API server]
    D --> E
    E --> F[event-exporter]
    F --> G[Loki]
    F --> H[S3]

The pattern is the production flow.

The cross-course references

• The Controller course covers the reconciler.
• The Observability course covers the events.
• The kubebuilder course covers the controller-runtime.

Quiz

Production discipline

• Use the event recorder. The API for emitting events.
• Configure the event broadcaster. The sink, the rate limit.
• Use the recorder in the reconciler. The event recording.
• Use the structured logging. The alternative sink.
• Use the event sink to the API server. The default.
• Document the recorder. The integration.

The event recorder is the API for emitting events. Operating it well is the recorder, the broadcaster, the sink, and the production patterns.