Watch, informers, and events — how controllers observe the cluster

Controllers don’t poll the API server for every reconcile. They maintain a local cache fed by watch events. This lesson covers the watch protocol, the informer architecture, and the related concept of Kubernetes Events (the resource kind, not the OS events).

The watch protocol

A watch is a long-running HTTP request to the API server:

GET /api/v1/namespaces/prod/pods?watch=true&resourceVersion=128034

The server responds with the current state, then streams events as they happen. The client maintains a local cache that mirrors the server’s view.

sequenceDiagram
    autonumber
    participant API as API server
    participant W as Watch client (informer)

    W->>API: GET /pods?watch=true&resourceVersion=128034
    API-->>W: initial state (list)
    loop events
        API-->>W: ADDED/MODIFIED/DELETED event
    end
    Note over W,API: connection drops
    W->>API: re-list from last known resourceVersion
    API-->>W: missing events since last seen

The watch protocol handles failures:

• Connection drop — client transparently re-lists from the last known resourceVersion; API server replays missing events.
• 410 Gone — the requested resourceVersion is too old (older than the watch cache retention); client must do a full re-list.
• Slow consumer — if the client cannot keep up, the API server closes the watch; the client reconnects.

The informer architecture

The informer (in client-go) is the canonical implementation of the watch + cache + indexing pattern. It has three parts:

flowchart TB
    API[API server] --> R[Reflector]
    R -->|list + watch events| D[Delta FIFO queue]
    D -->|pop events| P[Populator]
    P --> I[Indexer<br/>thread-safe map]
    I --> E[Event handlers]
    E --> Q[Workqueue]
    Q --> Rec[Reconciler]

Reflector

The reflector watches the API server. It:

• Lists resources on startup (gets the initial state)
• Opens a watch with the latest resourceVersion
• Forwards events to the Delta FIFO queue
• Re-lists on 410 Gone
• Re-lists periodically (default every 10 minutes) to catch any missed events

The reflector is resilient to API server hiccups. A brief network outage does not cause a controller to lose state; on reconnect, the reflector re-lists and catches up.

Delta FIFO queue

A thread-safe FIFO that holds pending events. Each event is a delta — the change between the previous and current state. The informer processes events in order.

Indexer

A thread-safe map keyed by <namespace>/<name>. The reconciler reads from the indexer, not from the watch stream. This decouples reconcile latency from watch event rate.

The indexer supports indexers — secondary indexes by label, annotation, or custom field. A controller can list Pods with labels=app=web efficiently via the indexer.

Event handlers

Custom functions that react to events. The most common pattern:

informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(obj interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(obj)
        workqueue.Add(key)
    },
    UpdateFunc: func(old, new interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(new)
        workqueue.Add(key)
    },
    DeleteFunc: func(obj interface{}) {
        key, _ := cache.DeletionHandlingMetaNamespaceKeyFunc(obj)
        workqueue.Add(key)
    },
})

The handler adds the object’s key to a workqueue. The reconciler dequeues, fetches the latest from the indexer, and reconciles.

Why informers are designed this way

The informer pattern solves several problems:

• API server load — without informers, every reconcile reads from the API server. With informers, reads hit the local cache. The API server is not flooded.
• Decoupling — the controller can take seconds to reconcile without affecting watch delivery.
• Resilience — a network blip does not cause the controller to lose state. The reflector re-lists.
• Atomic snapshots — the indexer provides a consistent view at the time of read.

Kubernetes Events (the resource)

Kubernetes Events are a resource kind (events.k8s.io/v1 or core/v1) that records things that happen in the cluster:

apiVersion: v1
kind: Event
metadata:
  namespace: prod
  name: web-abc.170c8e...
involvedObject:
  kind: Pod
  namespace: prod
  name: web-abc
  uid: ...
reason: FailedScheduling
message: '0/5 nodes are available: 3 Insufficient memory, 2 node(s) didn't match Pod's node affinity.'
type: Warning
source:
  component: default-scheduler
firstTimestamp: 2026-08-15T12:01:01Z
lastTimestamp: 2026-08-15T12:01:01Z
count: 1

Events are emitted by:

• The scheduler (FailedScheduling, Scheduled)
• kubelet (FailedMount, Pulling, Created, Started)
• Controllers (ReplicaSet created, Deployment updated)
• Admission webhooks (with custom messages)
• The Node controller (Node Ready, Node NotReady)

Events are not durable long-term: they live in etcd but are pruned (default 1 hour retention for events.k8s.io/v1, 5 minutes for older core/v1). Production clusters ship events to a central store (Loki, Elasticsearch) for long-term retention.

kubectl get events

kubectl get events -A --sort-by='.lastTimestamp'

LAST SEEN   TYPE      REASON              OBJECT                   MESSAGE
2m          Warning   FailedScheduling    pod/web-abc              0/5 nodes are available: insufficient memory
3m          Normal    Scheduled           pod/db-xyz               Successfully assigned db-xyz to worker-04
5m          Warning   BackOff             pod/api-123              Back-off pulling image my-app:v3
10m         Normal    Pulled              pod/api-123              Successfully pulled image my-app:v3
15m         Normal    Started             pod/api-123              Started container app

kubectl describe <object> includes the events for that object:

kubectl describe pod web-abc | grep -A 20 "Events:"

The Pod events tell the reconciliation story: why it was scheduled, why an image was pulled, why a container started or failed.

Event-storm scenarios

A common operational problem is an event storm: many events of the same kind in a short time, often from a single recurring failure.

sequenceDiagram
    autonumber
    participant C as Controller (broken)
    participant API as API server

    loop every N seconds
        C->>API: PATCH (fails)
        API-->>C: 409 Conflict
        C->>API: re-list (no progress)
        C->>API: emit Event: "ReconcileFailed"
    end
    Note over API: thousands of events/minute

Production consequences:

• The events fill the API server’s storage
• The events fill the cluster’s etcd (auto-pruned but still costly)
• The events fill monitoring pipelines

Defensive patterns:

• Coalesce repeated events: only emit a new Event if the state changed since the last
• Suppress events in the controller for known-harmless retries
• Rate-limit event emission at the controller level
• Filter events in the log pipeline before shipping

Watch latency

Watch latency is the time from a state change to the informer cache being updated. Production monitoring:

# API server metrics
apiserver_watch_events_total
apiserver_watch_duration_seconds

A healthy watch latency is < 100 ms p99. Sustained high latency (> 1 second) indicates:

• API server overloaded
• etcd commit latency high
• Network between API server and etcd slow

Diagnosing broken watches

A controller that is not picking up changes may have:

• Lost the watch — the reflector’s watch was dropped and re-list is failing
• Indexer out of sync — events are arriving but the indexer is not being updated (a bug)
• Event handler broken — events are arriving but the workqueue is not being populated

Diagnostic steps:

# 1. Is the API server healthy?
kubectl get --raw /healthz

# 2. Is the watch returning events?
kubectl get pods -w --v=8 2>watch.log
# (in another terminal)
kubectl delete pod web-abc
# Confirm an event appears in watch.log

# 3. Is the controller's workqueue growing?
controller.workqueue_depth

Cross-course references

• The Linux course part VI-Linux-Processes covers the process state transitions that map onto informer events.
• The Observability course part IX-Observability-Exporters covers the metrics surface that exposes watch latency.
• The Linux course part XXIV-Linux-Time covers chrony — watch timestamps depend on clock accuracy.
• The Docker course part XXX-Docker-Lifecycle covers the container lifecycle that drives much of the event stream.

Quiz

Production discipline

• Ship events to a central store with long retention; default etcd retention is too short for incident investigation.
• Defend against event storms: rate-limit event emission in controllers; coalesce repeated events.
• Monitor watch latency (apiserver_watch_duration_seconds) on the API server.
• Treat informer events as advisory: the reconciler should re-read the indexer, not trust the event payload.
• Audit metadata.managedFields to identify which controller owns each object’s state.