Level-triggered vs edge-triggered reconciliation

Kubernetes’s controllers are level-triggered: they act on the current observed state, not on a specific event. This is a deliberate design choice that makes controllers resilient to event ordering, drops, and restarts. This lesson explains why.

Level-triggered vs edge-triggered

A level-triggered controller acts on the current state:

“There are 3 Pods; I want 5. Create 2 more.”

An edge-triggered controller acts on each event:

“Event: ‘Pod created’. Add 1 to my counter of creations.” “Event: ‘Pod deleted’. Add 1 to my counter of deletions.” “Act when counter == 5 minus expected.”

flowchart TB
    subgraph Level["Level-triggered"]
        L[Observe current state<br/>3 Pods, want 5] --> LD[Diff: need 2 more]
        LD --> LA[Act: create 2]
        LA --> L
    end

    subgraph Edge["Edge-triggered"]
        E[Watch event stream] --> E1[Event: ADDED Pod X]
        E1 --> E2[Event: DELETED Pod Y]
        E2 --> E3[Event: ADDED Pod Z]
        E3 --> EA[Act: maintain ratio]
    end

Why level-triggered wins for Kubernetes

Level-triggered is correct under conditions where edge-triggered breaks.

Condition 1: dropped events

sequenceDiagram
    autonumber
    participant API as API server
    participant L as Level-triggered
    participant E as Edge-triggered

    Note over API: Pod A created
    API-->>L: event ADDED A
    API-->>E: event ADDED A
    Note over API: Pod B created (event lost!)
    API-->>L: (event lost — L doesn't care)
    API-->>E: (event lost — E thinks only A exists)
    L->>API: GET current state (sees A and B)
    L->>L: ok, no action needed
    E->>E: thinks only A exists, may act incorrectly

Level-triggered: a dropped event is harmless. The next reconcile re-reads the current state and finds A and B. Correct.

Edge-triggered: a dropped event causes the controller to lose track of state. Without idempotency, the controller’s state drifts from reality. Recovery requires a full re-list or restart.

Condition 2: out-of-order events

sequenceDiagram
    autonumber
    participant API as API server
    participant L as Level-triggered
    participant E as Edge-triggered

    API-->>L: event MODIFIED A=10
    API-->>L: event MODIFIED A=20
    API-->>L: event MODIFIED A=15 (out of order!)
    API-->>E: event MODIFIED A=10
    API-->>E: event MODIFIED A=20
    API-->>E: event MODIFIED A=15
    Note over L: L re-reads: A=15 (current state)
    Note over E: E thinks A went 10 -> 20 -> 15
    Note over E: E acts on its view of history

Level-triggered: the current state is A=15 regardless of event order. The controller’s decision is based on the current state, not the history.

Edge-triggered: out-of-order events produce incorrect state in the controller’s history. The controller may over-correct (e.g., revert A to 10 thinking it was the “right” state).

Condition 3: controller restart

sequenceDiagram
    autonumber
    participant API as API server
    participant L as Level-triggered restarted
    participant E as Edge-triggered restarted

    Note over L: L restarts and lists current state
    L->>API: GET current state (sees A=15)
    Note over L: L acts on current state - correct
    Note over E: E restarts and loses event history
    E->>E: has no record of past events
    Note over E: E cannot reason about A=15 without history

Level-triggered: a restart is harmless. The reflector re-lists on startup; the indexer is rebuilt; reconciliation proceeds from the current state.

Edge-triggered: a restart loses the event history. The controller must either replay the history (expensive) or accept that its view is incomplete (potentially wrong).

Condition 4: HA controllers

Two replicas of the same controller, both watching, both acting.

sequenceDiagram
    autonumber
    participant API as API server
    participant L1 as Level-triggered replica 1
    participant L2 as Level-triggered replica 2
    participant E1 as Edge-triggered replica 1
    participant E2 as Edge-triggered replica 2

    API-->>L1: event ADDED Pod A
    API-->>L2: event ADDED Pod A
    API-->>L1: event ADDED Pod B
    API-->>L2: event ADDED Pod B
    Note over L1,L2: Both re-read and both see A and B
    Note over L1,L2: Idempotent actions - one wins, the other is no-op
    Note over E1,E2: Both saw different events if they split
    Note over E1,E2: Both act based on partial history which may conflict

Level-triggered: both replicas converge to the same action because they observe the same current state. Optimistic concurrency in the API server serialises writes; one wins, the other gets 409 and re-reads.

Edge-triggered: both replicas have partial, different event histories. Their actions may conflict; the result depends on event ordering, which is racy.

The cost of level-triggered

Level-triggered is not free. Costs:

• Latency — every reconcile re-reads the world. With a watch cache, this is fast, but a controller that wants < 100 ms reconcile latency may be slow if its reads are not local.
• Throughput — a controller that wants to scale to 10,000 objects per reconcile must list them all. The informer cache helps, but a misbehaving controller can still hammer the API server.
• Periodic re-list — the reflector re-lists every 10 minutes by default. This is a safety net for missed events.

These costs are acceptable for Kubernetes’s reconciliation model. The alternative — edge-triggered with full event replay — is more complex and less robust.

When edge-triggered is appropriate

A few places in Kubernetes use edge-triggered reasoning:

The scheduler

The scheduler reacts to Pods being created (or becoming unschedulable after binding). It does not maintain a long-term view of “all unscheduled Pods”; it picks them up as they arrive.

This works because:

• A Pod’s spec.nodeName being unset is a stable observation
• The scheduler re-lists on every reconcile cycle anyway
• Edge-triggered scheduling is faster (no full re-list required)

Garbage collector

The GC reacts to deletion events. It needs to know “owner is gone, delete children” — an edge-triggered event. The GC also has a periodic full walk as a safety net.

Event recorders

Some controllers emit Events in response to state changes (“Deployment updated”). This is conceptually edge-triggered: the event is “something happened”.

Designing your own controller

If you write a custom Operator or controller, the design should be level-triggered. Specifically:

// Good: level-triggered
func Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    obj := &MyResource{}
    if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    desiredState := obj.Spec
    observedState := readWorldState(obj)

    if !reflect.DeepEqual(desiredState, observedState) {
        return r.act(ctx, obj, observedState, desiredState)
    }
    return ctrl.Result{}, nil  // converged
}

// Bad: edge-triggered
func Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    event := getCurrentEvent()  // fragile!
    switch event.Type {
    case "Created":
        return handleCreated(event)
    case "Deleted":
        return handleDeleted(event)
    }
    return ctrl.Result{}, nil
}

The level-triggered pattern:

1. Re-read the current state on every reconcile
2. Diff against the spec
3. Act to close the gap
4. Return — the workqueue will re-trigger when the state changes

This is robust under all the failure modes described above.

The control theory view

In control theory:

• Level-triggered = “act when state crosses threshold”
• Edge-triggered = “act on transitions”

Level-triggered is correct under noisy signals; edge-triggered requires clean transitions. Kubernetes’s watch stream is noisy (drops, ordering, restarts). Level-triggered is the appropriate choice.

The corollary: a controller that appears to depend on specific events is fragile. The right model is “act on current state”.

Cross-course references

• The Linux course part VI-Linux-Processes covers process state transitions; level-triggered reconciliation is the cluster’s analogue.
• The Observability course part CIX-Observability-InvestigationWorkflows covers investigation methodology that benefits from understanding the reconciliation model.
• The Linux course part XXIV-Linux-Time covers chrony — watch timestamps and resource versions depend on clock accuracy.
• The Docker course part XXX-Docker-Lifecycle covers the container lifecycle that drives much of the reconcile flow.

Quiz

Production discipline

• Always design controllers as level-triggered: read the current state, diff, act. Never depend on specific events.
• Test reconcilers for correctness under event drops, ordering changes, and restarts.
• Use optimistic concurrency (resourceVersion) to serialise concurrent reconciliations.
• Make every action idempotent: re-running the reconciler with the same input produces the same output.
• Treat watch events as hints, not commands. The reconciler re-reads state, not event payloads.