Watch semantics — list-watch, resourceVersion, informer

The watch feed is the API server’s mechanism for delivering state-change events to clients. Every controller, every scheduler, every operator, and every interested Pod relies on watch to react to changes without polling. This lesson walks the protocol, the resourceVersion cursor, and the informer’s role in client-go.

List-watch in one sentence

List-watch is the API server’s protocol for delivering state changes: a client opens a long-poll connection on the watch endpoint, the server sends events as state changes, and the client processes them and (if needed) re-establishes a new watch from the last acknowledged event.

sequenceDiagram
    autonumber
    participant Client
    participant AS as API server
    participant E as etcd
    Client->>AS: LIST pods (resourceVersion=N)
    AS->>E: range query
    E-->>AS: result + currentRevision=N+10
    AS-->>Client: 200 OK (objects + resourceVersion=N+10)
    Client->>AS: WATCH pods?resourceVersion=N+10
    AS->>AS: long-poll connection
    E-->>AS: pod X created (revision=N+11)
    AS-->>Client: event ADDED pod X (revision=N+11)
    E-->>AS: pod Y modified (revision=N+12)
    AS-->>Client: event MODIFIED pod Y (revision=N+12)
    Note over Client: connection lost
    Client->>AS: WATCH pods?resourceVersion=N+12
    AS->>AS: replay events since N+12

The LIST establishes the resourceVersion; the WATCH streams events from that revision forward. The client-supplied resourceVersion is the cursor.

The watch endpoint

# A short-lived API token for a ServiceAccount allowed to watch Pods in prod:
TOKEN=$(kubectl create token default -n prod)

# Watch Pods in the prod namespace, starting from resourceVersion 41289312
curl -k -H "Authorization: Bearer $TOKEN" \
  "https://api.example/api/v1/namespaces/prod/pods?watch=1&resourceVersion=41289312"

The connection stays open; events are written to the response stream as they occur.

{
  "type": "ADDED",
  "object": {"kind": "Pod", "metadata": {"name": "...", "namespace": "prod"}}
}
{
  "type": "MODIFIED",
  "object": {"kind": "Pod", "metadata": {"name": "...", "namespace": "prod", "resourceVersion": "..."}}
}
{
  "type": "DELETED",
  "object": {"kind": "Pod", "metadata": {"name": "...", "namespace": "prod"}}
}

Each event has a type (ADDED, MODIFIED, DELETED, BOOKMARK) and the object. The BOOKMARK type is a periodic marker that says “everything before this has been delivered”.

The resourceVersion model

Every Kubernetes object has a metadata.resourceVersion field that is the etcd revision in which the object last changed.

kubectl get pod web -n prod -o jsonpath='{.metadata.resourceVersion}'
# 41289312

The API server uses this as the cursor for watches. A client that supplies resourceVersion=N receives events for revisions > N.

flowchart LR
    N[resourceVersion: N] -->|"watch events >= N+1"| W[Watch stream]
    W -->|each event| ACK[Client acknowledges with N+1, N+2, ...]
    ACK -->|continue| W

If the client’s last acknowledged revision is N+5 and the watch breaks, the next watch resumes at N+5.

The watch reliability challenge

Three failure modes a watch must tolerate:

• Connection drop. The HTTP connection is lost; the client must reconnect.
• API server restart. Long-poll connections are closed when the API server restarts; clients must resume from a stable cursor.
• Compacted revision. Old revisions are compacted (per the API server’s retention); the watch cursor references a revision that no longer exists.

The protocol:

1. Reconnect with the last cursor. On reconnect, supply the last acknowledged resourceVersion.
2. Bookmarks as fallback. Periodically, BOOKMARK events confirm the cursor is still valid.
3. Re-list from scratch when 410 Gone. If the API server reports 410 Gone (resourceVersion too old), the client re-lists without a resourceVersion and starts a new watch.

$ kubectl get pods -A --watch --resource-version=41289312

(continues from the named resourceVersion)

Informer caches in client-go

The Kubernetes client library (client-go) implements informers — caches that maintain an in-memory view of cluster state. The informer’s pattern:

flowchart LR
    L[reflector: list + watch] -->|events| F[FIFO queue]
    F -->|processed| D[Indexer]
    D -->|read| C[Controller]

A reflector does the list-watch:

• Starts with a LIST to populate initial state.
• Opens a WATCH from the LIST’s resourceVersion.
• Pushes events to a queue.
• On connection break, re-lists and resumes watching.

The indexer is the cache (in-memory threadSafeMap) indexed by namespace/name. Controllers read from the indexer, not from the API server directly.

Resource version in watch headers

The watch response includes response headers like:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

The streaming JSON body carries events; the client parses events as they arrive.

WATCH with include-bookmarks

Modern watches request bookmarks to make reconnects reliable:

# A short-lived API token for a ServiceAccount allowed to watch Pods in prod:
TOKEN=$(kubectl create token default -n prod)

curl -k -H "Authorization: Bearer $TOKEN" \
  "https://api.example/api/v1/namespaces/prod/pods?watch=1&\
resourceVersion=41289312&\
allowWatchBookmarks=true"

The client receives BOOKMARK events every ~5 seconds (configurable); the bookmarks confirm the cursor is still alive.

Watch latency

Watches do not typically constrain latency; the API server’s main thread is the bottleneck for event generation, but watch events are typically delivered in sub-100 ms.

Reliability features

Three features that keep watch reliable in production:

1. resourceVersion with allow-bookmarks: prevent the 410 Gone on long-lived connections.
2. Resync periods: every --resync-period (default 10 minutes), the informer re-lists to recover from silent divergence. The resync is idempotent at the API server level; the list is a no-cost operation.
3. WatchListClientSide: for high-throughput watchers, the client requests the initial LIST and the subsequent WATCH as a single request.

flowchart LR
    I[Informer] -->|every 10 min| RL[Re-list]
    RL -->|rebuild cache| C[Cache]
    I -->|continuous| W[Watch events]
    W -->|update| C

Common watch failure modes

Quiz

Production discipline

• Operators use bookmarks. A controller that does not handle bookmarks is at risk of 410 Gone on long connections.
• Resync at the application level. The --resync-period of 10 minutes is the safety net for silent divergence.
• Watch latency is the controller’s reaction time. A watch that delivers events in 5 seconds is a controller that takes 5 seconds to react.
• Compact cadence matters. A 5-minute compaction with no bookmark support is a recipe for 410 Gone; align compaction with controller compatibility.
• Don’t trust a long-running cache uncritically. Add periodic resets to ensure cache freshness.

Watch is the API server’s connective tissue; understanding it is what makes controllers reliable.