completionMode — Indexed and NonIndexed, work-queue vs partitioned batches

completionMode is the field that decides how a parallel Job distributes work across its Pods. The two modes answer different questions: “do the Pods share work” (NonIndexed) or “is each Pod responsible for its own partition” (Indexed). The choice is fundamental; using the wrong mode produces either duplicate work or a Job that never completes.

NonIndexed — the original behaviour

apiVersion: batch/v1
kind: Job
metadata:
  name: process-queue
spec:
  completions: 100
  parallelism: 10
  completionMode: NonIndexed   # default if omitted
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: worker:v1
        env:
        - name: WORK_QUEUE_URL
          value: https://queue.internal/jobs

The 10 Pods share a queue. Each Pod polls the queue, takes a work item, processes it, returns for more. The Job completes when 100 work items have been processed (counted by 100 successful Pod exits, not by 100 unique items).

The pattern’s subtlety: the Job controller counts Pod completions, not unique work items. If two Pods happen to process the same work item (a queue race), the Job can complete before all items are processed. Or if one item is re-processed, the count is wrong.

flowchart LR
    Q["Work Queue<br/>100 items"] --> P1["Pod 1<br/>polls"]
    Q --> P2["Pod 2<br/>polls"]
    Q --> P3["Pod 3<br/>polls"]
    P1 -->|item 7| C[Process]
    P2 -->|item 42| C
    P3 -->|item 13| C
    C --> Q2[Done]

Indexed — partitioned work

apiVersion: batch/v1
kind: Job
metadata:
  name: process-shards
spec:
  completions: 8
  parallelism: 4
  completionMode: Indexed
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: worker:v1
        env:
        - name: INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']

The Job creates 8 Pods in total; up to 4 run at once. Each Pod gets a stable index in the range [0, 8). The Pod’s metadata.annotations['batch.kubernetes.io/job-completion-index'] field carries the index. The worker reads it, computes its partition (e.g., INDEX % total_shards), and processes only that partition’s data.

flowchart LR
    J["Job<br/>completions: 8<br/>parallelism: 4"] --> P0["Pod 0<br/>annotation: 0"]
    J --> P1["Pod 1<br/>annotation: 1"]
    J --> P2["Pod 2<br/>annotation: 2"]
    J --> P3["Pod 3<br/>annotation: 3"]
    P0 -->|completes| P4["Pod 4<br/>annotation: 4"]
    P1 -->|completes| P5["Pod 5<br/>annotation: 5"]
    P2 -->|completes| P6["Pod 6<br/>annotation: 6"]
    P3 -->|completes| P7["Pod 7<br/>annotation: 7"]

Each Pod is responsible for a specific shard. No two Pods process the same shard; no shard is processed twice. The Job completes when all 8 Pods succeed.

The Indexed mode replaced the older “worker reads its ordinal from JOB_COMPLETION_INDEX env var” pattern. With Indexed mode, the controller guarantees the index assignment; the worker just reads it.

How to read the index

env:
- name: INDEX
  valueFrom:
    fieldRef:
      fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']

The Downward API exposes the annotation as an env var. The container reads INDEX and computes its partition:

INDEX=${INDEX:-0}
SHARD_TOTAL=8
PARTITION=$((INDEX % SHARD_TOTAL))

For more complex partitioning (range-based instead of modular), the application reads INDEX and SHARD_TOTAL directly and slices accordingly.

When to use which mode

NonIndexed is correct when:

• The work items come from a queue (Kafka, SQS, RabbitMQ).
• The work items can be processed by any worker.
• Duplicate processing is harmless or guarded by an application-level lock.

Indexed is correct when:

• The work is a fixed partition (e.g., “process partition 3 of a Kafka topic”).
• Each Pod must process a specific, deterministic shard.
• The application cannot rely on a queue.
• The batch must be exactly-once at shard granularity.

Indexed Jobs and selector mutation

Indexed Jobs require spec.selector and spec.template. metadata.labels to match. The controller adds a label batch.kubernetes.io/job-completion-index to each Pod:

flowchart TB
    A["Indexed Job<br/>spec.template.metadata.labels"] --> B["Pod labels<br/>app + ..."]
    B --> C["Controller adds<br/>batch.kubernetes.io/job-completion-index"]
    C --> D[selector matches Pods]

Since 1.27, Jobs (Indexed and NonIndexed) gain a controllable selector via the --job-mutate-strategy or by explicit spec.selector declaration. This enables pause / resume / update without losing track of existing Pods.

Failure modes

Wrong mode for the workload

A worker-queue workload running on an Indexed Job has 8 Pods polling the same queue. The Indexed semantics guarantee the index is assigned but do not enforce that the workers use it correctly. The result is 8 Pods each polling the queue and processing duplicate work; the Job completes counting Pod exits, not unique items.

The mitigation: verify the worker reads the index and uses it to scope the work. A test with a single Pod and completions: 1 should produce the same outcome as the full parallel run.

Wrong parallelism

spec:
  completions: 10
  parallelism: 20

parallelism > completions is allowed but wasteful; the controller never runs more than completions Pods.

spec:
  completions: 100
  parallelism: 5

A small parallelism on a large batch is slow but correct. The right value depends on the workload’s resource footprint and the cluster’s capacity.

Scaled too aggressively

spec:
  completions: 1000
  parallelism: 200

A parallelism of 200 on a cluster with 16 free Pods per node will create 200 Pods that may not all schedule. Some Pods sit Pending. The Job’s progress is gated by the cluster capacity, not the parallelism count.

flowchart TB
    A["Job with parallelism=200"] --> B{Cluster capacity<br/>available?}
    B -->|yes| C[All Pods scheduled]
    B -->|no| D[Some Pods Pending]
    D --> E[Job progress blocked]

The right parallelism is the smaller of completions and the cluster’s available Pod slots.

Quiz

Production discipline

• Use Indexed mode for partitioned batches. A deterministic “Pod N processes shard N” pattern is easier to reason about than a shared queue.
• Read the index from the annotation. Older patterns use JOB_COMPLETION_INDEX env; the modern pattern is the Downward API against the annotation.
• Validate the parallelism against cluster capacity. A Job’s parallelism is not magic; it must fit in the cluster.
• Test with completions: 1 first. A single-Pod run validates the worker’s behaviour; the parallel run validates the partitioning.
• Pause and resume. Indexed Jobs support suspend: true. Use this to gate on external conditions without losing Pod state.

completionMode is the difference between a Job that does the right work in parallel and a Job that does the wrong work twice. Operators who understand both modes have batch systems that scale.