Job patterns — work queues, parallel shards, and indexed batches

Most production batch workloads fit one of three patterns: a work queue where Pods share a queue and each takes work, partitioned shards where each Pod owns a deterministic slice, or fan-out where the controller creates one Job per work item. This lesson walks each pattern, its configuration, and the workloads it fits.

Pattern 1: work-queue (NonIndexed)

A central queue (Kafka, SQS, RabbitMQ, Postgres) holds the work items. Each Pod polls the queue, takes one item, processes it, and returns.

apiVersion: batch/v1
kind: Job
metadata:
  name: process-orders
spec:
  completions: 1000
  parallelism: 20
  completionMode: NonIndexed
  backoffLimit: 5
  activeDeadlineSeconds: 7200
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: order-worker:v1
        env:
        - name: QUEUE_URL
          value: https://queue.internal/orders
        - name: WORKER_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name

flowchart LR
    Q["Order Queue<br/>1000 items"] --> P1[Pod 1]
    Q --> P2[Pod 2]
    Q --> P3[Pod 20]
    P1 -->|process| DB["(Database)"]
    P2 -->|process| DB
    P3 -->|process| DB

The pattern’s strength: it scales with the queue depth. The controller creates more Pods (up to parallelism) when work is available; the queue does the coordination.

The pattern’s weakness: it requires a queue. Without a queue (or with a queue that does not support atomic “take”), two Pods may process the same item. The Job completion count is not the same as unique items processed.

Use this pattern when:

• A queue already exists (most modern backends have one).
• The queue can guarantee exactly-once delivery (Kafka with transactional writes, SQS with FIFO).
• The work items are independent (one item does not depend on another’s output).

Pattern 2: partitioned shards (Indexed)

The work is divided into a fixed number of partitions; each Pod is responsible for one partition.

apiVersion: batch/v1
kind: Job
metadata:
  name: process-table
spec:
  completions: 8
  parallelism: 4
  completionMode: Indexed
  backoffLimit: 5
  activeDeadlineSeconds: 7200
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: table-worker:v1
        env:
        - name: INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']
        - name: SHARD_TOTAL
          value: "8"
        command:
        - /worker
        - --partition=$(INDEX)
        - --of=$(SHARD_TOTAL)

flowchart LR
    J["Job<br/>8 completions<br/>4 parallelism"] --> P0["Pod 0<br/>partition 0"]
    J --> P1["Pod 1<br/>partition 1"]
    J --> P2["Pod 2<br/>partition 2"]
    J --> P3["Pod 3<br/>partition 3"]
    P0 -->|completes| P4["Pod 4<br/>partition 4"]
    P1 -->|completes| P5["Pod 5<br/>partition 5"]
    P2 -->|completes| P6["Pod 6<br/>partition 6"]
    P3 -->|completes| P7["Pod 7<br/>partition 7"]

The pattern’s strength: no external coordination. Each Pod is responsible for a specific slice; the application’s internal logic ensures no duplicate work.

The pattern’s weakness: a Pod that fails to read its index will process the wrong partition. The application must verify the index is set correctly.

Use this pattern when:

• The work is a fixed partition of a known dataset (table rows, file paths, hash buckets).
• No queue exists or a queue is undesirable.
• The partition can be computed from a stable index.

Pattern 3: fan-out (one Job per item)

The controller creates one Job per work item, sequentially or in parallel. Each Job is independent.

# A controller (Argo Workflows, Tekton, or a custom operator)
# iterates over the work items and creates a Job per item.

# Each generated Job
apiVersion: batch/v1
kind: Job
metadata:
  name: process-customer-{{ITEM_ID}}
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: customer-worker:v1
        env:
        - name: CUSTOMER_ID
          value: "{{ITEM_ID}}"

flowchart LR
    A["Orchestrator<br/>Argo / Tekton / Operator"] --> J1["Job 1<br/>customer 42"]
    A --> J2["Job 2<br/>customer 43"]
    A --> J3["Job 3<br/>customer 44"]
    J1 -->|done| J4["Job 4<br/>customer 45"]
    J2 -->|done| J5["Job 5<br/>customer 46"]

The pattern’s strength: each work item has its own Job with its own retry budget, history, and observability. A failure on one item does not affect others.

The pattern’s weakness: a controller is required to manage the fan-out. Plain Jobs do not express dependencies between Jobs. Argo Workflows or Tekton provide this.

Use this pattern when:

• Work items have different scopes (some large, some small).
• Each item needs its own retry budget.
• Observability per item is valuable.
• The orchestrator exists (Argo, Tekton).

Choosing the pattern

flowchart TB
    A[What is the work?] --> B{Is there a queue?}
    B -->|yes| C["Work-queue<br/>NonIndexed"]
    B -->|no| D{Can it be partitioned?}
    D -->|yes, deterministic| E["Partitioned shards<br/>Indexed"]
    D -->|no, dependent items| F["Use Argo Workflows<br/>fan-out with dependencies"]

Real production example

A nightly batch that processes customer orders:

flowchart LR
    A["CronJob<br/>02:00 daily"] --> B["Job: extract-orders"]
    B -->|writes| Q[Order Queue]
    Q --> C["Job: process-orders<br/>20 parallel Pods"]
    C -->|writes| D["(Database)"]
    C -->|emits| E["Job: send-receipts<br/>fan-out per customer"]
    E -->|emails| M[Mail Provider]

Three patterns in one workflow:

1. Extract — a CronJob triggers a Job that pulls orders from the database and writes them to a queue.
2. Process — a NonIndexed Job with 20 parallel Pods reads from the queue.
3. Receipts — a fan-out Job per customer with Argo Workflows.

The CronJob’s concurrencyPolicy: Forbid ensures only one extract runs at a time; the processing Job has its own backoffLimit and activeDeadlineSeconds; the receipts are fan-out for per-customer observability.

Quiz

Production discipline

• Pick the pattern deliberately. A wrong pattern wastes resources or duplicates work.
• Test with completions: 1 first. A single-Pod run validates the worker’s behaviour; the parallel run validates the partitioning.
• Validate the queue. A work-queue Job assumes the queue is healthy and exactly-once; verify before going to production.
• Document the orchestrator. A fan-out Job pattern requires an orchestrator; document which (Argo, Tekton, custom).
• Set retry budgets per pattern. Work-queue Jobs retry per Pod; partitioned shards retry per partition; fan-out retries per item.

The Job pattern is the design choice that determines correctness, scalability, and observability. Operators who choose the right pattern have batch systems that scale.