Jobs — running Pods to completion

A Job is the controller for batch work — workloads that must run to completion, not run forever. A Job creates one or more Pods and tracks their success; when enough Pods have succeeded, the Job is Complete. The cluster retains the Job’s Pods for inspection (until the TTL sweeps them) but the controller does not restart them indefinitely. This is the right controller for one-shot tasks, batch processing, and parallel work queues.

What a Job is

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-db
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: migrator
        image: migrator:v1
        command: ["./migrate", "--once"]

The Job controller creates a Pod from the template. The Pod runs, exits, and the controller records the exit. If the exit is 0 (success), the Pod is counted as a successful completion. If the Pod fails or is evicted, the controller creates a new Pod, up to backoffLimit times.

stateDiagram-v2
    [*] --> Running: Job created
    Running --> Running: Pod running
    Running --> Complete: completions reached
    Running --> Failed: backoffLimit exceeded
    Running --> Suspended: suspend=true
    Suspended --> Running: suspend=false

Compared to Deployments

A Deployment’s Pods restart forever (RestartPolicy Always). A Job’s Pods run once (RestartPolicy OnFailure or Never). The difference is fundamental: a Deployment is a steady state; a Job is a transition.

flowchart LR
    D[Deployment] -->|Always| R1[Restart]
    R1 --> R1
    J[Job] -->|OnFailure| R2[Run to completion]
    R2 --> D1[Done]

completion and parallelism

spec:
  completions: 5
  parallelism: 2

The Job creates 5 Pods to completion, with up to 2 running at any time. The Pods run as a parallel pool; when one completes, the controller starts the next.

The use case: processing 100 work items with 10 parallel Pods. The application inside the Pod is responsible for “claiming” a work item; the Job controller only manages Pod count.

flowchart LR
    A["Job: 5 completions, 2 parallelism"] --> P1["Pod 1<br/>running"]
    A --> P2["Pod 2<br/>running"]
    P1 -->|completes| P3["Pod 3<br/>running"]
    P2 -->|completes| P4["Pod 4<br/>running"]
    P3 -->|completes| P5["Pod 5<br/>running"]
    P4 -->|completes| A1[Complete]
    P5 -->|completes| A1

restartPolicy

A Job’s Pod template must have restartPolicy: OnFailure or Never. Always is rejected by the API server because it conflicts with the Job’s completion semantics.

• OnFailure: the container is restarted within the Pod on failure (same Pod, new container instance). This is the default for Jobs and the recommended setting for most batch workloads.
• Never: the Pod is restarted as a new Pod on failure. Used when the application tracks state outside the container (e.g., the container must run from a clean state every retry).

flowchart TB
    A[Job Pod starts] --> B{Container exits?}
    B -->|0 success| C[Job counts completion]
    B -->|non-zero failure| D{restartPolicy}
    D -->|OnFailure| E[Restart container in same Pod]
    D -->|Never| F[Create new Pod]
    E --> G{retries > backoffLimit?}
    F --> G
    G -->|yes| H[Job marked Failed]
    G -->|no| A

activeDeadlineSeconds and backoffLimit

spec:
  activeDeadlineSeconds: 600
  backoffLimit: 4

activeDeadlineSeconds is a wall-clock budget. After 600 seconds, the Job is marked Failed regardless of completion status. The Pods are terminated.

backoffLimit is a retry budget. After 4 failed Pods, the Job is marked Failed. No new Pods are created.

flowchart TB
    A[Job starts] --> B{Elapsed > activeDeadlineSeconds?}
    B -->|yes| F1["Job Failed<br/>terminate Pods"]
    B -->|no| C{Failed Pod count > backoffLimit?}
    C -->|yes| F1
    C -->|no| D[Continue]

These are the budgets that prevent a misconfigured batch from running forever. Production Jobs always set both.

Suspend and resume

Since Kubernetes 1.27, Jobs support spec.suspend: true to pause execution without deleting the Job:

spec:
  suspend: true

The controller does not create new Pods. Existing Pods run to completion; the Job is in Suspended state. Setting suspend: false resumes.

This is used in production to gate batch execution on external conditions (maintenance window, downstream availability) without losing the Job’s metadata.

Workloads that are correct for Jobs

The test: is the workload a transition or a steady state? If a transition (run once, exit, move on), Job. If a steady state (run forever, restart, serve traffic), Deployment.

• Database migrations. A migration is a one-shot task; Job.
• One-off data processing. “Process this S3 prefix, exit.” Job.
• Batch image processing. “Resize 1000 images.” Job with parallelism.
• Work-queue processing. “Pull work items from a queue and process them.” Job with parallelism, with the application coordinating the queue.
• Test runners. A CI Job runs tests and exits; the result is reported back. Job.

Workloads that are wrong for Jobs

• Long-running services. A web server is a steady state, not a transition. The Pod must not exit; it serves traffic. Deployment.
• Stateful clusters. Databases, message brokers — Job with restartPolicy: OnFailure would restart the container, but a database cluster is not a single container. StatefulSet or Operator.
• Schedules that repeat. A Job runs once; a nightly-or-hourly task is CronJob.
• Workflows with dependencies. Step A must complete before step B starts; A’s success decides whether B runs. Argo Workflows, Tekton, or a custom controller. Plain Jobs do not express dependencies.

Inspecting a Job

kubectl get jobs -A
# NAMESPACE   NAME            COMPLETIONS   DURATION   AGE
# data        migrate-db      1/1           12s        1h
# batch       image-resize    47/100        3m         10m
# ad-hoc       bad-batch       0/5           30m        30m

kubectl describe job migrate-db -n data
# Name:           migrate-db
# Namespace:      data
# Selector:       controller-uid=7c8f2d8e-...
# Pods Statuses:  0 Active / 1 Succeeded / 0 Failed
# ...

$ kubectl get pods -l job-name=migrate-db -n data -o wide

NAME                  READY   STATUS      RESTARTS   AGE
migrate-db-abcde      0/1     Completed   0          30m

STATUS: Completed is the indicator that the Pod ran to success. The Pod stays around for inspection until the TTL sweeper deletes it.

Quiz

Production discipline

• Always set activeDeadlineSeconds. A Job without a deadline runs forever if the application hangs.
• Always set backoffLimit. A Job without a retry limit retries forever.
• Use restartPolicy: OnFailure by default. Never is for stateful retries.
• Track history with TTL. Set spec.ttlSecondsAfterFinished: 3600 (1 hour) to clean up completed Jobs.
• Don’t use Jobs for long-running work. The Pod exits successfully and the Job is Complete; nothing is running. Deployment or StatefulSet.

Jobs are the workhorse of batch processing in Kubernetes. Operators who understand the controller use Jobs for one-shot work and the right controller for everything else.