Restart policy, backoffLimit, and podFailurePolicy — Job resilience

A Job’s resilience is determined by five fields: the restart policy (the container-level contract), the backoffLimit (the retry budget), the activeDeadlineSeconds (wall-clock budget), the exponential backoff (time between retries), and the podFailurePolicy (selective Pod failure handling). Production Jobs configure all five; a Job without them can run forever or fail unpredictably.

restartPolicy

The container restart policy is set on the Pod template:

spec:
  template:
    spec:
      restartPolicy: OnFailure

OnFailure is the default and recommended setting. The Pod’s lifecycle is owned by the controller; the container restart within the Pod is automatic.

flowchart TB
    A[Job Pod starts] --> B{Container exits?}
    B -->|0 success| C[Pod Succeeded]
    B -->|non-zero failure| D{restartPolicy}
    D -->|OnFailure| E[Restart container in same Pod]
    D -->|Never| F["Pod Failed<br/>new Pod created"]
    E --> G{retries > backoffLimit?}
    F --> G
    G -->|yes| H[Job marked Failed]
    G -->|no| I["Wait backoff<br/>retry"]

backoffLimit

spec:
  backoffLimit: 4

The number of Pod retries before the Job is marked Failed. With OnFailure, the count includes container restarts within the same Pod; with Never, each Pod is a separate retry.

The retry count is the total across all Pods (or restart cycles for OnFailure). A Job that has failed 4 times is Failed; the controller does not create more Pods.

activeDeadlineSeconds

spec:
  activeDeadlineSeconds: 3600

A wall-clock budget. After 3600 seconds from Job creation, the Job is marked Failed and all Pods are terminated. Unlike backoffLimit, this is independent of retry count.

The use case: a long-running batch that should not exceed an hour. After an hour, the Job is Failed regardless of how many retries have happened.

flowchart TB
    A[Job created] --> B{Elapsed > activeDeadlineSeconds?}
    B -->|yes| F1["Job Failed<br/>terminate Pods"]
    B -->|no| C[Continue]
    C --> D{Failed Pods > backoffLimit?}
    D -->|yes| F1
    D -->|no| A

Exponential backoff between retries

Since Kubernetes 1.21, Jobs support exponential backoff between Pod retries:

spec:
  backoffLimit: 10
  podRetryPolicy: Exponential    # or Never

With podRetryPolicy: Exponential, the time between Pod retries doubles on each failure. The base delay is 10 seconds; the cap is 6 minutes. A Job that fails 10 times waits approximately:

10s + 20s + 40s + 80s + 160s + 320s + ... ≈ up to 6 minutes per backoff

This is critical for batch workloads that depend on external systems (a database, a third-party API) that may be temporarily unavailable. The exponential backoff avoids hammering the dependency while still retrying.

podFailurePolicy

spec:
  backoffLimit: 4
  podFailurePolicy:
    rules:
    - action: Ignore
      onExitCodes:
        container: 42
    - action: FailJob
      onExitCodes:
        container: 137

podFailurePolicy (alpha in 1.25, beta in 1.28) lets the operator decide what to do with specific failure modes:

• Ignore: the failure does not count against backoffLimit. Useful for “expected” failures (e.g., exit code 42 means “no work to do”).
• FailJob: the failure marks the Job Failed immediately, regardless of retry count.
• Count: the default — the failure counts toward backoffLimit.

The use case: a batch where some Pods return exit code 42 (“no work”) and others return 1 (“genuine error”). With podFailurePolicy, the no-work Pods do not count against the retry budget.

flowchart TB
    A[Pod fails] --> B{Exit code}
    B -->|0| C[Succeeded]
    B -->|42| D["podFailurePolicy<br/>Ignore"]
    B -->|137| E["podFailurePolicy<br/>FailJob"]
    B -->|other| F[backoffLimit counts]
    D --> G[Job continues]
    E --> H[Job Failed]
    F --> I{retries > backoffLimit?}
    I -->|yes| H
    I -->|no| J[Retry with backoff]

TTL after finished

spec:
  ttlSecondsAfterFinished: 3600

The Job (and its Pods) is deleted 3600 seconds after completion. The TTL controller sweeps completed Jobs automatically. Without TTL, completed Jobs accumulate forever.

Putting it together

A production Job manifest:

apiVersion: batch/v1
kind: Job
metadata:
  name: process-batch-20260816
spec:
  completions: 8
  parallelism: 4
  completionMode: Indexed
  backoffLimit: 5
  activeDeadlineSeconds: 7200
  ttlSecondsAfterFinished: 86400
  podRetryPolicy: Exponential
  template:
    spec:
      restartPolicy: OnFailure
      serviceAccountName: batch-runner
      containers:
      - name: worker
        image: worker:v1.2.3
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 1
            memory: 2Gi

Every field is set deliberately:

• completions and parallelism define the work shape.
• completionMode: Indexed partitions the work.
• backoffLimit: 5 bounds retries.
• activeDeadlineSeconds: 7200 bounds total duration.
• ttlSecondsAfterFinished: 86400 cleans up after 24 hours.
• podRetryPolicy: Exponential spaces retries apart.
• restartPolicy: OnFailure is the container’s contract.

Quiz

Production discipline

• Always set backoffLimit. A Job without it retries 6 times by default — too few for transient failures, but a Job without retry is dangerous.
• Always set activeDeadlineSeconds. A hanging Pod without a deadline runs until the cluster is exhausted.
• Use podRetryPolicy: Exponential for external dependencies. Network or third-party API calls are flaky; hammering them on a tight retry is rude.
• Use podFailurePolicy for expected failures. A Pod that exits with “no work” code should not count against the retry budget.
• Set ttlSecondsAfterFinished. Completed Jobs accumulate; the TTL controller sweeps them. Without TTL, manual cleanup is the only way.

Job resilience is operator discipline. The controller provides the tools; the operator chooses the values. The Job that runs every Monday at 02:00 to process the weekend’s batch should not run forever or fill etcd with completed Job objects.