Job troubleshooting — failed runs, TTL cleanup, and debugging

A Job that does not run is harder to diagnose than a Deployment that does not run, because the failure modes are batch-specific: the Pod never starts, the Job fails repeatedly, the Job succeeds but produces wrong output, or the Job is stuck at backoffLimit. This lesson walks each failure mode and the diagnostic discipline for production Jobs.

Failure mode 1: Job never starts

The Job is created but no Pod ever reaches Running:

kubectl get job migrate-db -n data
# NAME          COMPLETIONS   DURATION   AGE
# migrate-db    0/1           5m         5m

kubectl get pods -l job-name=migrate-db -n data
# NAME                  READY   STATUS    RESTARTS   AGE
# migrate-db-abcde      0/1     Pending   0          5m

The Pod is Pending. Common causes:

flowchart TB
    A[Pod Pending] --> B{Resource quota?}
    A --> C{Node selector<br/>matches?}
    A --> D{Image pull<br/>failing?}
    A --> E{PVC bound?}
    A --> F{Priority preemption<br/>denied?}

Diagnose with kubectl describe pod:

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  4m     default-scheduler  0/N nodes are available: insufficient cpu.

The fix: increase quota, fix nodeSelector, fix the image tag, or wait for the PVC to bind.

Failure mode 2: Pod fails immediately

The Pod reaches Running but the container exits with a non-zero code within seconds:

kubectl logs -l job-name=migrate-db -n data --tail=50
# panic: dial tcp 10.244.1.5:5432: connect: connection refused

The fix: the Job’s configuration is wrong. RBAC missing, wrong database URL, missing ConfigMap. Verify each one with the appropriate kubectl get / kubectl auth can-i.

Failure mode 3: Job fails repeatedly

The Pod fails, is restarted, fails again. The Job reaches backoffLimit and is marked Failed.

kubectl describe job migrate-db -n data
# ...
# Pods Statuses:    0 Active / 0 Succeeded / 5 Failed
# Conditions:
#   Type              Status
#   Failed            True
#   Reason            BackoffLimitExceeded
# Message:           Job has reached the specified backoff limit

Common causes:

flowchart TB
    A[BackoffLimit exceeded] --> B{Database<br/>not migrated?}
    A --> C{External API<br/>returning 500?}
    A --> D{Network<br/>policy denies?}
    A --> E{Disk full<br/>on target node?}

Diagnose by reading the failed Pod’s logs and the Job’s events:

kubectl logs -l job-name=migrate-db -n data --previous --tail=100
kubectl get events -n data --field-selector involvedObject.name=migrate-db

The fix: usually requires changing the Job’s configuration, not just retrying. Increase backoffLimit, fix the underlying issue, or use podFailurePolicy to ignore specific exit codes.

Failure mode 4: Job succeeds but output is wrong

The Pod exits 0; the Job is Complete. Downstream observes wrong data. This is the hardest failure mode: the Job’s status is healthy but the work is wrong.

flowchart TB
    A[Job Succeeded] --> B{Downstream<br/>verifies work?}
    A --> C{Application<br/>idempotent?}
    A --> D{Side effects<br/>observable?}
    A --> E{Silent<br/>failure mode}

Diagnose by:

1. Reading the Pod’s logs (kubectl logs).
2. Inspecting the data the Pod wrote (database, S3, etc.).
3. Re-running the Job manually with debug logging.

The fix: the Job’s exit code is not a sufficient signal. Production Jobs need an end-to-end verification step (a checksum, a row count, a downstream notification).

TTL controller and cleanup

The TTL controller watches for completed Jobs (Succeeded or Failed) and deletes them after ttlSecondsAfterFinished:

spec:
  ttlSecondsAfterFinished: 86400   # 24 hours

Without TTL, completed Jobs accumulate. A daily Job running for a year leaves 365 completed Jobs in etcd. Without TTL:

# Manual cleanup
kubectl delete job -n data -l app.kubernetes.io/component=batch \
  --field-selector=status.conditions[0].type=Complete

# Or with kubectl wait for completion + delete
kubectl wait --for=condition=Complete job/migrate-db -n data --timeout=300s
kubectl delete job migrate-db -n data

The TTL controller is the right answer for production Jobs. The manual cleanup is the fallback.

Inspecting Job status

kubectl describe job migrate-db -n data | tail -30

Key fields:

$ kubectl get job migrate-db -n data -o yaml | grep -A 2 'status:'

status:
active: 0
completionTime: 2026-08-16T02:01:34Z
conditions:
- lastProbeTime: 2026-08-16T02:01:34Z
  lastTransitionTime: 2026-08-16T02:01:34Z
  status: "True"
  type: Complete
succeeded: 1

CronJob-specific failure modes

A CronJob has additional failure modes:

flowchart TB
    A[CronJob not firing] --> B{Schedule syntax<br/>valid?}
    A --> C{Controller<br/>running?}
    A --> D{suspend=true?}
    A --> E{timeZone<br/>correct?}
    A --> F{previous Job<br/>still running?}

For a CronJob, the LAST SCHEDULE field on the kubectl get cronjob output is the operator’s first signal:

kubectl get cronjob db-backup -n data
# NAME        SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
# db-backup   0 2 * * *     False     0        30d            30d
#                                              ^ expected 02:00 today

If LAST SCHEDULE is older than the schedule interval, a run is missed.

Quiz

Production discipline

• Set activeDeadlineSeconds and backoffLimit. A Job without budgets can run forever or retry forever.
• Verify work, not exit codes. A 0 exit code is not a guarantee of correctness; production Jobs need a verification step.
• Use the TTL controller. Manual cleanup is error-prone; TTL is automatic.
• Alert on Job failure. A Job that fails for a week before the operator notices is a silent failure. Alert on kube_job_status_failed > 0.
• Alert on CronJob misses. time() - last_schedule > 2 * interval is the alert query for missed runs.
• Document the run’s recovery path. A Job that produces data needs a documented “re-run” or “recover” procedure.

Jobs are the workhorse of batch processing in Kubernetes. The failure modes are well-defined; the diagnostic discipline is kubectl describe + kubectl logs. Operators who debug Jobs efficiently have batch systems that recover quickly.