BackoffLimitExceeded: A Job's pods failed more times than its backoffLimit allows

The Job retried a failing pod up to backoffLimit times — six by default — and then gave up. The Job is marked Failed and will not retry again without intervention.

Applies to: All Kubernetes versions

What it means

A Job creates pods until the required number complete successfully. Each pod failure counts against backoffLimit, with an exponentially increasing delay between retries, and once the limit is reached the Job's Failed condition is set with reason BackoffLimitExceeded. The Job stops creating pods at that point. What matters for diagnosis is that the failure count is per Job, not per pod, and the failed pods are retained precisely so their logs can be read — which is the only place the actual error lives, since the Job's own status says only that the retries ran out.

Most common causes

• The job's work genuinely fails every time — bad input, a missing dependency, a code defect.
• A transient failure that happened to recur more than backoffLimit times.
• The container being OOM-killed on each attempt.
• A missing ConfigMap, Secret, or volume, so the pod never gets far enough to do anything.
• A backoffLimit set too low for a task that legitimately needs retries against a flaky dependency.
• The image being wrong or unpullable, which burns the retry budget without ever running the work.

How to diagnose it

1. Read the Job's status: kubectl describe job JOB shows the condition and the failure count.
2. List the failed pods, which are kept on purpose: kubectl get pods -l job-name=JOB.
3. Read their logs: kubectl logs POD. This is where the real error is.
4. Check the exit code and reason: kubectl describe pod POD. A 137 with OOMKilled changes the diagnosis entirely.
5. Check whether all attempts failed identically, which indicates a deterministic problem rather than flakiness.

How to fix it

1. Fix the underlying failure, then delete and recreate the Job. A failed Job does not restart on its own.
2. Raise backoffLimit only when the failure is genuinely transient. Raising it for a deterministic failure just wastes more time.
3. Set activeDeadlineSeconds as well, so a Job that hangs rather than failing is also bounded.
4. Give the container enough memory if it is being OOM-killed.
5. Use a pod failure policy, where available, to distinguish retryable failures from ones that should stop the Job immediately.

Notes

Failed pods are retained deliberately so the logs survive. Combined with a CronJob running frequently, that means failed pods accumulate until they become their own problem — ttlSecondsAfterFinished and the CronJob history limits are what keep this bounded.

Related

• DeadlineExceeded — The Job exceeded its active deadline
• OOMKilled — Killed by the kernel for exceeding the memory limit

Sources

• Kubernetes documentation — Jobs
• Kubernetes documentation — TTL Controller for Finished Resources
• Kubernetes documentation — Debug Pods