OOMKilled: The container exceeded its memory limit and was killed by the kernel

OOMKilled means the Linux kernel killed a process in your container because the container's cgroup hit its memory limit. It appears with exit code 137. Raising the limit is one fix; it is not always the right one.

Applies to: All Kubernetes versions, Linux nodes

What it means

Kubernetes does not kill containers for using too much memory — the kernel does. A container's resources.limits.memory becomes a cgroup memory limit, and when processes inside that cgroup try to allocate beyond it, the kernel's OOM killer terminates a process, usually the largest one. The kubelet then reports OOMKilled as the termination reason with exit code 137, which is 128 plus signal 9 (SIGKILL). Two things follow from this that surprise people. First, the kill is abrupt: the process gets no signal it can handle and no chance to shut down cleanly. Second, it is the container's own limit that matters, not the node's free memory — a container can be OOM-killed on a node with plenty of memory available.

Most common causes

• The memory limit is genuinely too low for the workload's steady-state usage.
• A real memory leak in the application, which shows up as a slow climb to the limit followed by a kill, repeating on a regular cycle.
• A JVM, Node.js, or other runtime with its own heap settings that do not know about the cgroup limit, so it sizes its heap for the whole node.
• A burst of work — a large request, a big file upload, a batch job — that briefly needs far more memory than normal operation.
• Memory used by page cache or tmpfs mounts counting against the container's limit.
• The limit was set by copying another workload's manifest rather than by measuring this one.

How to diagnose it

1. Confirm the reason and exit code: kubectl describe pod POD, then read Last State: Terminated with Reason: OOMKilled and Exit Code: 137.
2. Compare actual usage against the limit over time: kubectl top pod POD gives a snapshot, but a metrics dashboard showing the trend is far more informative — a sawtooth climb indicates a leak, a flat line near the limit indicates an undersized limit.
3. Check whether the runtime is container-aware. For the JVM, confirm -XX:+UseContainerSupport is in effect (it is the default on modern JVMs) and check -XX:MaxRAMPercentage. For Node.js, check --max-old-space-size.
4. Look at whether the kill correlates with specific traffic or a scheduled job rather than happening at random.

How to fix it

1. If usage is steadily near the limit, raise resources.limits.memory to a measured value with headroom, and raise the request to match so scheduling reflects reality.
2. If the pattern is a sawtooth, fix the leak. Raising the limit only lengthens the interval between kills.
3. For JVM workloads, set the heap as a percentage of the container limit rather than a fixed size, so the two cannot drift apart.
4. For genuine bursts, consider whether the work belongs in a separate Job with its own larger limit rather than inflating the limit of a long-running service.
5. Set requests and limits deliberately rather than equally by habit — but be aware that a container with a memory limit above its request has a lower QoS class and is more likely to be evicted under node pressure.

Notes

Exit code 137 does not always mean OOM. It means the process received SIGKILL, which also happens when a container fails to stop within its termination grace period. Check the reason field, not just the code.

Related

• Exit code 137 — Process killed by SIGKILL
• CrashLoopBackOff — Repeated restarts with increasing delay
• Evicted — Pod removed because the node ran short of resources

Sources

• Kubernetes documentation — Resource Management for Pods and Containers
• Kubernetes documentation — Assign Memory Resources to Containers and Pods
• Kubernetes documentation — Quality of Service for Pods