failed to garbage collect required amount of images: The kubelet tried to reclaim disk by deleting images and could not free enough

This message means image garbage collection ran, deleted what it could, and the node is still above its disk threshold. The space is being held by something other than unused images.

Applies to: All Kubernetes versions

What it means

When the node crosses its image garbage collection high threshold, the kubelet deletes unused images, oldest first, until it drops below the low threshold. It logs this failure when it cannot get there. The useful inference is about what is holding the space: images in use by running containers cannot be deleted, so if the node is full of images that are all in use, garbage collection has nothing to work with. More often the space is not images at all — it is container logs, emptyDir volumes, or files written into container writable layers, none of which image garbage collection touches.

Most common causes

• The disk is filled by container logs rather than by images.
• emptyDir volumes or writable layers consuming the filesystem.
• All cached images are in use by running containers and cannot be removed.
• A node disk that is simply too small for the number and size of images the workload uses.
• Garbage collection thresholds set too close together, leaving no room to work.
• Something outside Kubernetes writing to the same filesystem.

How to diagnose it

1. Find what is actually using the space on the node: du -xh --max-depth=1 /var/lib and /var/log.
2. Check total image size against the disk: crictl images and the runtime's storage directory size.
3. Check log volume under /var/log/pods, which is a common answer.
4. Check the kubelet's garbage collection thresholds in its configuration.
5. Confirm whether the node also reports DiskPressure, which indicates how urgent this is.

How to fix it

1. Reduce log volume, and ensure rotation is configured with a size limit the node can sustain.
2. Set ephemeral-storage limits and emptyDir size limits so pods cannot fill the filesystem.
3. Increase node disk size if the image working set genuinely requires it.
4. Reduce image size and count — large base images multiply across every node that runs them.
5. Tune the garbage collection thresholds so reclamation starts earlier and has more headroom.

Notes

Image garbage collection only ever deletes images, so a node whose disk is full of logs will report this message repeatedly while the actual consumer goes unaddressed. Reading the message as "the kubelet cannot fix this by deleting images" points at the right investigation.

Related

• DiskPressure — The node is low on disk or inodes
• no space left on device — A write failed because a filesystem is full

Sources

• Kubernetes documentation — Node-pressure Eviction
• Kubernetes documentation — Kubelet Configuration reference
• Kubernetes documentation — Logging Architecture