DiskPressure: The node is low on disk space or inodes

DiskPressure means the node crossed an eviction threshold for filesystem space or inodes. The kubelet garbage-collects images and containers, and if that is not enough it starts evicting pods.

Applies to: All Kubernetes versions

What it means

The kubelet watches two filesystems — the one holding images and container writable layers, and the one holding the node's root — and tracks both free bytes and free inodes on each. Crossing a threshold sets DiskPressure, taints the node, and triggers reclamation: unused images are deleted first, then dead containers, then pods are evicted. The inode dimension is the one that catches people, because a filesystem can be at 30 percent of its byte capacity and still be completely out of inodes if something is creating enormous numbers of small files. The symptoms are identical and the usual diagnostic, checking free space, shows nothing wrong.

Most common causes

• Accumulated container images that are no longer used but have not been garbage-collected.
• Container logs growing without rotation, particularly from a verbose workload.
• emptyDir volumes with no size limit consuming node disk.
• Containers writing large files to their own writable layer instead of to a volume.
• Inode exhaustion from many small files, with free space still available.
• A node whose disk was sized for a smaller workload than it now runs.

How to diagnose it

1. Confirm the condition and which filesystem it refers to: kubectl describe node NODE.
2. On the node, check both dimensions: df -h and df -i. Skipping the second is the most common way to miss this.
3. Find the consumers: image cache size via crictl images, log volume under /var/log/pods, and the kubelet's pod directories.
4. Check for a single pod dominating ephemeral usage: kubectl describe node NODE reports ephemeral-storage requests, and du on the node finds the rest.
5. Check whether image garbage collection is running, and what thresholds it uses.

How to fix it

1. Send application logs to stdout so the node's rotation applies, rather than writing files inside containers.
2. Set sizeLimit on emptyDir volumes and ephemeral-storage requests and limits on containers, so one workload cannot fill the node.
3. Tune image garbage collection thresholds so unused images are reclaimed earlier.
4. Increase node disk size where the workload genuinely needs it.
5. For inode exhaustion, reduce the number of files. On ext2, ext3 and ext4 the inode table is fixed when the filesystem is created, so growing the filesystem adds no inodes. XFS allocates inodes dynamically from free space, so growing an XFS filesystem does help — and XFS is the default root filesystem on several common node images, so check which you have before assuming.

Notes

A node under disk pressure refuses new pods and evicts existing ones, so a single workload writing logs to a file inside its container can destabilise every other pod on the node. Ephemeral storage limits are the containment mechanism and are much less commonly set than memory limits.

Related

• no space left on device — A write failed because a filesystem is full
• Evicted — Pod removed because the node ran short of resources

Sources

• Kubernetes documentation — Node-pressure Eviction
• Kubernetes documentation — Logging Architecture
• Kubernetes documentation — Node Status