no space left on device: A write failed because the volume, the node's disk, or its inodes are full

ENOSPC in a container means something ran out of room — the mounted volume, the node's own filesystem, or the inode table, which can be exhausted while free space remains.

Applies to: All Kubernetes versions, Linux nodes

What it means

no space left on device is the kernel's ENOSPC surfacing in an application log. In a container there are three separate things it can refer to, and they are diagnosed differently. A full PersistentVolume is the obvious one. A full node filesystem is less obvious: the container's writable layer and any emptyDir live on the node's disk, so a container writing large temporary files can fill the node without touching its volume. And inode exhaustion produces the identical message while df -h reports plenty of free space, because the limit reached is the number of files rather than the number of bytes.

Most common causes

• A PersistentVolume that is genuinely full.
• The node's filesystem full, from image layers, container logs, or ephemeral writes.
• An emptyDir volume consuming node disk with no size limit set.
• Inode exhaustion from very large numbers of small files, with free bytes still available.
• Log files written inside the container rather than to stdout, growing without rotation.
• A container writing to its own writable layer, which counts as ephemeral storage against the node.

How to diagnose it

1. Check the volume from inside the pod: kubectl exec POD -- df -h.
2. Check inodes specifically, which the byte view hides: kubectl exec POD -- df -i.
3. Check the node itself: kubectl describe node NODE for DiskPressure, and df -h on the node.
4. Find what is consuming space on the node — image cache, container logs under /var/log/pods, or emptyDir directories under the kubelet root.
5. Identify the largest directories inside the volume: kubectl exec POD -- du -xh --max-depth=1 /mount/path.

How to fix it

1. Expand the PersistentVolumeClaim if the StorageClass allows it, and let the filesystem resize complete.
2. Delete or archive data inside the volume, or add retention to whatever writes it.
3. Write application logs to stdout so the node's log rotation applies, rather than to files inside the container.
4. Set sizeLimit on emptyDir volumes and ephemeral-storage requests and limits on containers, so one pod cannot fill a node.
5. For inode exhaustion, reduce the number of files rather than the bytes. Whether expanding helps depends on the filesystem: ext2, ext3 and ext4 fix their inode table at creation time, while XFS allocates inodes dynamically and does gain capacity when grown.

Notes

Inode exhaustion is the version of this that wastes the most time, because every instinct is to check free space and free space looks fine. df -i should be the second command, not the tenth.

Related

• Evicted — Pod removed because the node ran short of resources
• DiskPressure — The node is low on disk or inodes

Sources

• Kubernetes documentation — Node-pressure Eviction
• Kubernetes documentation — Volumes: emptyDir
• Kubernetes documentation — Logging Architecture