FailedUnmount: The kubelet could not unmount a volume from a terminating pod

FailedUnmount means a pod is being cleaned up and one of its volumes will not release. It is the usual reason a pod sits in Terminating for a long time with no other explanation.

Applies to: All Kubernetes versions

What it means

When a pod terminates, the kubelet unmounts each of its volumes before the pod's resources are released. If an unmount fails, the kubelet retries and emits FailedUnmount, and the pod cannot finish terminating. The most common underlying cause is a network filesystem whose server has become unreachable: the unmount blocks in the kernel waiting for the server, and no amount of retrying at the Kubernetes layer helps. The second most common is a process still holding a file open inside the mount, which happens when a container did not stop cleanly or when a subPath mount is involved.

Most common causes

• An NFS or SMB server that is unreachable, so the unmount blocks in the kernel.
• A process still holding an open file inside the mount.
• A container that was force-deleted at the API level but whose process is still running.
• A subPath mount that is bind-mounted elsewhere and not released in order.
• The CSI node plugin having crashed mid-operation, leaving the mount in an inconsistent state.
• A device-level error on the node after storage connectivity was lost.

How to diagnose it

1. Read the event: kubectl describe pod POD names the volume that will not unmount.
2. On the node, check what is using the mount: fuser -vm MOUNTPOINT or lsof +D MOUNTPOINT.
3. Check the kubelet log for the kernel-level error: journalctl -u kubelet | grep -i unmount.
4. For network filesystems, test the server's reachability from the node directly.
5. Check whether the CSI node plugin is running and healthy on that node.

How to fix it

1. Restore reachability to the network filesystem server. Once it answers, the pending unmount usually completes on its own.
2. Stop the process holding the mount open.
3. Restart the CSI node plugin if it is in a bad state.
4. Use a lazy unmount on the node only as a deliberate last resort, understanding that it defers rather than resolves the problem.
5. For workloads on network filesystems, configure mount options with sensible timeouts so an unreachable server produces an error rather than an indefinite block.

Notes

A hard NFS mount with default options blocks indefinitely when the server is unreachable, and that block propagates all the way up to a pod that will not terminate and a node that becomes hard to drain. Choosing mount options deliberately is worth doing before this happens.

Related

• Terminating — Pod stuck in deletion
• FailedDetachVolume — The volume could not be detached from a node

Sources

• Kubernetes documentation — Volumes
• Kubernetes documentation — Pod Lifecycle: termination of pods
• Kubernetes documentation — Debug Pods