permission denied on a mounted volume: The container's user cannot read or write the mounted volume

A volume that mounts successfully but cannot be written usually means the container runs as a user that does not own the volume's files. fsGroup is the mechanism for fixing this.

Applies to: All Kubernetes versions, Linux nodes

What it means

Mounting a volume and being able to use it are separate things. A freshly provisioned volume is typically owned by root, while a security-conscious container runs as a non-root user, so the mount succeeds and the first write fails with permission denied. Kubernetes provides securityContext.fsGroup for this: the volume's group ownership is set to that GID and the setgid bit is applied, so the container's process can write regardless of the file owner. Two important limits: fsGroup applies to volume types that support ownership management, so it does nothing for NFS, where permissions are the server's business; and on large volumes the recursive ownership change can make pod startup very slow, which fsGroupChangePolicy: OnRootMismatch exists to mitigate.

Most common causes

• The container runs as a non-root user via runAsUser while the volume is owned by root.
• No fsGroup is set, so no ownership adjustment is made at mount time.
• The volume is an NFS or SMB share where permissions are enforced by the server and fsGroup has no effect.
• A subPath mount that inherits ownership from a directory created by a different user.
• A read-only mount being written to — readOnly: true on the mount, or a readOnlyRootFilesystem security context.
• ConfigMap and Secret volumes, which are mounted read-only by design and cannot be written at all.

How to diagnose it

1. Check the ownership and mode inside the container: kubectl exec POD -- ls -ln /mount/path.
2. Check the effective user: kubectl exec POD -- id.
3. Read the pod's security context: kubectl get pod POD -o jsonpath='{.spec.securityContext}'.
4. Confirm whether the mount is read-only: kubectl exec POD -- mount | grep /mount/path.
5. For network filesystems, check what the server exports and how it maps users — root squashing is a frequent surprise.

How to fix it

1. Set securityContext.fsGroup to the group the container runs as. This is the intended mechanism and handles most cases.
2. Set fsGroupChangePolicy: OnRootMismatch to avoid a recursive ownership pass on every mount of a large volume.
3. For NFS, fix permissions and squashing on the server. Kubernetes cannot override them.
4. Use an init container to set ownership only when the volume type does not support fsGroup and the server cannot be changed.
5. Do not attempt to write to ConfigMap or Secret volumes — copy their contents to a writable location if the application insists on writing.

Notes

On a volume with a very large number of files, the recursive ownership change implied by fsGroup can add minutes to pod startup, and it happens on every mount. This is a common and confusing cause of slow-starting stateful pods.

Related

• FailedMount — A volume could not be mounted into the pod
• RunContainerError — The container was created but failed to start

Sources

• Kubernetes documentation — Configure a Security Context for a Pod or Container
• Kubernetes documentation — Volumes
• Kubernetes documentation — Persistent Volumes