cgroup driver mismatch: The kubelet and the container runtime are managing cgroups differently

The kubelet and the runtime must use the same cgroup driver — systemd or cgroupfs. A mismatch produces unstable nodes, wrong resource accounting, and containers that fail to start in ways that look unrelated.

Applies to: All Kubernetes versions, Linux nodes

What it means

Linux cgroups can be managed either directly through the filesystem interface or through systemd, and the two approaches create incompatible hierarchies. Kubernetes requires the kubelet and the container runtime to agree, because both write to the cgroup tree for every container. When they disagree, containers may be created under paths the other component does not expect, resource limits may not apply, eviction may act on wrong numbers, and container creation may fail outright. The reason this is worth calling out separately is that the symptoms are diverse and none of them names cgroups — you get instability rather than an error that points at the cause.

Most common causes

• A node built with the runtime defaulting to cgroupfs while the kubelet was configured for systemd, or the reverse.
• A runtime upgrade that changed the default driver.
• A host operating system upgrade moving from cgroup v1 to v2, where systemd is effectively required.
• A manually built node where the kubelet configuration was copied from a different setup.
• Configuration management that sets one side but not the other.

How to diagnose it

1. Check the kubelet's setting: cgroupDriver in its configuration file, or the flag in its unit.
2. Check containerd's: the SystemdCgroup setting under its runc runtime options.
3. Check the node's cgroup version: stat -fc %T /sys/fs/cgroup reports cgroup2fs for v2.
4. Look for cgroup path errors in the kubelet and runtime logs.
5. Compare a working node against a failing one, if the cluster has both.

How to fix it

1. Set both sides to systemd on any node using systemd as its init system. This is the recommended configuration and is required in practice on cgroup v2.
2. Restart both the kubelet and the runtime after changing the driver, and drain the node first — existing containers are created under the old hierarchy.
3. Bake the setting into node images rather than applying it after the fact, so new nodes are correct from the start.
4. Verify the driver as part of node provisioning checks, since the failure mode is diffuse and slow to attribute.

Notes

Changing the cgroup driver on a node with running containers leaves them under the previous hierarchy, which produces a mixed state that is worse than either configuration alone. Draining the node before the change is not optional in practice.

Related

• OCI runtime create failed — The low-level runtime refused to create the container
• failed to create shim task — The runtime shim could not start the container process

Sources

• Kubernetes documentation — Container Runtimes
• containerd CRI plugin configuration
• Kubernetes documentation — Kubelet Configuration reference