DaemonSet not scheduling on a node: A DaemonSet has no pod on some nodes

A DaemonSet should run one pod per eligible node. Missing pods mean the node was filtered out — by a taint the DaemonSet does not tolerate, by a node selector, or by insufficient resources.

Applies to: All Kubernetes versions

What it means

The DaemonSet controller creates one pod per node that matches its criteria, and the scheduler places them like any other pod. A node with no DaemonSet pod was therefore excluded at one of two stages: the DaemonSet's own nodeSelector or affinity did not match it, or the scheduler rejected it — most often because of a taint the pod does not tolerate, or because the node has no room. Taints are the usual answer, because DaemonSets frequently need to run on nodes that ordinary workloads are deliberately kept off: control-plane nodes, GPU nodes, nodes marked unschedulable during maintenance.

Most common causes

• A node taint the DaemonSet does not tolerate — control-plane taints and specialised node-pool taints are the usual ones. Cordoning is not one of them; that toleration is added automatically.
• A nodeSelector or node affinity on the DaemonSet that the node does not satisfy.
• Insufficient allocatable resources on the node for the DaemonSet pod's requests.
• A node condition taint the DaemonSet does not tolerate. Note that cordoning is not one of these: the DaemonSet controller adds a toleration for the unschedulable taint automatically, which is exactly why kubectl drain needs --ignore-daemonsets.
• A different operating system or architecture than the DaemonSet's pods support.
• A pod on the node already using a host port the DaemonSet pod requires.

How to diagnose it

1. Compare desired against current: kubectl get daemonset DS -n NAMESPACE shows both, and the gap is the count of excluded nodes.
2. Find which nodes are missing a pod: kubectl get pods -l app=NAME -o wide against kubectl get nodes.
3. Check the node's taints: kubectl describe node NODE.
4. Compare against the DaemonSet's tolerations: kubectl get daemonset DS -o jsonpath='{.spec.template.spec.tolerations}'.
5. Check the node's labels against any nodeSelector.
6. Check for a host port conflict if the DaemonSet declares one.

How to fix it

1. Add the tolerations the DaemonSet needs for the taints that are actually present. Infrastructure DaemonSets commonly tolerate everything, which is appropriate for a CNI or a node monitoring agent and not for most other things.
2. Correct the node selector or the node's labels so they match.
3. Reduce the DaemonSet's resource requests, remembering they are charged against every node in the cluster.
4. Label nodes by operating system and architecture, and select on those labels, so a DaemonSet only targets nodes it can actually run on.
5. Resolve host port conflicts by changing the port or removing the conflicting workload.

Notes

DaemonSet resource requests are multiplied by the node count, so a request that looks small costs a lot across a large cluster. On a hundred-node cluster, an extra 100 mCPU per DaemonSet pod reserves ten CPUs cluster-wide.

Related

• FailedScheduling — The scheduler could not find a suitable node
• SchedulingDisabled — The node has been cordoned

Sources

• Kubernetes documentation — DaemonSet
• Kubernetes documentation — Taints and Tolerations
• Kubernetes documentation — Perform a Rolling Update on a DaemonSet