ClusterIP unreachable: The Service has endpoints but its ClusterIP does not work from some or all pods

If a Service's endpoints are healthy and the pod IPs work directly but the ClusterIP does not, the problem is in the layer that translates ClusterIPs to pod IPs — normally kube-proxy on the client's node.

Applies to: All Kubernetes versions

What it means

A ClusterIP is a virtual address with nothing listening on it. Traffic to it is rewritten on the client's own node, by kube-proxy through iptables or IPVS rules, or by the CNI in clusters that replace kube-proxy entirely. So a ClusterIP that does not work while the backing pod IPs do is a statement about the client's node, not about the Service or the backend. That makes the diagnosis unusually tractable: test the same ClusterIP from pods on several nodes, and whether it fails everywhere or only on one node tells you immediately whether to look at the Service or at a single node's proxy.

Most common causes

• kube-proxy is not running on the client's node, or is crash-looping there.
• kube-proxy's rules are stale because it lost its connection to the API server.
• The cluster runs without kube-proxy, using CNI-based service handling, and the CNI agent is unhealthy on that node.
• A conflicting iptables ruleset on the node, often from host-level firewall management that flushes chains.
• The service CIDR overlaps something else in the node's routing.
• IPVS mode with a missing kernel module, so rules cannot be programmed.

How to diagnose it

1. Confirm endpoints exist: kubectl get endpointslices -l kubernetes.io/service-name=SERVICE -n NAMESPACE.
2. Test a backing pod IP directly. If that works and the ClusterIP does not, the backend is fine.
3. Test the ClusterIP from pods on different nodes to find whether it is node-specific.
4. Check kube-proxy on the failing node: kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide and read its logs.
5. On the node, check that rules for the ClusterIP exist: iptables-save | grep CLUSTER_IP, or ipvsadm -Ln in IPVS mode.
6. Check whether any host firewall tooling is managing the same chains.

How to fix it

1. Restart or repair kube-proxy on the affected node.
2. Stop host-level firewall tooling from flushing the chains kube-proxy owns, or configure it to coexist.
3. Load the kernel modules IPVS mode requires, or fall back to iptables mode.
4. Repair the CNI agent in clusters that handle Services without kube-proxy.
5. Ensure the service CIDR does not overlap the pod CIDR or the node network.

Notes

Because a ClusterIP is programmed per node, a partial failure is normal and expected in this scenario: the Service works from most pods and fails from a few. Testing from a single client and concluding the Service is broken is the usual wrong turn.

Related

• Service has no endpoints — No ready pods behind the Service
• i/o timeout — The connection attempt got no response

Sources

• Kubernetes documentation — Service: virtual IPs and service proxies
• Kubernetes documentation — Debug Services
• Kubernetes documentation — Cluster Networking