i/o timeout: The connection attempt got no response at all

dial tcp …: i/o timeout means packets were sent and nothing came back. Something is dropping traffic silently — a network policy, a firewall, a routing gap, or an unreachable destination.

Applies to: All Kubernetes versions

What it means

A timeout means the packets went into the network and no response returned, not even a rejection. Silent dropping is characteristic of firewalls and policies, which are usually configured to drop rather than reject so as not to confirm that a host exists. Inside a cluster the candidates are a NetworkPolicy denying the traffic, a cloud security group between nodes, a CNI problem leaving pods on different nodes unable to reach each other, or a destination that is simply gone. It is worth being systematic here, because each of these is invisible from the client's perspective and they are distinguished by where you test from rather than by the error text.

Most common causes

• A NetworkPolicy denying ingress to the destination or egress from the source.
• A cloud firewall or security group blocking the pod network between nodes.
• A CNI failure so that pods on different nodes cannot reach each other, while same-node traffic works.
• The destination pod is gone and its IP is stale, for example a cached address after a rollout.
• conntrack table exhaustion on a node, which drops new connections silently.
• An external destination that is unreachable or filtered outside the cluster.
• MTU mismatch, where the handshake completes and larger packets vanish — which presents as a timeout only for some requests.

How to diagnose it

1. Narrow the scope: test from a pod on the same node as the destination, then from a different node. Same-node working and cross-node failing points at the CNI or the underlying network.
2. Test the destination's pod IP directly and its ClusterIP separately, to separate a Service problem from a network problem.
3. List policies that could apply. Run this once per namespace — -n takes a single value, and passing it twice silently uses only the last one: kubectl get networkpolicy -n SOURCE_NS then kubectl get networkpolicy -n DEST_NS, and read them — a single default-deny policy affects everything in its namespace.
4. Check conntrack usage on the source node.
5. For external destinations, test from the node itself to see whether the pod network is involved at all.
6. If small requests succeed and large ones hang, suspect MTU and test with a fixed packet size.

How to fix it

1. Add the NetworkPolicy rules the traffic needs, in both directions if both namespaces have policies.
2. Open the required ports in the cloud firewall, including the ports the CNI uses for its overlay.
3. Repair or restart the CNI agent on the affected nodes.
4. Raise conntrack limits on nodes that are exhausting the table.
5. Set the CNI's MTU correctly for the underlying network, particularly where an overlay adds encapsulation overhead.

Notes

The single most useful step is testing from a pod on the same node as the destination. It splits the problem in half in one command and is faster than reading any amount of policy configuration.

Related

• connection refused — The destination was reached and the port was closed
• NetworkPolicy blocking traffic — A policy is dropping the connection

Sources

• Kubernetes documentation — Debug Services
• Kubernetes documentation — Network Policies
• Kubernetes documentation — Network Plugins