NetworkPolicy blocking traffic: A policy is silently dropping connections that used to work

NetworkPolicies deny by default once any policy selects a pod. Traffic they block is dropped without any error, so the symptom is always a timeout with nothing in any log.

Applies to: Clusters with a CNI that enforces NetworkPolicy

What it means

NetworkPolicy is additive and default-deny in a specific way that surprises people: a pod with no policy selecting it accepts everything, but as soon as one policy selects it for a direction, only traffic explicitly allowed by some policy in that direction is permitted. So adding a single ingress policy to allow one client silently blocks every other client. Enforcement happens in the CNI, and blocked packets are dropped rather than rejected, meaning no error is generated anywhere — the client times out and the server never sees the connection. There is nothing to find in application logs on either side, which is why this is one of the harder cluster problems to attribute.

Most common causes

• A new policy selecting pods that previously had none, implicitly denying everything not listed.
• A default-deny policy applied namespace-wide without exceptions for DNS, monitoring, or health probes.
• Ingress allowed but egress forgotten, or the reverse — the two directions are independent and both must permit the flow.
• A policy in the destination namespace and another in the source namespace, where only one was updated.
• Namespace selectors matching on labels the namespace does not actually carry.
• A CNI that does not enforce NetworkPolicy at all, so a policy intended to restrict traffic does nothing — the opposite failure, and worth checking.

How to diagnose it

1. List policies affecting both ends: kubectl get networkpolicy -n SOURCE_NS and -n DEST_NS, then read them rather than skimming.
2. Check whether the destination pod is selected by any policy at all — if it is, everything not allowed is denied.
3. Test by temporarily deploying a permissive policy in a non-production namespace to confirm the diagnosis before changing anything real.
4. Check namespace labels used by namespaceSelector: kubectl get ns --show-labels.
5. Confirm the CNI enforces policy at all — several do not, and a policy that is not enforced is a false sense of security rather than a cause of blocked traffic.
6. Check the CNI's own logs, which some implementations use to record denied flows.

How to fix it

1. Write the missing allow rules, remembering that ingress on the destination and egress on the source are separate requirements.
2. Always pair a default-deny policy with an explicit DNS exception for UDP and TCP port 53 to the cluster DNS service.
3. Allow the kubelet's probe traffic to reach pods, or readiness and liveness probes will fail cluster-wide.
4. Label namespaces consistently so namespaceSelector rules are reliable.
5. Introduce default-deny in a staging environment first. The failure mode is silent, so the cost of getting it wrong in production is high.

Notes

The single most common outage caused by NetworkPolicy is a default-deny egress rule with no DNS exception. Every name resolution in the namespace fails, everything becomes intermittently broken, and the policy is rarely the first suspect.

Related

• i/o timeout — The connection attempt got no response
• DNS timeout — Cluster DNS did not answer

Sources

• Kubernetes documentation — Network Policies
• Kubernetes documentation — Network Plugins
• Kubernetes documentation — Debug Services