unable to upgrade connection: kubectl exec or attach could not establish its streaming connection to the kubelet

exec, attach, port-forward, and logs need a streaming connection that goes from the API server to the kubelet. This error means that second hop failed, even though ordinary API calls work.

Applies to: All Kubernetes versions

What it means

Most kubectl commands are simple API requests. exec, attach, port-forward, and streaming logs are different: the API server upgrades the connection and proxies it to the kubelet on the pod's node. That introduces a second network path — API server to kubelet on port 10250 — which ordinary requests never use. When it fails, everything else about the cluster works perfectly and only these commands break, which is a distinctive and initially confusing symptom. The message text varies: pod does not exist, error dialing backend, or a TLS error, each pointing at a different part of that hop.

Most common causes

• A firewall blocking the API server from reaching kubelets on port 10250.
• The kubelet's serving certificate not being trusted by the API server.
• The pod having been rescheduled, so the node in the API server's view is stale.
• The node being NotReady or unreachable.
• A network policy or security group between the control plane and the node network.
• The container having no shell, for exec specifically — a different failure with a similar-looking message.
• A proxy in front of the API server that does not support connection upgrades.

How to diagnose it

1. Check the pod is running and on which node: kubectl get pod POD -o wide.
2. Try another pod on a different node. If one node fails and others work, the problem is that node's connectivity.
3. Check the node's status and whether the kubelet is healthy.
4. Check whether ordinary API calls work — if kubectl get succeeds and exec fails, the second hop is confirmed as the problem.
5. For exec failures naming a missing binary, check whether the image has a shell at all: distroless images do not.

How to fix it

1. Open port 10250 from the control plane to the nodes.
2. Ensure kubelet serving certificates are signed by an authority the API server trusts, and that certificate rotation is approved if the cluster requires it.
3. Retry after a reschedule, since the stale-node case resolves itself.
4. Restore the unhealthy node.
5. Use an ephemeral debug container for images with no shell: kubectl debug -it POD --image=busybox --target=CONTAINER.
6. Configure any intervening proxy to support connection upgrades.

Notes

This is one of the few failures where the cluster is entirely functional and only interactive access is broken. Workloads keep serving, deployments keep rolling, and only the ability to look inside a pod is lost — which is why it often goes unnoticed until an incident.

Related

• kubectl port-forward drops — The forwarded connection closes unexpectedly
• NotReady — The node is not accepting work

Sources

• Kubernetes documentation — kubectl exec
• Kubernetes documentation — Debug Running Pods
• Kubernetes documentation — Troubleshooting Clusters