connection reset by peer: An established connection was terminated abruptly by the other side

A reset means the connection was working and then the far end sent a TCP RST. In a cluster this usually means the pod behind it went away mid-request, or something in the path expired the connection's state.

Applies to: All Kubernetes versions

What it means

ECONNRESET is different from a refusal or a timeout: the connection had been established, and then the peer's stack sent a reset. The most common cause in Kubernetes is a pod terminating while it still held connections — during a rollout, a scale-down, or an eviction. It also happens when a load balancer or proxy in the path closes an idle connection that the client still believes is open, and when a node's conntrack entry for the flow is dropped, after which returning packets no longer match a known connection and are reset. Because it is a mid-flight failure, it tends to produce a small number of failed requests during otherwise successful operations, which is what makes it easy to dismiss and hard to eliminate.

Most common causes

• A pod terminating during a rollout while still serving requests, because it did not drain connections on SIGTERM.
• An idle connection closed by an intermediate load balancer whose idle timeout is shorter than the client's.
• conntrack entries being evicted on a busy node, so an established flow stops being recognised.
• The application closing connections abruptly on error rather than completing the response.
• A service mesh sidecar shutting down before the application it fronts.
• Backend limits — a server closing connections when its accept queue or connection limit is exceeded.

How to diagnose it

1. Correlate the resets with deployments, scale events, and evictions: kubectl get events --sort-by=.lastTimestamp.
2. Check whether resets cluster around pod terminations — if so, the fix is graceful shutdown rather than anything network-level.
3. Compare the idle timeouts of every hop: client, ingress, load balancer, and server. The shortest one governs.
4. Check conntrack table usage and eviction counters on the nodes carrying the traffic.
5. Check the application logs on the server side at the same timestamp for its own view of the close.

How to fix it

1. Handle SIGTERM: stop accepting new connections, finish in-flight requests, then exit. This removes the largest share of resets during rollouts.
2. Add a PreStop hook with a short sleep so the pod is removed from endpoints before it stops accepting, since endpoint propagation is not instantaneous.
3. Set the client's idle timeout below the shortest timeout in the path, so the client closes connections before an intermediary does.
4. Raise conntrack limits on saturated nodes.
5. Fix server-side connection limits if resets correlate with load rather than with deployments.

Notes

The gap between a pod being deleted and every kube-proxy removing it from its forwarding rules is real and is not zero. A PreStop delay of a few seconds is the standard way to cover it, and it is why a perfectly graceful application can still see resets during a rollout without one.

Related

• Exit code 143 — Terminated by SIGTERM
• i/o timeout — The connection attempt got no response

Sources

• Kubernetes documentation — Pod Lifecycle: termination of pods
• Kubernetes documentation — Service
• Kubernetes documentation — Debug Services