Ingress 504: The backend did not respond within the ingress controller's timeout

A 504 means the ingress connected to a backend and waited, and the response did not arrive in time. The backend is slow, hung, or waiting on something else that is.

Applies to: All Kubernetes versions with an ingress controller

What it means

A gateway timeout is generated by the ingress controller when its read timeout for the upstream expires. The connection succeeded — so endpoints exist, the Service is correct, and the network works — and the backend simply did not answer in time. Most controllers default to a read timeout of around 60 seconds, which is shorter than many long-running operations, so a 504 is as often a mismatch between the controller's timeout and the application's legitimate work as it is a sign of a hung backend. Distinguishing them takes measuring how long the request actually takes when called directly.

Most common causes

• The backend is genuinely slow for this request — a heavy query, a large export, an unindexed lookup.
• The controller's proxy read timeout is shorter than the operation's normal duration.
• The backend is waiting on its own dependency that has stalled.
• The backend's worker pool or connection pool is exhausted, so the request never gets picked up.
• A deadlock or thread starvation in the application.
• A long-polling or streaming endpoint being run through a timeout intended for ordinary requests.

How to diagnose it

1. Time the same request directly against the Service from inside the cluster, bypassing the ingress.
2. Read the controller's logs for the upstream response time it recorded.
3. Check the backend's own logs and metrics for request duration at that moment.
4. Check whether the backend is saturated — CPU throttling, a full worker pool, or a queue depth that is growing.
5. Check the backend's downstream dependencies for stalls at the same timestamp.

How to fix it

1. Fix the slow operation, if the duration is not intentional. This is the correct fix in most cases.
2. Raise the controller's read timeout for the specific path or Ingress, if the duration is legitimate. Do it per route rather than globally.
3. Use a separate Ingress or route with a longer timeout for streaming and long-polling endpoints.
4. Scale the backend, or raise its concurrency limits, if requests are queuing rather than executing slowly.
5. Move genuinely long operations to an asynchronous pattern — accept, return a job id, and let the client poll — rather than holding an HTTP connection open for minutes.

Notes

Raising the timeout is the fastest fix and the one most likely to hide a real problem. It is worth measuring the actual duration first, because a request that takes 55 seconds today will take 65 next month.

Related

• Ingress 502 — The backend returned an invalid response
• Probe: timeout — The probe got no answer within its timeout

Sources

• Kubernetes documentation — Ingress
• Kubernetes documentation — Ingress Controllers
• Kubernetes documentation — Debug Services