Exit code 127: The command was not found

Exit code 127 means the command in the pod spec does not exist at that path in the image. It is one of the most reliably diagnosable container failures, because the answer is always inside the image.

Applies to: All Kubernetes versions, Linux nodes

What it means

The shell convention reserves 127 for a command that could not be found. In a container this means the path named in command, or the image's entrypoint, does not resolve — either the file is genuinely absent, or something it depends on to load is. That second case is the one that wastes time: a dynamically linked binary whose interpreter or shared library is missing produces no such file or directory naming the binary, even though the binary is sitting right there. This happens constantly when a binary built against glibc is copied into an Alpine image, which uses musl and has no /lib64/ld-linux-x86-64.so.2.

Most common causes

• A typo in the command path, or a binary that is not on PATH.
• A binary built on a glibc system copied into a musl-based image such as Alpine, so its dynamic loader is absent.
• A shell script whose shebang names an interpreter the image does not have — /bin/bash in an image that only ships /bin/sh.
• A multi-stage build that did not copy the binary into the final stage.
• The command is a shell builtin or alias that only exists in an interactive shell.
• A CRLF line ending on the shebang, making the interpreter path include an invisible carriage return.

How to diagnose it

1. Read the event or log message: kubectl describe pod POD. It usually names the path exactly.
2. Check the image directly with an overridden entrypoint: kubectl run debug --rm -it --image=YOUR_IMAGE --command -- sh, then ls -l the path.
3. If the binary exists but still reports not found, check its dynamic dependencies with ldd /path/to/binary. A missing loader is the classic case.
4. Check the shebang of any script: head -1 script.sh | cat -A will reveal a trailing ^M.
5. For distroless or scratch images, remember there is no shell at all — a command written as a shell string cannot work.

How to fix it

1. Correct the path, or install the missing tool in the image.
2. Build against the same libc as the runtime image, or build a fully static binary, or switch to a glibc-based base image.
3. Use /bin/sh rather than /bin/bash in minimal images, or install bash explicitly.
4. Ensure multi-stage builds copy every artefact the final stage needs.
5. Normalise line endings — configure the repository so shell scripts are checked out with LF.

Notes

Distroless and scratch images have no shell, so command: ["sh", "-c", "…"] fails with 127 even though the underlying program is present. The exec form without a shell is required there.

Related

• Exit code 126 — Command found but not executable
• RunContainerError — The container was created but failed to start

Sources

• GNU Bash Reference Manual — Exit Status
• Kubernetes documentation — Define a Command and Arguments for a Container
• Kubernetes documentation — Debug Pods