KubeErrors

Exit code 132: The process was terminated by SIGILL — an illegal instruction

132 is 128 + 4, meaning SIGILL. The CPU was asked to execute an instruction it does not implement, which in containers almost always means the binary was built for a different processor than the node has.

Applies to: All Kubernetes versions, Linux nodes

What it means

Signal 4 is SIGILL, raised when a process attempts to execute an illegal, malformed, or unimplemented machine instruction. In a container the overwhelmingly likely cause is an architecture or instruction-set mismatch: a binary compiled with optimisations for a newer CPU running on an older one. This is easy to produce accidentally in a heterogeneous cluster, where a node pool added later has different hardware, or where a build used -march=native on a build machine that is not representative of the nodes. It is distinct from an outright architecture mismatch, such as an arm64 image on amd64, which normally fails earlier with an exec format error rather than reaching the point of running instructions.

Most common causes

How to diagnose it

  1. Identify which node the pod ran on and what CPU it has: kubectl get pod POD -o wide, then check /proc/cpuinfo on that node.
  2. Check whether the failure is node-specific. If the pod runs fine on some nodes and dies on others, the diagnosis is settled.
  3. Compare the instruction sets the binary requires against the node's CPU flags.
  4. Look for a stack trace or core dump in the logs — some runtimes report the offending instruction address.

How to fix it

  1. Build for a conservative baseline architecture rather than the build machine's own CPU. Never use -march=native for anything that will be distributed.
  2. Use libraries that select CPU-specific code paths at runtime rather than fixing them at compile time.
  3. Label nodes by CPU capability and schedule sensitive workloads with a node selector if a specific instruction set is genuinely required.
  4. Make node pools homogeneous where practical, so a workload that passes testing cannot fail on a subset of the fleet.

Notes

This failure is intermittent in exactly the way that makes it hard to trust — the same image works perfectly on most of the cluster. The pattern of which nodes fail is the diagnosis, so record it before rescheduling anything.

Related

Sources

Pages on this site are written with AI assistance from the primary sources listed on each page, then checked against those sources before publishing.