KubeErrors

failed calling webhook: The API server could not reach an admission webhook

This is not a policy rejection — the webhook never answered. With failurePolicy: Fail, that blocks every request the webhook intercepts, which can stop the whole cluster from accepting changes.

Applies to: All Kubernetes versions with admission webhooks

What it means

Before admitting an object, the API server calls each webhook registered for it. When that call fails — the service has no endpoints, the connection times out, TLS verification fails — the failurePolicy decides what happens. Ignore admits the object anyway, silently skipping the policy. Fail, which is the safer choice for a security policy and the default, rejects the request. The consequence is a failure mode that is easy to underestimate: a webhook intercepting pod creation, with failurePolicy: Fail, that becomes unavailable will prevent every new pod in the cluster from being created — including the webhook's own replacement pods.

Most common causes

How to diagnose it

  1. Check the webhook's pods and endpoints: kubectl get pods,endpointslices -n WEBHOOK_NS.
  2. Read the exact failure in the error — a TLS error, a timeout, and no endpoints are three different problems.
  3. Check the webhook configuration's failurePolicy, timeoutSeconds, and namespaceSelector: kubectl get validatingwebhookconfiguration NAME -o yaml.
  4. Confirm the configuration still refers to something that exists — an orphaned configuration is a common cause after an uninstall.
  5. Check the webhook's own logs and latency if it is running.

How to fix it

  1. Restore the webhook's pods. If pod creation itself is blocked, deleting the webhook configuration temporarily may be the only way out — which is why this failure is worth designing against in advance.
  2. Exclude system namespaces with a namespaceSelector so a broken webhook cannot prevent the cluster's own components from starting.
  3. Fix the certificate trust between the API server and the webhook.
  4. Raise timeoutSeconds if the webhook is legitimately slow, and make it faster.
  5. Remove orphaned webhook configurations when uninstalling the component that owned them.
  6. Run at least two webhook replicas with a PodDisruptionBudget, since a single replica is a single point of failure for the whole API.

Notes

Excluding kube-system and the webhook's own namespace via namespaceSelector is the standard protection against a webhook that cannot be repaired because it is blocking its own repair. It is worth setting before it is needed.

Related

Sources

Pages on this site are written with AI assistance from the primary sources listed on each page, then checked against those sources before publishing.