failed calling webhook: The API server could not reach an admission webhook
This is not a policy rejection — the webhook never answered. With failurePolicy: Fail, that blocks every request the webhook intercepts, which can stop the whole cluster from accepting changes.
Applies to: All Kubernetes versions with admission webhooks
What it means
Before admitting an object, the API server calls each webhook registered for it. When that call fails — the service has no endpoints, the connection times out, TLS verification fails — the failurePolicy decides what happens. Ignore admits the object anyway, silently skipping the policy. Fail, which is the safer choice for a security policy and the default, rejects the request. The consequence is a failure mode that is easy to underestimate: a webhook intercepting pod creation, with failurePolicy: Fail, that becomes unavailable will prevent every new pod in the cluster from being created — including the webhook's own replacement pods.
Most common causes
- The webhook's Service has no ready endpoints because its pods are down.
- The webhook's certificate is not trusted by the API server, so the TLS handshake fails.
- A network policy blocking the API server from reaching the webhook's pods.
- The webhook is too slow and exceeds its
timeoutSeconds. - The webhook was uninstalled without removing its configuration object, leaving a rule pointing at nothing.
- The webhook intercepting resources in its own namespace, creating a circular dependency during cluster startup.
- The API server being unable to reach the pod network at all, which is the case in some control-plane topologies.
How to diagnose it
- Check the webhook's pods and endpoints:
kubectl get pods,endpointslices -n WEBHOOK_NS. - Read the exact failure in the error — a TLS error, a timeout, and no endpoints are three different problems.
- Check the webhook configuration's
failurePolicy,timeoutSeconds, andnamespaceSelector:kubectl get validatingwebhookconfiguration NAME -o yaml. - Confirm the configuration still refers to something that exists — an orphaned configuration is a common cause after an uninstall.
- Check the webhook's own logs and latency if it is running.
How to fix it
- Restore the webhook's pods. If pod creation itself is blocked, deleting the webhook configuration temporarily may be the only way out — which is why this failure is worth designing against in advance.
- Exclude system namespaces with a
namespaceSelectorso a broken webhook cannot prevent the cluster's own components from starting. - Fix the certificate trust between the API server and the webhook.
- Raise
timeoutSecondsif the webhook is legitimately slow, and make it faster. - Remove orphaned webhook configurations when uninstalling the component that owned them.
- Run at least two webhook replicas with a PodDisruptionBudget, since a single replica is a single point of failure for the whole API.
Notes
Excluding kube-system and the webhook's own namespace via namespaceSelector is the standard protection against a webhook that cannot be repaired because it is blocking its own repair. It is worth setting before it is needed.
Related
- admission webhook denied the request — An admission webhook rejected the object
- x509: certificate signed by unknown authority — The client does not trust the server's certificate
Sources
- Kubernetes documentation — Dynamic Admission Control
- Kubernetes documentation — Admission Controllers Reference
- Kubernetes documentation — Disruptions