unable to fetch metrics: The HorizontalPodAutoscaler cannot read the metric it scales on

An HPA showing <unknown> for its current metric cannot make a scaling decision. The usual causes are a missing metrics server, or pods with no resource requests to compute a percentage against.

Applies to: All Kubernetes versions with an HPA

What it means

An HPA scales on metrics it reads from an API — the resource metrics API for CPU and memory, or a custom or external metrics API for anything else. When it cannot read the metric, it reports unable to fetch metrics and its target column shows <unknown>. Two causes dominate. The metrics server may not be installed or healthy, in which case nothing gets CPU metrics. Or the pods may have no CPU request set, which matters because a utilisation target is a percentage of the request — with no request there is no denominator, and the HPA cannot compute anything even though the raw metric exists.

Most common causes

• The metrics server is not installed, or is not running correctly.
• The target pods have no CPU or memory requests, so utilisation cannot be computed.
• The custom metrics adapter is missing or unhealthy for a non-resource metric.
• The metrics API is registered but its APIService is unavailable.
• The metrics server cannot reach kubelets, often a TLS or network problem.
• The pods are too new — metrics take a short time to appear after a pod starts.
• The HPA targets a workload that does not exist or is misnamed.

How to diagnose it

1. Read the HPA's status and events: kubectl describe hpa HPA.
2. Check the metrics API answers at all: kubectl top pods -n NAMESPACE. If this fails, the problem is the metrics pipeline, not the HPA.
3. Check the APIService: kubectl get apiservices | grep metrics and look for availability.
4. Confirm the target pods have requests set: kubectl get pod POD -o jsonpath='{.spec.containers[*].resources.requests}'. Empty is the answer for a utilisation-based HPA.
5. For custom metrics, check the adapter's logs and that its API is registered.
6. Confirm the HPA's scaleTargetRef names an object that exists.

How to fix it

1. Install or repair the metrics server, and confirm it can reach the kubelets.
2. Set CPU and memory requests on the target pods. A utilisation-based HPA cannot work without them.
3. Install or fix the custom metrics adapter for non-resource metrics.
4. Use AverageValue rather than Utilization if requests genuinely cannot be set, since that form does not need a denominator.
5. Correct the scaleTargetRef if it points at the wrong workload.

Notes

The requirement for resource requests is easy to miss because it is not a metrics problem at all — the metric is available, and the HPA still reports it as unknown. Checking whether the pods have requests should come before investigating the metrics pipeline.

Related

• Insufficient cpu — No node has enough unreserved CPU
• Error from server (NotFound) — The named object does not exist

Sources

• Kubernetes documentation — Horizontal Pod Autoscaling
• Kubernetes documentation — Resource Metrics Pipeline
• Kubernetes documentation — HorizontalPodAutoscaler Walkthrough