unable to get metric http_requests: no metrics returned from custom metrics API

[harshalk91]

Hi,
I followed the instructions and deployed podinfo app and i am able to scale the app on http_requests. However, whenever i run my own app or just nginx, i am getting error as "unable to get metric http_requests: no metrics returned from custom metrics API". Deployment files are exactly the same, just changed port number, health_checks and labels.

[ec2-user@ip-192-168-100-253 nginx]$ kubectl get hpa
NAME               REFERENCE            TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
nginx-custom-hpa   Deployment/nginx     <unknown>/10   2         10        2          10m
podinfo            Deployment/podinfo   888m/10        2         10        2          23m

[ec2-user@ip-192-168-100-253 nginx]$ kubectl describe hpa nginx-custom-hpa
Name:                       nginx-custom-hpa
Namespace:                  default
Labels:                     <none>
Annotations:                <none>
CreationTimestamp:          Mon, 19 Aug 2019 08:32:58 +0000
Reference:                  Deployment/nginx
Metrics:                    ( current / target )
  "http_requests" on pods:  <unknown> / 10
Min replicas:               2
Max replicas:               10
Deployment pods:            2 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric http_requests: no metrics returned from custom metrics API
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedComputeMetricsReplicas  8m18s (x12 over 11m)  horizontal-pod-autoscaler  failed to get object metric value: unable to get metric http_requests: no metrics returned from custom metrics API
  Warning  FailedGetPodsMetric           63s (x41 over 11m)    horizontal-pod-autoscaler  unable to get metric http_requests: no metrics returned from custom metrics API

Is my app running??

[ec2-user@ip-192-168-100-253 nginx]$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.100.0.1       <none>        443/TCP          21d
nginx-svc    NodePort    10.100.195.60    <none>        9899:31199/TCP   13m
podinfo      NodePort    10.100.112.143   <none>        9898:31198/TCP   24m

[ec2-user@ip-192-168-101-39 ~]$ curl -I http://10.100.195.60:9899
HTTP/1.1 200 OK
Server: nginx/1.17.3

[ec2-user@ip-192-168-100-253 ~]$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "podinfo-7b8c9bc5c9-xgppt",
        "apiVersion": "/v1"
      },
      "metricName": "http_requests",
      "timestamp": "2019-08-19T08:48:38Z",
      "value": "888m"
    },
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "podinfo-7b8c9bc5c9-zkkbd",
        "apiVersion": "/v1"
      },
      "metricName": "http_requests",
      "timestamp": "2019-08-19T08:48:38Z",
      "value": "888m"
    }
  ]
}

I would like to know why i am not able to see http_requests for nginx app?

[harshalk91]

@stefanprodan Any clue?

[harshalk91]

@stefanprodan What changes do i need to perform when i wish to run my apps under separate namespace?

[kjdeepthi]

Any update on this?, i'm running into same issue

[Go-Pomegranate]

Any update?

[krish512]

Guys, any update on this? Was any one able to achive this with nginx or any other service?
Please up vote this, we need help here

[ntman4real]

@krish512 Was wondering the same. How do we get the http_requests custom metrics from any pod we deploy? Is there something we need to insert/configure with the deployment or image itself? It only works with that podinfo deployment.

[krish512]

I guess the pod needs to expose this metrics and Custom Metrics Exporters can just capture these and make them available for HPA
Not sure if this is correct, can someone confirm if my understanding is correct

[Mr97Sir]

did anyone resolve this issue?
if you guys did can someone point me in the path as to the changes that need to be done to the existing pod to allow Prometheus to read data from it?

[ntman4real]

I was finally able to figure it out. Each webserver that needs monitoring(like the go example or apache or in our case nginx), has its own mechanism to expose metrics for prom to scrape. In the case of nginx, I had to enable the module nginx-module-vts, and then create a simple rule. Here is a link to my repo if you want to see.

https://github.com/ntman4real/nginxmets

Let me know if you have any questions, and I am now seeing how to do the same with springboot. More to come!

[Mr97Sir]

I was finally able to figure it out. Each webserver that needs monitoring(like the go example or apache or in our case nginx), has its own mechanism to expose metrics for prom to scrape. In the case of nginx, I had to enable the module nginx-module-vts, and then create a simple rule. Here is a link to my repo if you want to see.

Thanks i'm looking to expose metrics for Go & Node... but i'll check it out... Thanks!

[ChuyueZh]

Add a rule to custom-metrics-config-map.yaml:

- seriesQuery: '{__name__=~"^http_requests_.*",kubernetes_pod_name!="",kubernetes_namespace!=""}'
      seriesFilters: []
      resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_pod_name:
            resource: pod
      name:
        matches: ^(.*)_(total)$
        as: "${1}"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)