CustomResourceStateMetrics for VPA

[andreivaloshyn]

Hello,
I'm trying to create a custom metric for vertical pod autoscaler.

KSM SA role was updated with

  - verbs:
      - list
      - watch
    apiGroups:
      - autoscaling.k8s.io
    resources:
      - verticalpodautoscalers
  - verbs:
      - list
      - watch
    apiGroups:
      - apiextensions.k8s.io
    resources:
      - customresourcedefinitions

VPA resource example:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
......
status:
  conditions:
    - lastTransitionTime: '2023-07-28T08:16:32Z'
      status: 'True'
      type: RecommendationProvided
  recommendation:
    containerRecommendations:
      - containerName: config-reloader
        lowerBound:
          cpu: 10m
          memory: '52428800'
        target:
          cpu: 11m
          memory: '52428800'
        uncappedTarget:
          cpu: 11m
          memory: '52428800'
        upperBound:
          cpu: 318m
          memory: '1048640246'
      - containerName: prometheus
        lowerBound:
          cpu: 103m
          memory: '1738122210'
        target:
          cpu: 163m
          memory: '1836551791'
        uncappedTarget:
          cpu: 163m
          memory: '1836551791'
        upperBound:
          cpu: 5871m
          memory: '53122290233'

Based on the examples here https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md
I created config:

            - --custom-resource-state-config
            -  |
                spec:
                    resources:
                      - groupVersionKind:
                          group: autoscaling.k8s.io
                          kind: "VerticalPodAutoscaler"
                          version: "v1"
                        labelsFromPath:
                          verticalpodautoscaler: [metadata, name]
                          namespace: [metadata, namespace]
                        metrics:
                          - name: "memorytarget"
                            help: "Kubernetes annotations converted to Prometheus labels."
                            each:
                              type: Gauge
                              gauge:
                                path: [status, recommendation, containerrecommendations, "0", target, memory]

I expected a metric about memory target for config-reloader container, instead I'm getting a lot of "kube_customresource_memorytarget" err="[status,recommendation,containerrecommendations,0,target,memory]: got nil while resolving path" :

I0728 10:31:15.663414       1 wrapper.go:98] "Starting kube-state-metrics"
W0728 10:31:15.664040       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0728 10:31:15.664298       1 server.go:198] "Used resources" resources=[services namespaces pods secrets validatingwebhookconfigurations certificatesigningrequests deployments statefulsets cronjobs daemonsets storageclasses networkpolicies resourcequotas jobs limitranges persistentvolumeclaims persistentvolumes configmaps horizontalpodautoscalers endpoints ingresses poddisruptionbudgets replicasets mutatingwebhookconfigurations nodes replicationcontrollers volumeattachments]
I0728 10:31:15.664359       1 types.go:184] "Using all namespaces"
I0728 10:31:15.664388       1 server.go:225] "Metric allow-denylisting" allowDenyStatus="Excluding the following lists that were on denylist: "
W0728 10:31:15.664408       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0728 10:31:15.665248       1 utils.go:70] "Tested communication with server"
I0728 10:31:15.765357       1 utils.go:75] "Run with Kubernetes cluster version" major="1" minor="24+" gitVersion="v1.24.15-eks-a5565ad" gitTreeState="clean" gitCommit="af0b16a331376384fe84445fe2f8d4cb468b4148" platform="linux/amd64"
I0728 10:31:15.765401       1 utils.go:76] "Communication with server successful"
I0728 10:31:15.766506       1 server.go:347] "Started metrics server" metricsServerAddress="[::]:8080"
I0728 10:31:15.766543       1 server.go:336] "Started kube-state-metrics self metrics server" telemetryAddress="[::]:8081"
I0728 10:31:15.766900       1 server.go:72] levelinfomsgListening onaddress[::]:8080
I0728 10:31:15.766929       1 server.go:72] levelinfomsgTLS is disabled.http2falseaddress[::]:8080
I0728 10:31:15.767284       1 server.go:72] levelinfomsgListening onaddress[::]:8081
I0728 10:31:15.767301       1 server.go:72] levelinfomsgTLS is disabled.http2falseaddress[::]:8081
I0728 10:31:21.767032       1 config.go:82] "Using custom resource plural" resource="autoscaling.k8s.io_v1_VerticalPodAutoscaler" plural="verticalpodautoscalers"
I0728 10:31:21.767193       1 discovery.go:274] "discovery finished, cache updated"
I0728 10:31:21.767229       1 metrics_handler.go:99] "Autosharding disabled"
I0728 10:31:21.769850       1 custom_resource_metrics.go:79] "Custom resource state added metrics" familyNames=[kube_customresource_memorytarget]
I0728 10:31:23.765632       1 builder.go:275] "Active resources" activeStoreNames="certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments,autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
E0728 10:31:25.364087       1 registry_factory.go:662] "kube_customresource_memorytarget" err="[status,recommendation,containerrecommendations,0,target,memory]: got nil while resolving path"
.......

Could somebody help me and explain what I'm doing wrong? :)

[andreivaloshyn]

Seems like it works for a single container, at least I see metrics from KSM.
But when using "*" in path: [status, recommendation, containerrecommendations, "*", target, memory] instead of path: [status, recommendation, containerrecommendations, "0", target, memory] I see nothing.