[cinder-csi-plugin] Volumes are not exposed to kubelet_volume_stats_* metrics.

[celiawa]

All the currently maintained binaries are:

• cinder-csi-plugin
  /kind bug

What happened:
In the previous release, we use intree cinder provider and the volumes created could be exposed to via command:
curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer ******" https://kubernetes.default.svc:443/api/v1/nodes/control-0/proxy/metrics | grep "^kubelet_volume_stats_available_bytes".

While with cinder csi, the volumes can't be exposed.

What you expected to happen:
The volumes can be exposed to kubelet_volume_stats_* metrics.

Anything else we need to know?:
Environment:

• cinder-csi-plugin: v1.17.0
• kubernetes version: v1.17.4

[ramineni]

@celiawa GET_VOLUME_STATS node capability is implemented as part of #941 , and would be available as part of 1.19 release. If you would like to use, you could use :latest tag image where the capability would be available.

[jichenjc]

@ramineni
#913
seems talked about metrics, that issue need upgrade snapshot side car version
so without #913, can we still get metrics above?

[ramineni]

@jichenjc From the issue I see that its about volume usage details (kubelet_volume_stats_available_bytes), so yes, volume usage details can be retrieved using GET_VOLUME_STATS node capability.

[jichenjc]

@ramineni thanks for the info, so we can wait for @celiawa test then close this

[jichenjc]

@ramineni I used local env with latest image:

kubectl describe pods csi-cinder-nodeplugin-grdqc -n kube-system | grep Image:
    Image:         quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
    Image:         docker.io/k8scloudprovider/cinder-csi-plugin:latest


# kubectl get pods -n kube-system
NAME                            READY   STATUS    RESTARTS   AGE
csi-cinder-controllerplugin-0   5/5     Running   0          3m48s
csi-cinder-nodeplugin-grdqc     2/2     Running   0          3m46s
kube-dns-58bc79b99f-wfvvb       3/3     Running   0          3m53s

however, I can't find kubelet_volume_xxx in output of

curl -k https://localhost:6443/api/v1/nodes/127.0.0.1/proxy/metrics | grep kubelet_volume
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  100k    0  100k    0     0  2716k      0 --:--:-- --:--:-- --:--:-- 2716k

there are a lot of output such as following if I only run

curl -k https://localhost:6443/api/v1/nodes/127.0.0.1/proxy/metrics

kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="5"} 3
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="10"} 3
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="+Inf"} 3
kubelet_run_podsandbox_duration_seconds_sum{runtime_handler=""} 3.2561e-05
kubelet_run_podsandbox_duration_seconds_count{runtime_handler=""} 3
# HELP kubelet_running_container_count [ALPHA] Number of containers currently running
# TYPE kubelet_running_container_count gauge
kubelet_running_container_count{container_state="created"} 2
kubelet_running_container_count{container_state="exited"} 10
kubelet_running_container_count{container_state="running"} 10
# HELP kubelet_running_pod_count [ALPHA] Number of pods currently running
# TYPE kubelet_running_pod_count gauge
kubelet_running_pod_count 3
# HELP kubelet_runtime_operations_duration_seconds [ALPHA] Duration in seconds of runtime operations. Broken down by operation type.
# TYPE kubelet_runtime_operations_duration_seconds histogram
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.005"} 53
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.01"} 62
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.025"} 73
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.05"} 74
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.1"} 75
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.25"} 75
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.5"} 75

@celiawa do you know any special settings to be enabled in order to get volume metrics if it's supported?

[jichenjc]

@celiawa @ramineni any comments on this?

[ramineni]

@celiawa could you confirm ? else we will close this issue , if no response further

[ramineni]

/close

[k8s-ci-robot]

@ramineni: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.