karmada controller can't find metrics workqueue_depth

[CharlesQQ]

Please provide an in-depth description of the question you have:

curl karmada metrics endpoint, like curl http://127.0.0.1:10358/metrics, can't found metrics workqueue_depth

controller_runtime_reconcile_total{controller="cluster",result="requeue"} 0
controller_runtime_reconcile_total{controller="cluster",result="requeue_after"} 20665
controller_runtime_reconcile_total{controller="clusterResourceBinding_status_controller",result="requeue"} 0
controller_runtime_reconcile_total{controller="clusterResourceBinding_status_controller",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="clusterresourcebinding",result="requeue"} 0
controller_runtime_reconcile_total{controller="clusterresourcebinding",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="cronfederatedhpa",result="requeue"} 0
controller_runtime_reconcile_total{controller="cronfederatedhpa",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="federatedhpa",result="requeue"} 0
controller_runtime_reconcile_total{controller="federatedhpa",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="federatedresourcequota",result="requeue"} 0
controller_runtime_reconcile_total{controller="federatedresourcequota",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="namespace",result="requeue"} 0
controller_runtime_reconcile_total{controller="namespace",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="remedy-controller",result="requeue"} 0
controller_runtime_reconcile_total{controller="remedy-controller",result="requeue_after"} 0
controller_runtime_reconcile_total{controller="resourceBinding_status_controller",result="requeue"} 0
controller_runtime_reconcile_total{controller="resourceBinding_status_controller",result="requeue_after"} 0
10controller_runtime_reconcile_total{controller="resourcebinding",result="requeue"} 0
0 6139controller_runtime_reconcile_total{controller="resourcebinding",result="requeue_after"} 0
6controller_runtime_reconcile_total{controller="serviceimport",result="requeue"} 0
 controller_runtime_reconcile_total{controller="serviceimport",result="requeue_after"} 0
 controller_runtime_reconcile_total{controller="work",result="requeue"} 0
 controller_runtime_reconcile_total{controller="work",result="requeue_after"} 0
 controller_runtime_reconcile_total{controller="workload-rebalancer",result="requeue"} 0
0controller_runtime_reconcile_total{controller="workload-rebalancer",result="requeue_after"} 0
 61396    0     0  8595k      0 --:--:-- --:--:-- --:--:-- 9992k

But I can find workqueue_depth metric in other controller which developed by ourselves. The versions of controller-runtime are the same with karmada.

curl http://127.0.0.1:8891/metrics  |grep -i queue
controller_runtime_reconcile_total{controller="deployment",result="requeue"} 0
0 15955 controller_runtime_reconcile_total{controller="deployment",result="requeue_after"} 12
   0 15955    0     0   200k      0 --:--:-- --:--:-- --:--:--  202k
# HELP workqueue_adds_total Total number of adds handled by workqueue
# TYPE workqueue_adds_total counter
workqueue_adds_total{name="deployment"} 998
# HELP workqueue_depth Current depth of workqueue
# TYPE workqueue_depth gauge
workqueue_depth{name="deployment"} 0
# HELP workqueue_longest_running_processor_seconds How many seconds has the longest running processor for workqueue been running.
# TYPE workqueue_longest_running_processor_seconds gauge
workqueue_longest_running_processor_seconds{name="deployment"} 0
# HELP workqueue_queue_duration_seconds How long in seconds an item stays in workqueue before being requested
# TYPE workqueue_queue_duration_seconds histogram
workqueue_queue_duration_seconds_bucket{name="deployment",le="1e-08"} 0

What do you think about this question?:
why karmada controller can't find metrics workqueue_depth

workqueue_depth is necessary to check whether there is congestion in the workqueue.

Environment:

• Karmada version:
  v1.11
• Kubernetes version:
  v1.23
• Others:
  sigs.k8s.io/controller-runtime v0.18.4

[XiShanYongYe-Chang]

Hi @chaosi-zju, do you know it?

[chaosi-zju]

I have researched this issue and found that it is related to the mixing of workqueue.prometheusMetricsProvider and metrics.workqueueMetricsProvider.

If I remove the following line, all metrics can be found:

karmada/vendor/k8s.io/component-base/metrics/prometheus/workqueue/metrics.go

Line 108 in 19b32a2

workqueue.SetProvider(prometheusMetricsProvider{})

The reason for I try this is that karmada-scheduler only use metrics.workqueueMetricsProvider and its metrics runs fine.

[chaosi-zju]

I have no further ideas now. If this problem is blocking you, can you try again based on it?

[CharlesQQ]

kubernetes/kubernetes#128326