Can't use kubectl top pods/nodes on RKE2 #5805
Comments
|
Hi @XiShanYongYe-Chang Please help to confirm if this is a known issue. @KhalilSantana Is there any specific reason that you must use |
|
Hi @KhalilSantana @RainbowMango this is a known issue.
We can see that kubectl top actually sends two requests. One is to request Node resources, and the other is to request NodeMetrics. For karmada-apiserver, the kubectl command cannot obtain the nodes of all member clusters (karmadactl can do this). and NodeMetrics resources can be obtained through the karmada-metrics-adapter component. So the end result is what is shown in the picture |
|
@RainbowMango Actually I've written kubectl in the issue, but what we actually care about is the API kubectl consumes/invokes, since the actual application that manages this cluster uses the k8s python lib. Our code does basically this: config.load_kube_config(kubeconfig_path)
self.kubeapi = client.CoreV1Api()
configuration = config.load_kube_config(kubeconfig_path)
with kubernetes.client.ApiClient(configuration) as api_client:
self.api_instance = kubernetes.client.CustomObjectsApi(api_client)
k8s_nodes_metrics = self.api_instance.list_cluster_custom_object("metrics.k8s.io", "v1beta1", "nodes")Which looks like Karmada metrics adapter should answer since it registers the endpoint: However, looking more closely I've noticed that the host cluster of the karmada apiserver has it failing for that service: And the metrics-adapter logs says it can't verify some certificate:
I'm actually observing different behavior, I can see the nodes (since I've deployed the Proxy Global Resources). Thus, kubectl is able to see the nodes from member clusters: So it seems I got Karmada in some sort of invalid state currently (at least w.r.t metrics-adapter), I'll reinstall it and try again, and report back with more details sorry for the delay in the response. |
|
OK, I've had to re-install the entire host cluster for Karmada because attempting to re-install karmada (even after cleaning out the var lib and etc directories of karmada didn't work). After this, the metrics-adapter pod seems to be doing OK, but still no metrics being available. Then, when I try probing at the endpoints I see that the metrics node endpoint isn't responding, even though the metrics adapter seems to have registered itself for the metrics endpoint and looks healthy (no errors at least): $ export KUBECONFIG=~/.kube/redacted-karmada-proxy.yaml
$ kubectl get --raw '/apis/metrics.k8s.io/v1beta1/nodes'
Error from server (ServiceUnavailable): the server is currently unable to handle the request
$ kubectl get svc -A |grep metrics
karmada-system karmada-metrics-adapter ExternalName <none> karmada-metrics-adapter.karmada-system.svc.cluster.local <none> 32m
$ kubectl get apiservices.apiregistration.k8s.io|grep metrics
v1beta1.custom.metrics.k8s.io karmada-system/karmada-metrics-adapter True 32m
v1beta1.metrics.k8s.io karmada-system/karmada-metrics-adapter True 32m
v1beta2.custom.metrics.k8s.io karmada-system/karmada-metrics-adapter True 32m@XiShanYongYe-Chang Just to be clear, in your previous comment you mean that the endpoint for the NodeMetrics ( If so, what could explain what I'm observing? The NodeResources is working (AFAIK) since I have the proxy global resources, but the metrics-adapter component seems to be non-operational. |
What happened:
Hi, I'm attempting to use Karmada v1.11 on three RKE2 1.28.x clusters, one for karmada itself (cluster "vm2"), and two others for workloads (cluster "vm1" and "vm3"). Moreover, my kubeconfig is configured to use the Karmada resourse proxy for pods and nodes, just like the docs.
However, I can't seem to get
kubectl top nodesworking when using karmada (using each clusters' kubeconfig works)I've deployed the following plugins to the cluster: karmada-descheduler karmada-metrics-adapter karmada-search
Thus we see the metrics-adapter service running in the cluster:
What you expected to happen:
kubectl top nodes should be able to gets its metrics and present them to the user just like karmadactl can
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
kubectl-karmada versionorkarmadactl version): kubectl karmada version: version.Info{GitVersion:"v1.11.2", GitCommit:"80841d1cf6802c32c40af34ebe004633d74f4141", GitTreeState:"clean", BuildDate:"2024-10-31T11:00:02Z", GoVersion:"go1.22.7", Compiler:"gc", Platform:"linux/amd64"}The text was updated successfully, but these errors were encountered: