fix: duplicate custom resources

[murphd40]

What this PR does / why we need it:

Fix issue where custom resources are duplicated

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)

No change

Which issue(s) this PR fixes:
Fixes # #2044

---

Testing:

Before change

$ go run main.go --port=8080 --telemetry-port=8081 --kubeconfig=$KUBECONFIG \
--custom-resource-state-config='{"spec":{"resources":[{"groupVersionKind":{"group":"operators.coreos.com","version":"v1alpha1","kind":"ClusterServiceVersion"},"metrics":[{"name":"csv_info","help":"Cluster Service Version install status","each":{"type":"Info","info":{"labelsFromPath":{"name":["metadata","name"],"status":["status","phase"]}}}}]}]}}' \
--resources clusterserviceversions
I0514 02:05:12.826277   10899 wrapper.go:98] "Starting kube-state-metrics"
I0514 02:05:12.827409   10899 server.go:191] "Used resources" resources=[clusterserviceversions clusterserviceversions]
I0514 02:05:12.827474   10899 types.go:184] "Using all namespaces"
I0514 02:05:12.827484   10899 server.go:218] "Metric allow-denylisting" allowDenyStatus="Excluding the following lists that were on denylist: "
I0514 02:05:12.829363   10899 server.go:357] "Tested communication with server"
I0514 02:05:13.400436   10899 server.go:362] "Run with Kubernetes cluster version" major="1" minor="26" gitVersion="v1.26.3+k3s1" gitTreeState="clean" gitCommit="01ea3ff27be0b04f945179171cec5a8e11a14f7b" platform="linux/amd64"
I0514 02:05:13.400496   10899 server.go:363] "Communication with server successful"
I0514 02:05:13.402199   10899 server.go:314] "Started metrics server" metricsServerAddress="[::]:8080"
I0514 02:05:13.402238   10899 metrics_handler.go:99] "Autosharding disabled"
I0514 02:05:13.402307   10899 server.go:303] "Started kube-state-metrics self metrics server" telemetryAddress="[::]:8081"
I0514 02:05:13.402448   10899 custom_resource_metrics.go:79] "Custom resource state added metrics" familyNames=[kube_customresource_csv_info]
I0514 02:05:13.402740   10899 custom_resource_metrics.go:79] "Custom resource state added metrics" familyNames=[kube_customresource_csv_info]
I0514 02:05:13.403061   10899 builder.go:246] "Active resources" activeStoreNames="clusterserviceversions,clusterserviceversions"
I0514 02:05:13.403142   10899 server.go:73] levelinfomsgListening onaddress[::]:8080
I0514 02:05:13.403162   10899 server.go:73] levelinfomsgTLS is disabled.http2falseaddress[::]:8080
I0514 02:05:13.403197   10899 server.go:73] levelinfomsgListening onaddress[::]:8081
I0514 02:05:13.403296   10899 server.go:73] levelinfomsgTLS is disabled.http2falseaddress[::]:8081

$ curl localhost:8080/metrics
# HELP kube_customresource_csv_info Cluster Service Version install status
# TYPE kube_customresource_csv_info info
kube_customresource_csv_info{customresource_group="operators.coreos.com",customresource_kind="ClusterServiceVersion",customresource_version="v1alpha1",name="t8c-client-operator.v1.2.2",status="Succeeded"} 1
kube_customresource_csv_info{customresource_group="operators.coreos.com",customresource_kind="ClusterServiceVersion",customresource_version="v1alpha1",name="packageserver",status="Succeeded"} 1
# HELP kube_customresource_csv_info Cluster Service Version install status
# TYPE kube_customresource_csv_info info
kube_customresource_csv_info{customresource_group="operators.coreos.com",customresource_kind="ClusterServiceVersion",customresource_version="v1alpha1",name="packageserver",status="Succeeded"} 1
kube_customresource_csv_info{customresource_group="operators.coreos.com",customresource_kind="ClusterServiceVersion",customresource_version="v1alpha1",name="t8c-client-operator.v1.2.2",status="Succeeded"} 1

After Change

$ go run main.go --port=8080 --telemetry-port=8081 --kubeconfig=$KUBECONFIG \
--custom-resource-state-config='{"spec":{"resources":[{"groupVersionKind":{"group":"operators.coreos.com","version":"v1alpha1","kind":"ClusterServiceVersion"},"metrics":[{"name":"csv_info","help":"Cluster Service Version install status","each":{"type":"Info","info":{"labelsFromPath":{"name":["metadata","name"],"status":["status","phase"]}}}}]}]}}' \
--resources clusterserviceversions     
I0514 02:04:35.340743   10694 wrapper.go:98] "Starting kube-state-metrics"
I0514 02:04:35.341747   10694 server.go:191] "Used resources" resources=[clusterserviceversions]
I0514 02:04:35.341813   10694 types.go:184] "Using all namespaces"
I0514 02:04:35.341824   10694 server.go:218] "Metric allow-denylisting" allowDenyStatus="Excluding the following lists that were on denylist: "
I0514 02:04:35.343590   10694 server.go:357] "Tested communication with server"
I0514 02:04:35.931327   10694 server.go:362] "Run with Kubernetes cluster version" major="1" minor="26" gitVersion="v1.26.3+k3s1" gitTreeState="clean" gitCommit="01ea3ff27be0b04f945179171cec5a8e11a14f7b" platform="linux/amd64"
I0514 02:04:35.931434   10694 server.go:363] "Communication with server successful"
I0514 02:04:35.933402   10694 server.go:303] "Started kube-state-metrics self metrics server" telemetryAddress="[::]:8081"
I0514 02:04:35.933432   10694 metrics_handler.go:99] "Autosharding disabled"
I0514 02:04:35.933422   10694 server.go:314] "Started metrics server" metricsServerAddress="[::]:8080"
I0514 02:04:35.933617   10694 custom_resource_metrics.go:79] "Custom resource state added metrics" familyNames=[kube_customresource_csv_info]
I0514 02:04:35.933989   10694 builder.go:246] "Active resources" activeStoreNames="clusterserviceversions"
I0514 02:04:35.934409   10694 server.go:73] levelinfomsgListening onaddress[::]:8080
I0514 02:04:35.934447   10694 server.go:73] levelinfomsgTLS is disabled.http2falseaddress[::]:8080
I0514 02:04:35.934422   10694 server.go:73] levelinfomsgListening onaddress[::]:8081
I0514 02:04:35.934512   10694 server.go:73] levelinfomsgTLS is disabled.http2falseaddress[::]:8081

curl localhost:8080/metrics
# HELP kube_customresource_csv_info Cluster Service Version install status
# TYPE kube_customresource_csv_info info
kube_customresource_csv_info{customresource_group="operators.coreos.com",customresource_kind="ClusterServiceVersion",customresource_version="v1alpha1",name="packageserver",status="Succeeded"} 1
kube_customresource_csv_info{customresource_group="operators.coreos.com",customresource_kind="ClusterServiceVersion",customresource_version="v1alpha1",name="t8c-client-operator.v1.2.2",status="Succeeded"} 1

[rexagod]

Duplication can be suppressed by specifying --custom-resources-only flag for the native metrics, which will force KSM to generate metrics only from the CRS configuration, and which has been the supported behaviour till now for when the same (internal) object is specified under both flags.

[rexagod]

One case that satisfies the default case is when a custom-resource-state-only flag is not specified, and the resources flag is non-nil. In this case if we also specify custom-resource-state-config flag (with a CRS configuration), we'd get duplicate metrics if the entries in both overlap (for a common native resource).

resources in the code snippet above are the extracted resources we obtain from the CRS configuration, and removing them here would result in removing the current ability for KSM to generate and serve native and custom metrics (for different objects), for instance, kube-state-metrics --kubeconfig=$KUBECONFIG --custom-resource-state-config-file=... --resources=....

IIUC with this change, the CRS resources extracted will be ignored entirely (for the aforementioned case), which will be a breaking change, since that's the expected behaviour ever since CRS was introduced.

[murphd40]

Hi @rexagod . In that case would it be better to add logic to remove duplicates from the resources list?

[dgrisonnet]

Holding since I am not a fan of using custom-state-metrics as a way to hack around kube-state-metrics. I am moving the discussion back to the issue.

/hold

[dgrisonnet]

/triage accepted
/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rexagod]

Can we use sets here instead?

[ikarlashov]

@murphd40 any progress? :)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: murphd40
Once this PR has been reviewed and has the lgtm label, please ask for approval from dgrisonnet. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rexagod]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[rexagod]

/remove-lifecycle rotten

[murphd40]

Hi @ikarlashov @rexagod. Apologies for the delay. I have updated the PR as suggested.

However, when I run

go run main.go --port=8080 --telemetry-port=8081 --kubeconfig=$KUBECONFIG \
--custom-resource-state-config='{"spec":{"resources":[{"groupVersionKind":{"group":"operators.coreos.com","version":"v1alpha1","kind":"ClusterServiceVersion"},"metrics":[{"name":"csv_info","help":"Cluster Service Version install status","each":{"type":"Info","info":{"labelsFromPath":{"name":["metadata","name"],"status":["status","phase"]}}}}]}]}}' \
--resources clusterserviceversions

It fails:

go run main.go --port=8080 --telemetry-port=8081 --kubeconfig=$KUBECONFIG \
--custom-resource-state-config='{"spec":{"resources":[{"groupVersionKind":{"group":"operators.coreos.com","version":"v1alpha1","kind":"ClusterServiceVersion"},"metrics":[{"name":"csv_info","help":"Cluster Service Version install status","each":{"type":"Info","info":{"labelsFromPath":{"name":["metadata","name"],"status":["status","phase"]}}}}]}]}}' \
--resources clusterserviceversions
I0711 16:23:01.705060   97371 wrapper.go:120] "Starting kube-state-metrics"
I0711 16:23:01.706299   97371 server.go:203] "Used resources" resources={"clusterserviceversions":{}}
E0711 16:23:01.706366   97371 wrapper.go:40] "Failed to run kube-state-metrics" err="failed to set up resources: resource clusterserviceversions does not exist. Available resources: networkpolicies,validatingwebhookconfigurations,horizontalpodautoscalers,ingresses,jobs,cronjobs,namespaces,rolebindings,replicasets,resourcequotas,storageclasses,clusterrolebindings,endpointslices,poddisruptionbudgets,endpoints,serviceaccounts,services,persistentvolumes,secrets,volumeattachments,leases,mutatingwebhookconfigurations,persistentvolumeclaims,ingressclasses,pods,replicationcontrollers,statefulsets,deployments,limitranges,nodes,daemonsets,roles,certificatesigningrequests,clusterroles,configmaps"
exit status 1

It is failing on main as well so it is not related to my code. Am I missing something?

It is failing at this point:

https://github.com/kubernetes/kube-state-metrics/blob/85d1423b5b896f4fbce8394afed20cfe17371251/pkg/app/server.go#L205C12-L207

	if err := storeBuilder.WithEnabledResources(resources); err != nil {
		return fmt.Errorf("failed to set up resources: %v", err)
	}

It seems that the custom resource config is no longer added to storeBuilder before this check so the store builder does not recognize it...

[mrueg]

Can you add a unit test that produces duplicate CRs as well?

[ikarlashov]

@murphd40 Please :)

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sebastiangaiser]

Is this PR still needed as #2502 got merged?