3 Alerts firing from the Start - why?

[realulim]

I've set up kube-prometheus on top of a pristine K3s cluster and three alerts started firing from the start (see screenshot 1).

I realise there has been a discussion about this at https://groups.google.com/g/prometheus-users/c/_aI-HySJ-xM/m/kqrL1FYVCQAJ but my case seems to be different, because no targets are down (see screenshot 2) or rather I cannot see any targets for the services mentioned in the alerts (KubeControllerManager, KubeProxy, KubeScheduler).

How can I check whether this is a scraping error (nothing was scraped) or a Prom-QL error (scraping data is there but the query doesn't select it)?

[paulfantom]

It looks like you are not gathering metrics from those services. Which is a common thing for managed clusters (like EKS). You have 2 courses of action:

1. Remove those alerts (if you use managed kubernetes). This can be done by specifying platform option as described in https://github.com/prometheus-operator/kube-prometheus/tree/main/jsonnet/kube-prometheus/platforms
2. Configure services to expose metrics (if you are running kubernetes by yourself)

Relates to #1004 and #717

[realulim]

I am running my own K3s cluster. So I gather kube-prometheus is configured to scrape from endpoints that do not exist in K3s. Is there a way to see where Prometheus expects the scraping endpoint to be for those alerts? Or can I configure a different scraping endpoint?

Also, for removing the alerts, how would I do this exactly? The docs refer to generating jsonnet files, but how do I apply those generated files to the cluster? I am using the Helm chart for kube-prometheus.

[paulfantom]

This project is about jsonnet-based deployment. The one where jsonnet is used to generate YAML/JSON manifests which are later applied to the cluster. We don't ship nor support helm charts.

If you are interested in how all this can work with k3s, feel free to take a look at my personal cluster at https://github.com/thaum-xyz/ankhmorpork/tree/master/apps/monitoring

[realulim]

Ok, I have found an answer to some of my questions:

The rules are accessible via kubectl get prometheusrule -n mynamespace and can be edited there and I suppose also deleted. It is necessary to reload AlertManager afterwards, so it re-reads the rules. This can be done via:

curl -X POST $(echo "http://$(kubectl get svc prometheus-kube-prometheus-alertmanager -n prometheus|grep ClusterIP|tr -s " "|cut -f3 -d " "):9093/alertmanager/-/reload")

But I'd rather find the place where the scraping targets are defined, so I can satisfy the rules.

[paulfantom]

To have scrape configuration you need to have 2 things:

1. ServiceMonitor - which is essentially an information about scrape configuration passed to prometheus
2. Service - which is exposing actual /metrics endpoints and can be scraped by prometheus

Since you are using k3s, there is no dedicated components for controller manager or scheduler as all of them are bundled into a single application (k3s binary). For this reason there is no point in keeping those ServiceMontors and those 3 alerts can also be disabled/removed.

[realulim]

Hm, the ServiceMonitor prometheus-kube-prometheus-kube-scheduler refers to a service with the selector app=kube-prometheus-stack-kube-scheduler. A service with this selector exists in the kube-system namespace. But this service has no IP address and is thus not reachable for the scraper.

[realulim]

@paulfantom BTW, thanks a lot for your explanations, I am learning a lot here. So it seems the real problem is that K3s does not expose those metrics and in order to change that, I'd have to re-configure and re-build the K3s binary.

[paulfantom]

I think you are missing a point here. There is no need to re-build k3s, just to reconfigure kube-prometheus.

In short (specific to k3s ONLY):

• k3s bundles all control-plane metrics in one application
• with ServiceMonitor responsible for scraping kubelet, you already have metrics from API server, controller manager, kubelet, and scheduler
• those 3 alerts are meaningless in k3s case and SHOULD be removed

[realulim]

Ok, gotcha, many thanks! The good part of all the head-scratching is that I now have a pretty good idea what needs to be done to deploy metrics and alerts for my own application.