Change prometheus config from additional scrape config to probe

[dbason]

Is your feature request related to a problem? Please describe.
Our infrastructure is handled with GitOps and as such our Prometheus object is statically defined. Modifying a global object breaks this pattern.

Describe the solution you'd like
Rather than using additional scrape configs would a Probe be an option? This should handle the single endpoint nature of cluster metrics. It might also be nice to include a ServiceMonitor for scraping node metrics

Describe alternatives you've considered
We have considered bootstrapping the minio additional scrape config however this would be an anti pattern as the secret and naming is minio specific for a global object

Additional context
Add any other context or screenshots about the feature request here.

[dbason]

We have manually created a probe to confirm this works:

spec:
  authorization:
    credentials:
      key: token
      name: token-secret
    type: Bearer
  jobName: minio-tenant-test-job
  prober:
    path: /minio/v2/metrics/cluster
    scheme: https
    url: minio.test.svc.cluster.local
  targets:
    staticConfig:
      static:
      - minio.test.svc.cluster.local
  tlsConfig:
    ca:
      configMap:
        key: ca.crt
        name: kube-root-ca.crt

If this is something that people agree is a good idea I could look at creating a PR for this. It would need the ability to add arbitrary labels to the probe incase there is a probeSelector on the prometheus

[wagnst]

+1 minio tries to modify stuff in my prometheus namespace which is obviously not exclusive for minio.
See also here: https://github.com/minio/operator/blob/v6.0.4/pkg/controller/prometheus.go#L131
I've created additional scrape config for something else, now minio operator is erroring...

main-controller.go:1592] error syncing 'minio/xxx: kube-prometheus-prometheus-scrape-confg is alreay set as additional scrape config in prometheus

[pschichtel]

@dbason I wonder if this could also just be part of the tenant helm chart. is there a reason for the operator to manage this object?

[pschichtel]

ah ok I forget about the token, which the operator injects into the scrape config. The token is apparently generated once (and updated when credentials change) with a validity of a hundred years using the root username as the subject and the root password as the secret for the HS512 algorithm, so that can easily be generated manually by running jwt encode -i prometheus --exp="$(($(date +%s) + (100*365*24*60*60)))" -s <username> -A HS512 -S <password> from here.

I think solving this issue would probably also resolve #1955 or at least make it simpler to fix.