docs/prometheus/

[giscus[bot]]

docs/prometheus/

How to deploy kuberentes cluster monitoring solution based on Prometheus. Installation based on Prometheus Operator using kube-prometheus-stack project.

https://picluster.ricsanfre.com/docs/prometheus/

[i5Js]

I have had to uncheck the client authentication in the Client configuration, otherwise I have had an error after login: code_to_token_error invalid client credentials

[i5Js]

FIXED! Finally! I found it checking the env variables inside the grafana pod:

kube-prometheus-stack-grafana-8f748478-5z7jr:/usr/share/grafana$ env | grep GF_
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<secret_uncoded>
GF_PATHS_HOME=/usr/share/grafana
GF_PATHS_PROVISIONING=/etc/grafana/provisioning
GF_SECURITY_ADMIN_PASSWORD=prom-operator
GF_SECURITY_ADMIN_USER=admin
GF_PATHS_DATA=/var/lib/grafana/
GF_PATHS_LOGS=/var/log/grafana
GF_PATHS_PLUGINS=/var/lib/grafana/plugins
GF_PATHS_CONFIG=/etc/grafana/grafana.ini

Then I have realized there was one miss: GF_AUTH_GENERIC_OAUTH_CLIENT_ID

And that explains why always had userId="null". So I modified the secret:

apiVersion: v1
data:
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: <clientsecret_base64>
  GF_AUTH_GENERIC_OAUTH_CLIENT_ID: <clientid_base64>
kind: Secret
metadata:
  creationTimestamp: null
  name: grafana-env-secret
  namespace: monitoring

And restarted the pod: kubectl -n monitoring rollout restart deployment kube-prometheus-stack-grafana

And finally, checked the envs:

kube-prometheus-stack-grafana-8f748478-5z7jr:/usr/share/grafana$ env | grep GF_
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<secret_uncoded>
**GF_AUTH_GENERIC_OAUTH_CLIENT_ID**=grafana-oauth
GF_PATHS_HOME=/usr/share/grafana
GF_PATHS_PROVISIONING=/etc/grafana/provisioning
GF_SECURITY_ADMIN_PASSWORD=prom-operator
GF_SECURITY_ADMIN_USER=admin
GF_PATHS_DATA=/var/lib/grafana/
GF_PATHS_LOGS=/var/log/grafana
GF_PATHS_PLUGINS=/var/lib/grafana/plugins
GF_PATHS_CONFIG=/etc/grafana/grafana.ini

And boom! working:

logger=context userId=0 orgId=0 uname= t=2024-04-19T12:40:28.158783866Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.244.0.0 time_ms=0 duration=239.164µs size=345 referer= handler=/login/:name status_source=server
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259028574Z level=debug msg="Getting user info"
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259125406Z level=debug msg="Extracting user info from OAuth token"
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259833854Z level=debug msg="Received id_token" raw_json="{\"exp\":1713530731,\"iat\":1713530431,\"auth_time\":1713529886,\"jti\":\"e74d1c01-a1ac-4fd8-ac49-0362be7c2021\",\"iss\":\"https://sso.home.domain.com/realms/home\",\"aud\":\"grafana-oauth\",\"sub\":\"d67a175d-d4ae-4d62-9070-09aa309b2ee0\",\"typ\":\"ID\",\"azp\":\"grafana-oauth\",\"session_state\":\"8dfd088e-7055-4c82-aa87-e1bc06868814\",\"at_hash\":\"mcxdogCSuT-NeDvnEoeExw\",\"acr\":\"0\",\"sid\":\"8dfd088e-7055-4c82-aa87-e1bc06868814\",\"email_verified\":true,\"roles\":[\"editor\",\"viewer\",\"offline_access\",\"admin\",\"default-roles-home.javisalas.com\",\"uma_authorization\"],\"name\":\"Grafana Admin\",\"preferred_username\":\"grafanadmin\",\"given_name\":\"Grafana\",\"family_name\":\"Admin\",\"email\":\"[email protected]\"}" data="Name: Grafana Admin, Displayname: , Login: , Username: , Email: [email protected], Upn: , Attributes: map[]"

[ricsanfre]

Hi Javi,

In my Prometheus packaged application for ArgoCD, I am specifying both environment variables GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET and GF_AUTH_GENERIC_OAUTH_CLIENT_ID. That way I can configure not only the password but the client_id from a external secret defined in my Secret Manager. See External Secret definition https://github.com/ricsanfre/pi-cluster/blob/master/argocd/system/monitoring/templates/grafana-env-externalsecret.yaml

When I did manual installation tests specifying GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET as environment variable, adding GF_AUTH_GENERIC_OAUTH_CLIENT_ID was not a requirement,. client_id could be specified in the grafana.ini section within the values.yaml.

grafana:
  grafana.ini:
      # SSO configuration
      auth.generic_oauth:
        enabled: true
        name: Keycloak-OAuth
        allow_sign_up: true
        client_id: grafana

According to Grafana's documentation the "GF_" environment variables are only overriding specific configuration parameters you have in the grafana.ini file. So it should not be mandatory to add both environment variables if the client_id is part of the grafana.ini.

Is that not working? Is it needed to a both as Grafana's environment variables in the external secret?
If it is mandatory to add both variables, I will modify the documentation.

Thanks
Regards
Ricardo

[i5Js]

Hello Ricardo,

I made a new test: I have removed the client_id from the values files, and it's still working, so, it's not needed.

Happy to help.

Regards

Javier.

[ricsanfre]

Hi Javi,

What I wanted to test is the opposite. Keeping client_id in the values file and removing GF_AUTH_GENERIC_OAUTH_CLIENT_ID from the secret. If that was your initial set-up, and it was not working, I will modify the documentation.

Regards
Ricardo

[i5Js]

Hi Ricardo,

That was my initial setup, and it was not working. Perhaps, they have changed something and it's something different than client_id, but trust me, it was not working. Two weeks on this!! :)

Cheers

Javi

[i5Js]

The Nginx Grafana dashboard doesn't work "out of the box", we need to adjust nginx settings and promethus settings described here:

https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/monitoring.md

[ricsanfre]

Hi @i5Js,

I am not finding any issue with my nginx grafana dashboard.
Prometheus ServiceMonitor resource is created by default by nginx helm chart using this configuration:

controller:
  # Enabling Prometheus metrics and Service Monitoring
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true

My nginx dashboard works fine without any additional prometheus configuration applied to nginx

Regards
Ricardo

[benbourner]

The Ingress for the grafana dashboard rewrites the URL to remove /grafana, but grafana is configured to be served from /grafana, so the ingress doesn't work. Commenting out these lines in the grafana ingress solves it:

    # Rewrite target
    # nginx.ingress.kubernetes.io/use-regex: "true"
    # nginx.ingress.kubernetes.io/rewrite-target: /$1

Also no further need for the regex group match on the path:

    path: /grafana

[ricsanfre]

Hi @benbourner,

In my cluster, I have the configuration as it is stated in the guide and everything works fine. Grafana endpoint is reachable at https://monitoring.picluster.ricsanfre.com/grafana/ and grafana metrics also are available at /metrics endpoint.

Have you check whether you have this grafana configuration to make it can work behind a proxy.

grafana:
  # Configuring grafana to run behind a proxy under a subpath
  grafana.ini:
    server:
      domain: monitoring.picluster.ricsanfre.com
      root_url: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
      # Running Grafana behind proxy rewrite path
      # https://grafana.com/tutorials/run-grafana-behind-a-proxy/
      serve_from_sub_path: true

My working ingress resource is

# Ingress Grafana
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-grafana
  namespace: monitoring
  annotations:
    # Linkerd configuration. Configure Service as Upstream
    nginx.ingress.kubernetes.io/service-upstream: "true"
    # Rewrite target
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    # Enable cert-manager to create automatically the SSL certificate and store in Secret
    cert-manager.io/cluster-issuer: letsencrypt-issuer
    cert-manager.io/common-name: monitoring.picluster.ricsanfre.com

spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - monitoring.picluster.ricsanfre.com
      secretName: monitoring-tls
  rules:
    - host: monitoring.picluster.ricsanfre.com
      http:
        paths:
          - path: /grafana/(.*)
            pathType: Prefix
            backend:
              service:
                name: kube-prometheus-stack-grafana
                port:
                  number: 80

Please, let me know if you have followed all the steps in the guide and if you still have issues.

Regards
Ricardo

[sbatururimi]

Hi @ricsanfre and @benbourner, looks like I faced the same problem despite following the guideline here.
Adding the path to service monitor helped to fix the problem in prometheus to check the Grafana's status:

serviceMonitor:
    path: /grafana/metrics
    labels:
      release: kube-prometheus-stack
    relabelings:

For grafana ingress, only adjusting (commenting the regex & rewrite + adding service-upstream) as follow helped:

annotations:
    # Linkerd configuration. Configure Service as Upstream
    nginx.ingress.kubernetes.io/service-upstream: "true"
    # # Rewrite target
    # nginx.ingress.kubernetes.io/use-regex: "true"
    # nginx.ingress.kubernetes.io/rewrite-target: /$1
...
rules:
    - host: monitoring.kube.cluster1.renty.life
      http:
        paths:
          # - path: /grafana/(.*)
          - path: /grafana

What could be wrong that these changes was necessary for me and @benbourner?

[ricsanfre]

Hi @sbatururimi, I have detected an inconsistency between the documentation and my current grafana/prometheus working configuration.

Since rewrite rules are configured within NGINX (reverse-proxy), grafana's server_from_sub_path has to be set to false

grafana:
  # Configuring grafana to run behind a proxy under a subpath
  grafana.ini:
    server:
      domain: monitoring.picluster.ricsanfre.com
      root_url: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
      # Running Grafana behind proxy rewrite path
      # https://grafana.com/tutorials/run-grafana-behind-a-proxy/
      serve_from_sub_path: false

With this configuration, Ingress resources should work.

I have updated documentation to fix this inconsistency.

Let me know if that works for you.

Regards
Ricardo

[sbatururimi]

In order to fix the sso issue login to grafana Invalid parameter: redirect_uri, I added a few modifications that looks like to be important:

1. in the values.yaml

root_url: "https://%(domain)s/grafana/"

2. in the ingress yaml I added an additional rule (I have setup keycloak for prometheus) otherwise using <domain>/grafana was redirecting to the next rule, which was the keycloak oauth-2 proxy for prometheus in my case:

http:
  paths:
    # Match `/grafana` (no trailing slash)
    - path: /grafana
      pathType: Exact
      backend:
        service:
          name: kube-prometheus-stack-grafana
          port:
            number: 80
    # Match `/grafana/` and subpaths
    - path: /grafana/(.*)
      # - path: /grafana
      pathType: ImplementationSpecific
      backend:
        service:
          name: kube-prometheus-stack-grafana
          port:
            number: 80

the problem was that when clicking the sso login with keycloak (Sign in with Keycloak-OAuth) the redirect link used http and the port instead of https and no port like follow: /protocol/openid-connect/auth?client_id=grafana&redirect_uri=http://:3000/grafana/login/generic_oauth`

[ricsanfre]

1. That is the proper value for root_url parameter to be used when integrating with Keycloak. It was already documented in the SSO section configuration: https://picluster.ricsanfre.com/docs/prometheus/#grafana-sso-configuration

2. Related to additional NGINX rule it is only needed if you do not specify the traiiling '/' character, which I always do :-).. I will test your additional rule and add it to the configuration

By the way you can find all my latest working configuration (Flux CD packaged applications) in my repo https://github.com/ricsanfre/pi-cluster. I try to keep up to date the documentation with my latest tested configuration since the new versions of the helm charts introduces modifications but sometimes I forgot to copy all the changes.
Any PR containing improvements to both the documentation and the original configuration is welcome.

Thanks

[sbatururimi]

Hi, Ricardo. Wonderful documentation in general and one of the best. In the section What has been deployed by kube-stack? is it the example of objects deployed during the installation or those are recommended objects to be recreated to replace those one installed by the operator?

[ricsanfre]

Thanks @sbatururimi,
That section contains sample Prometheus and AlertManager objects created automatically by kube-prom-stack helm chart that are used by the operator. Samples objects are in the doc only to explain how helm chart works. The actual objects generated will be different depending on the values provided to the helm chart and the helm chart version used.

[sbatururimi]

Thank you for the explanation!

[sbatururimi]

Looks like this is not required in newest version of grafana and part of it by default?

# List of grafana plugins to be installed
  plugins:
    - grafana-piechart-panel

[ricsanfre]

Thanks @sbatururimi for the info.
It seems that it is already included since Grafana 8.0
I will update the documentation.

[sbatururimi]

Looks like the community dashboard for ElasticSearch has been deleted.
I found it in this old commit.
But I still get some graphics broken and unworking requiring a lot of search of available metrics.
@ricsanfre could you update with that please?
Maybe most of metrics are not available and that's why or broken...

That would help lot.

[sbatururimi]

If we add new datasources or databases how to have them back if the pod is grafana recreated?
In docker, generally with mounted volumes for /dashboards, /datasources. But how about this configuration here?

[sbatururimi]

I answered to myself. To allow the persistence of datasources/dashboards added from UI, then add this to the values.yaml

grafana:
  persistence:
    enabled: true
    type: sts
    storageClassName: "longhorn"
    accessModes:
      - ReadWriteOnce
    size: 10Gi

  # Configuring grafana to run behind a proxy under a subpath
  grafana.ini:
  ....