Best Practices for Integrating Grafana Operator with kube-prometheus-stack for GitOps

[jeffcollaboro]

Hello,

I am using the kube-prometheus-stack for monitoring in my Kubernetes environment, which includes Prometheus Operator but find it strange that the stack (seemingly) does not natively support the Grafana Operator. This operator would greatly simplify gitops/argocd dashboard rollouts using the custom CRD's provided, along with improving the upgrade experience.

My questions are:

1. Does the kube-prometheus-stack community have any recommendations or best practices for integrating the Grafana Operator with an alternate Grafana implementation from the one natively provided by the stack?

2. Given that the stack supports Prometheus Operator, what is the rationale behind not including Grafana Operator, and are there considerations I should be aware of when attempting this integration?

3. Are there recommended approaches or existing examples for setting up Grafana using the Grafana Operator alongside kube-prometheus-stack, ensuring seamless GitOps provisioning of dashboards and data sources?

4. What alternative strategies are people using to rollout dashboards using gitops principles (Configmaps?).

Any guidance or shared experiences on this integration would be invaluable.

Thank you!

[filip-franek]

Regarding the alternative strategies, how about:

1. patching the current deployment?

Here is an example of how I patched my Grafana deployment to mount the Grafana data location to my persistent volume. I have it applied over Flux. It assumes the Grafana Persistent Volume Claim (PVC) is already created.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-prom-stack-grafana
  namespace: observability
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: kube-prom-stack
      app.kubernetes.io/name: grafana
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: kube-prom-stack
        app.kubernetes.io/name: grafana
    spec:
      containers:
      - name: grafana
        volumeMounts:
        - mountPath: "/var/lib/grafana"
          name: <storage-class>
      volumes:
      - name: <storage-class>
        persistentVolumeClaim:
          claimName: <grafana-pvc>

2. Upgrading it with helm? as mentioned in this comment?

An alternative would be to update the helm installation directly. The command should look like this:

helm upgrade -n observability kube-prom-stack kube-prometheus-stack \
  --repo https://prometheus-community.github.io/helm-charts \
  --reuse-values -f new_values.yaml

Keep in mind that the intent of the addon is to simply make it easy to deploy the stack, not handle lifecycle and further operations.