feat(thanos): adding initial plugin (#183)

* feat(thanos): adding initial plugin currently the plugin deploys the Query only. Compactor is already in but not yet tested/activated * Automatic application of license header --------- Co-authored-by: License Bot <[email protected]> Co-authored-by: Richard Tief <[email protected]>
cloudoperators · Jul 25, 2024 · 1efb196 · 1efb196
1 parent 65c4a3c
commit 1efb196
Show file tree

Hide file tree

Showing 11 changed files with 593 additions and 0 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -31,4 +31,5 @@ ct.yaml                   @cloudoperators/greenhouse-backend
 /opentelemetry-operator   @cloudoperators/greenhouse-observability
 /service-proxy/           @cloudoperators/greenhouse-backend @databus23 
 /teams2slack/             @cloudoperators/greenhouse-backend @voigts 
+/thanos/                  @cloudoperators/greenhouse-observability
 ui/                       @cloudoperators/greenhouse-frontend
diff --git a/thanos/README.md b/thanos/README.md
@@ -0,0 +1,115 @@
+---
+title: Thanos
+---
+
+# Information
+
+This plugin deploys the following Thanos components:
+
+* Query
+<!--* Query Frontend-->
+* Compact
+<!--* (Ruler)-->
+<!--* Storegateway-->
+
+Requirements (detailed steps below):
+* ready to use credentials for a [compatible object store](https://thanos.io/tip/thanos/storage.md/)
+* thanos-sidecar enabled in Prometheus (usually with [Prometheus Operator](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1.ThanosSpec))
+
+# Owner
+
+1. Tommy Sauer (@viennaa) 
+2. Richard Tief (@richardtief) 
+3. Martin Vossen (@artherd42) 
+
+# Configuration
+
+## Object Store preparation
+
+To run Thanos, you need object storage credentials. Get the credentials of your provider and add them to a Kubernetes Secret. The [Thanos documentation](https://thanos.io/tip/thanos/storage.md/) provides a great overview on the different supported store types.
+
+Usually this looks somewhat like this
+
+```yaml
+type: $STORAGE_TYPE
+config:
+    user:
+    password:
+    domain:
+    ...
+```
+
+If you've got everything in a file, deploy it in your remote cluster in the namespace, where Prometheus and Thanos will be.
+
+**Important:** `$THANOS_PLUGIN_NAME` is needed later for the respective Thanos plugin and they must not be different!
+
+```
+kubectl create secret generic $THANOS_PLUGIN_NAME-metrics-objectstore --from-file=thanos.yaml=/path/to/your/file
+```
+
+
+## kube-monitoring plugin enablement 
+
+Prometheus in kube-monitoring needs to be altered to have a sidecar and ship metrics to the new object store too. You have to provide the Secret you've just created to the (most likely already existing) kube-monitoring plugin. Add this:
+
+```yaml
+spec:
+    optionValues:
+      - name: kubeMonitoring.prometheus.prometheusSpec.thanos.objectStorageConfig.existingSecret.key
+        value: thanos.yaml
+      - name: kubeMonitoring.prometheus.prometheusSpec.thanos.objectStorageConfig.existingSecret.name
+        value: $THANOS_PLUGIN_NAME-metrics-objectstore
+```
+
+## Thanos Querier
+
+This is the real deal now: Define your Thanos query by creating a plugin.
+
+**NOTE1:** `$THANOS_PLUGIN_NAME` needs to be consistent with your secret created earlier.
+**NOTE2:** The `releaseNamespace` needs to be the same as to where kube-monitoring resides. By default this is kube-monitoring. 
+
+
+```yaml
+apiVersion: greenhouse.sap/v1alpha1
+kind: Plugin
+metadata:
+  name: $YOUR_CLUSTER_NAME
+spec:
+  pluginDefinition: thanos
+  disabled: false
+  clusterName: $YOUR_CLUSTER_NAME 
+  releaseNamespace: kube-monitoring
+```
+
+## [OPTIONAL] Handling your Prometheus Store
+By default Thanos Query would check for a service `prometheus-operated` in the same namespace with this GRPC port to be available `10901`. The cli option looks like this and is configured in the PluginDefinition:
+
+`--store=prometheus-operated:10901`
+
+This would be fine, unless you've got more than one prometheus running in this namespace. Then you would need to add all prometheus stores dedicatedly:
+
+```yaml
+spec:
+  optionsValues:
+  - name: thanos.query.stores
+    value:
+      - kube-monitoring-1-prometheus:10901 
+      - kube-monitoring-2-prometheus:10901 
+```
+
+
+# Operations
+
+## Thanos Compactor
+
+If you deploy the plugin with the default values, Thanos compactor will be shipped too and use the same secret (`$THANOS_PLUGIN_NAME-metrics-objectstore`) to retrieve, compact and push back timeseries.
+
+It will use a 100Gi PVC to not extensively occupy ephermeral storage. Depending on the amount of metrics this might be not enought and bigger volumes are needed. It is always safe to delete the compactor volume and increase it as needed. 
+
+The object storage costs will be heavily impacted on how granular timeseries are being stored (reference [Downsampling](https://thanos.io/tip/components/compact.md/#downsampling)). These are the pre-configured defaults, you can change them as needed:
+
+```
+raw: 777600s (90d)
+5m: 777600s (90d)
+1h: 157680000 (5y)
+```
diff --git a/thanos/charts/Chart.yaml b/thanos/charts/Chart.yaml
@@ -0,0 +1,20 @@
+# SPDX-FileCopyrightText: 2024 SAP SE or an SAP affiliate company and Greenhouse contributors
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: v2
+description: Base chart for thanos monitoring deployments
+icon: https://raw.githubusercontent.com/thanos-io/thanos/main/docs/img/Thanos-logo_fullmedium.png 
+type: application
+maintainers:
+  - name: viennaa
+  - name: richardtief
+name: thanos 
+sources:
+  - https://github.com/cloudoperators/greenhouse-extensions
+version: 0.0.8
+# thanos-release
+appVersion: v0.35.0
+keywords:
+  - thanos
+  - storage
+  - metrics
diff --git a/thanos/charts/templates/_helpers.tpl b/thanos/charts/templates/_helpers.tpl
@@ -0,0 +1,21 @@
+{{/*
+Generic plugin name
+*/}}
+{{- define "release.name" -}}
+{{- printf "%s" $.Release.Name | trunc 50 | trimSuffix "-" -}}
+{{- end}}
+
+{{/* Generate plugin specific labels */}}
+{{- define "plugin.labels" -}}
+plugindefinition: thanos 
+plugin: {{ $.Release.Name }}
+{{- if .Values.global.commonLabels }}
+{{ tpl (toYaml .Values.global.commonLabels) . }}
+{{- end }}
+{{- end }}
+
+{{/* Base labels to be glued on everything */}}
+{{- define "thanos.labels" -}}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+release: {{ $.Release.Name | quote }}
+{{- end }}
diff --git a/thanos/charts/templates/compactor-svc.yaml b/thanos/charts/templates/compactor-svc.yaml
@@ -0,0 +1,26 @@
+# SPDX-FileCopyrightText: 2024 SAP SE or an SAP affiliate company and Greenhouse contributors
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: v1
+kind: Service
+metadata:
+  {{- if .Values.thanos.compactor.annotations }}
+  annotations:
+    {{ toYaml .Values.thanos.compactor.annotations | nindent 8 }}
+  {{- end }}
+  labels:
+    {{- include "plugin.labels" . | nindent 4 }}
+    {{- include "thanos.labels" . | nindent 4 }}
+    {{- if .Values.thanos.compactor.serviceLabels }}
+    {{ toYaml .Values.thanos.compactor.serviceLabels | nindent 4 }}
+    {{- end }}
+  name: {{ include "release.name" . }}-compactor
+spec:
+  ports:
+  - name: http
+    port: 10902
+    protocol: TCP
+    targetPort: 10902
+  selector:
+    app.kubernetes.io/managed-by: {{ include "release.name" . }}
+    app.kubernetes.io/name: compactor
diff --git a/thanos/charts/templates/compactor.yaml b/thanos/charts/templates/compactor.yaml
@@ -0,0 +1,93 @@
+# SPDX-FileCopyrightText: 2024 SAP SE or an SAP affiliate company and Greenhouse contributors
+# SPDX-License-Identifier: Apache-2.0
+
+{{ if .Values.thanos.compactor.enabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    {{- include "plugin.labels" . | nindent 4 }}
+    {{- include "thanos.labels" . | nindent 4 }}
+    {{- if .Values.thanos.compactor.labels }}
+    {{ toYaml .Values.thanos.compactor.labels | nindent 4 }}
+    {{- end }}
+  name: {{ include "release.name" . }}-compactor
+spec:
+  progressDeadlineSeconds: 600
+  replicas: 1
+  revisionHistoryLimit: 10
+  selector:
+    matchLabels:
+      app.kubernetes.io/managed-by: {{ include "release.name" . }} 
+      app.kubernetes.io/name: compactor
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      {{- if .Values.thanos.compactor.annotations }}
+      annotations:
+        {{ toYaml .Values.thanos.compactor.annotations | nindent 8 }}
+      {{- end }}
+      labels:
+        app.kubernetes.io/managed-by: {{ include "release.name" . }} 
+        app.kubernetes.io/name: compactor
+        {{- if .Values.thanos.compactor.containerLabels }}
+        {{ toYaml .Values.thanos.compactor.containerLabels | nindent 8 }}
+        {{- end }}
+      name: {{ include "release.name" . }}-compactor
+    spec:
+      containers:
+      - args:
+        - compact
+        - --log.level={{ default "info" .Values.thanos.compactor.logLevel }}
+        - --http-address={{ default "0.0.0.0:10902" .Values.thanos.httpAddress }}
+        - --http-grace-period={{ default "120s" .Values.thanos.compactor.httpGracePeriod }}
+        - --data-dir=/data
+        - --objstore.config-file=/etc/config/thanos.yaml
+        - --consistency-delay={{ default "1800s" .Values.thanos.compactor.consistencyDelay }}
+        - --retention.resolution-raw={{ default "7776000s" .Values.thanos.compactor.retentionResolutionRaw }}
+        - --retention.resolution-5m={{ default "7776000s" .Values.thanos.compactor.retentionResolution5m }}
+        - --retention.resolution-1h={{ default "157680000s" .Values.thanos.compactor.retentionResolution1h }}
+        - --compact.concurrency={{ default (int 1) .Values.thanos.compactor.compact.concurrency }}
+        - --compact.cleanup-interval={{ default "1800s" .Values.thanos.compactor.compact.cleanupInterval }}
+        - --wait
+        - --wait-interval={{ default "900s" .Values.thanos.compactor.compact.cleanupInterval }}
+        {{- range .Values.thanos.compactor.additionalArgs }}
+        - {{ . }}
+        {{- end }}
+        image: "{{ .Values.thanos.image.repository }}:{{ .Values.thanos.image.tag | default .Chart.AppVersion }}"
+        imagePullPolicy: {{ default "IfNotPresent" .Values.thanos.image.pullPolicy }}
+        name: compactor
+        ports:
+        - containerPort: 10902
+          name: http
+          protocol: TCP
+        volumeMounts:
+        - mountPath: /etc/config/
+          name: objectstore-secret
+          readOnly: true
+        - mountPath: /data
+          name: data-volume
+        securityContext:
+          runAsUser: 1000
+          runAsGroup: 3000
+          fsGroup: 2000
+      initContainers:
+        - name: init-permissions
+          image: busybox
+          command: ['sh', '-c', 'chown 1000:3000 /data && chmod 750 /data']
+          volumeMounts:
+          - mountPath: /data
+            name: data-volume
+      dnsPolicy: ClusterFirst
+      restartPolicy: Always
+      terminationGracePeriodSeconds: 30
+      volumes:
+      - name: objectstore-secret
+        secret:
+          defaultMode: 420
+          secretName: {{ include "release.name" . }}-metrics-objectstore 
+      - name: data-volume
+        persistentVolumeClaim:
+          claimName: {{ include "release.name" . }}-compactor
+{{ end }}
diff --git a/thanos/charts/templates/pvc.yaml b/thanos/charts/templates/pvc.yaml
@@ -0,0 +1,23 @@
+# SPDX-FileCopyrightText: 2024 SAP SE or an SAP affiliate company and Greenhouse contributors
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  {{- if .Values.thanos.compactor.annotations }}
+  annotations:
+    {{ toYaml .Values.thanos.compactor.annotations | nindent 8 }}
+  {{- end }}
+  labels:
+    {{- include "plugin.labels" . | nindent 4 }}
+    {{- include "thanos.labels" . | nindent 4 }}
+    {{- if .Values.thanos.compactor.volume.labels }}
+    {{ toYaml .Values.thanos.compactor.volume.labels | nindent 4 }}
+    {{- end }}
+  name: {{ include "release.name" . }}-compactor
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: {{ default "100Gi" .Values.thanos.compactor.volume.size }}
diff --git a/thanos/charts/templates/query-svc.yaml b/thanos/charts/templates/query-svc.yaml
@@ -0,0 +1,30 @@
+# SPDX-FileCopyrightText: 2024 SAP SE or an SAP affiliate company and Greenhouse contributors
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: v1
+kind: Service
+metadata:
+  {{- if .Values.thanos.compactor.annotations }}
+  annotations:
+    {{ toYaml .Values.thanos.compactor.annotations | nindent 8 }}
+  {{- end }}
+  labels:
+    {{- include "plugin.labels" . | nindent 4 }}
+    {{- include "thanos.labels" . | nindent 4 }}
+    {{- if .Values.thanos.query.serviceLabels }}
+    {{ toYaml .Values.thanos.query.serviceLabels | nindent 4 }}
+    {{- end }}
+  name: {{ include "release.name" . }}-query
+spec:
+  ports:
+  - name: grpc
+    port: 10901
+    protocol: TCP
+    targetPort: grpc
+  - name: http
+    port: 10902
+    protocol: TCP
+    targetPort: http
+  selector:
+    app.kubernetes.io/managed-by: {{ include "release.name" . }}
+    app.kubernetes.io/name: query