How to mount a secret for alertmanager pod ?

[NargiT]

Hello,

Is there a way to configure prometheus.jsonnet in order to mount volumes for alertmanager pod ?

I need to provide a integration-key to pager duty but I do not want to hardcode the integration-key inside alertmanager-config.yaml file. Since we use prometheus.jsonnet and everything is pushed to github I was expecting a way to mount a file using an existing secret deployed sealed secret externally.

I check here: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md but I do not feel it useful.

This one looks nice : https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#alertmanagerspec but I do not know how to integrated it with prometheus.jsonnet ?

Any advice ?

[s-urbaniak]

Whenever you import and configure main.libsonnet you can still add arbitrary properties which were not part of the constructor, example:

local kp =
  (import 'kube-prometheus/main.libsonnet') + {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },
    },
  } + {
    alertmanager+:: {
      alertmanager+: {
        spec+: {
          volumes: [
            {
              name: 'pager-duty',
              secret: {
                secretName: 'pager-duty',
              },
            },
          ],
          volumeMounts: [
            {
              mountPath: '/etc/pager-duty',
              name: 'pager-duty',
            },
          ],
        },
      },
    },
  };

but maybe @paulfantom knows of a more idiomatic way.

[NargiT]

thanks I was able to mount secrets as I found that alertmanager is able to manage them using secrets fields following this spec https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#alertmanagerspec it will be mounted inside /etc/alertmanager/secrets/my-secret/key e.g. with a secret call my-secret and this data

data:
  key: token (base64)

Now I don't know how to fill routing_key inside alertmanager-config.yaml according to the documentation routing_key is a tmpl_secret but I would like it to hold token or to be evaluated to it.

[s-urbaniak]

My bad, I was focusing so much on mounting a secret, forgetting that yo simply want to configure pagerduty. How about leveraging AlertmanagerConfig custom resource https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#alertmanagerconfig which has a configuration option for PagerDuty which lets you point to a secret as well: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#pagerdutyconfig

[NargiT]

As I am using plain alertmanager-config.yaml file I was able to manage this in a different way. The reason I did differently is because I do not want, right now, people maintaining this file to learn vanilla alert manager and prometheus-operator abstraction. But definitely something I will have a look once people are familiar with AM.

The 2nd reason I am doing this with secret it's because we are using a gitops approch and no secret must be commited to our github repository, and thanks to kubeseal I only need a reference to it, but you can simply create a secret manually for this example.

Create a file called pagerduty.integration.key.tmpl with the following content {{ define "pagerduty.integration.key" }} YOUR_INTEGRATION_KEY {{ end }}. Then create a k8s secret from this file (e.g --from-file)

apiVersion: v1
data:
  pagerduty.integration.key.tmpl: IyBDcmFmdCB5....
kind: Secret
metadata:
  creationTimestamp: null
  name: pagerduty-secrets
  namespace: monitoring

Then add a reference to prometheus.jsonnet, so the secret 'pagerduty-secrets' will be mounted in the same namespace as alertmanager. The path will be /etc/alertmanager/secrets/pagerduty-secrets/pagerduty.integration.key.

...
alertmanager+:: {
  alertmanager+: {
    spec+: {
      secrets: ['pagerduty-secrets']
    }
  }
}
...

Finally inject the integration key directly to pager duty using tmpl_secret feature available for pagerduty.

alertmanager-config.yaml

# declare 
templates:
- /etc/alertmanager/secrets/pagerduty-secrets/pagerduty.integration.key

# inside pagerduty configs block
receivers:
- name: foo
  pagerduty_configs:
    - routing_key: '{{ template "pagerduty.integration.key" . }}'

Hope this will help someone, but the solution provided by @s-urbaniak using prometheus-operator abstraction is the way to go but requires more knowledge for the maintainer. #889 (reply in thread)

[metalmatze]

Thanks for the elaborate answer for others! 💯

[Labidi-github]

Many thanks for the detailed answer