Not getting alerts when one my kafka broker is down

[shreyasarani23]

Hi I have followed the steps in this link https://strimzi.io/docs/operators/latest/deploying.html#assembly-metrics-str . And below is the configuration of alert-manager-config file.

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-alertmanager
type: Opaque
stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 1m

    route:
      receiver: 'gmail-notifications'

    receivers:
    - name: 'gmail-notifications'
      email_configs:
      - to: [email protected]
        from: [email protected]
        smarthost: smtp.gmail.com:587
        auth_username: [email protected]
        auth_identity: [email protected]
        auth_password: app-password
        send_resolved: true

But I am not getting any alerting mails when one of my kafka broker/zookeeper pod/entity operator/user operator is down. Please tell me if there is anything wrong in my configuration or any additional things need to be specified?

[scholzj]

Well, do you see the alerts in the Alert Manager console?

[scholzj]

I don't know. That will depend on your environment and on how you installed the Alert Manager. Normally it has some service etc.

[shreyasarani23]

ok let me check

[shreyasarani23]

well I checked the logs of alert manager pod it says the below

level=warn ts=2021-05-24T14:04:10.800Z caller=notify.go:674 component=dispatcher receiver=gmail-notifications integration=email[0] msg="Notify attempt failed, will retry later" attempts=10 err="*email.loginAuth auth: 534 5.7.9 Application-specific password required. Learn more at\n5.7.9  https://support.google.com/mail/?p=InvalidSecondFactor g72sm11150655pfb.33 - gsmtp"

FYI I have specified the application password

[scholzj]

I think you really need to ask this either at some Prometheus forum or whoever implemented this gmail-notification plugin (since I'm not sure whether that is part of the regular Prometheus Alert Manager deployment). I in the past used only Slack notifications - thas is also what is part of our examples.

[shreyasarani23]

Well I have changed my app password and now it is working , I am able to get the alerts . Thanks @scholzj for the help.