WIP [main] Implement alerting

[jbiers]

Issue: #672

[diogoasouza]

No need to support interval or limit?

[mallardduck]

Nice.

Only thing I'm wondering about is, "are there any alerts that BRO should provide by default that cannot be disabled". In other words, are there any features of this new monitoring/alerting area of BRO that (when enabled) are required? I guess this thought goes just as much for these alerting rules as it does for any metrics collection things too - since I could imagine this factor relates more to dashboards than alerts TBH. 🤔

[alexandreLamarre]

I would also suggest putting this default alert rule definition in helm template in something like : templates/default-alerting-rules.yaml behind a template flag like .Values.useDefaultAlerts or something and the current way it works could be behind something like .Values.additionalRules

Edit: if we do put the default alerting rules behind that flag, make sure to pass in some form : .defaultAlerts.AdditionalLabels to the alerting rule, since AlertManager uses label matchers to send stuff to specific remote integrations, so users will want to pass in their own labels

[alexandreLamarre]

Let's cleanup some metric name best practices :

• https://prometheus.io/docs/concepts/escaping_schemes/
• https://prometheus.io/docs/practices/naming/
• https://prometheus.io/docs/practices/instrumentation/

Some best practices for rules/alerting:

• https://prometheus.io/docs/practices/alerting/
• https://prometheus.io/docs/practices/rules/