False positive status reason by timing issues in alert rules

[TeodorSAP]

We observed single datapoints where the health status was reported negative which was not the case. That mainly is caused by having the status evaluation based on two rule evaluations, which might be evaluated slightly at a different point in time. So a beginning ingestion might have been evaluated to true already where the starting export might not have been evaluated yet, so that there is a short time window where the status is seen as "no export possible".

A refactoring of the alerting was done for the LogPipeline already (#1397) and here no false positives were observed anymore.

The existing self-monitor alerts, based on PromQL queries, should be refactored such that:

• A single PromQL query (i.e. alert) is mapped to each unhealthy condition (i.e. no Go code is involved in logically evaluating the firing alerts)
• The for clause is used to avoid timing issues (where justified)
• Alerts in firing state should strictly be used for negative/unhealthy scenarios

This refactoring should follow the changes introduced in: #1397

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to the lack of recent activity.
/lifecycle rotten