Prometheus: ingress_controller_configuration_push_count should tell between intermittent errors and lock-ups

[mflendrich]

Is there an existing issue for this?

• I have searched the existing issues

Problem Statement

@seh reported that they'd like to use the ingress_controller_configuration_push_count Prometheus metric to alert in case a configuration lock-up (see #2195) happens, but that metric today does not tell between transient errors (e.g. network disconnect) and those that require fixing a config conflict (e.g. conflicting consumers #2324, #680)

Proposed Solution

Add a metric label on the ingress_controller_configuration_push_count metric telling between failures that require fixing a conflict and those not requiring fixing a conflict.

Additional information

Note that the existing metric ingress_controller_translation_count[success=true|false] may be answering this question already.

Acceptance Criteria

• Our prometheus integration includes a new label on the metric ingress_controller_translation_count[success=true|false] where true is successfully pushed configs while false is failed pushes
• Our prometheus OOTB dashboard is updated with a new widget using the new label
• Documentation is updated for the new metric/label

No response

[czeslavo]

I changed back the milestone to KIC v2.7.0 because the functional changes managed to get into this release.

[czeslavo]

Grafana dashboard got updated in the KIC's repository https://github.com/Kong/kubernetes-ingress-controller/blob/main/grafana.json, but I'm waiting for Grafana (the company) to respond to my question regarding problems with the dashboard visibility on their catalog.

[jrsmroz]

So, what's left to close the issue? We should close KIC 2.7 milestone soon.

[czeslavo]

We're missing a review on Kong/docs.konghq.com#4500. The dashboard JSON in our repository has been updated so I'd consider the acceptance criteria for it complete.

Regarding the dashboard on grafana.com, I think we can take it separately. I've created an issue to track this: #2991