Spike: figure out a path forward for misconfigured KongPlugin resources

[hbagdi]

Users always run into issues with malformed plugin configurations which brings down the entire reconciliation loop.
Figure out a way to avoid this from happening.
One are to explore will be to use Kong's /schema endpoint a lot more to verify if each KongPlugin is valid or not.
This will need to be cached to avoid performance issues when the network latency between the controller and Kong is significant (> 10s of milliseconds)

Acceptance criteria

TBD (@shaneutt to shepherd this issue)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[acoder-dev]

We have run into this problem many times. Our entire cluster goes down since the gateway fails if one plugin is misconfigured. This is a very concerning issue to us. We may drop Kong altogether because of it.

• Are there any plans to address this issue?
• Is there some kind of work around such as validating the plugin config in a CICD pipeline?

[hbagdi]

Please configure and install validating webhook: https://docs.konghq.com/kubernetes-ingress-controller/2.1.x/deployment/admission-webhook/#main

That will get rid of the problem in most cases.

[acoder-dev]

That's great thanks for the quick response we will look in to this.

[shaneutt]

In #2195 we are addressing this problem holistically and attempting to do it in ways that wont technically require the validating webhook (however, please do install it). We're going to close this in favor of #2195, please subscribe to that issue to track the progress of improvements in this regard.