KongConsumer Secrets being required to exist before the KongConsumer causes some friction for deployments

[shaneutt]

Current Behavior

Prior to v2.x of the ingress controller we had the problem #729 wherein the data-plane can become completely locked up when KongConsumers with reference Secret credentials were configured in such a way that was in violation of unique constraints in the Kong Gateway (from our purposes here, the "data-plane").

#2015 attempted to correct for this in the validation webhook, however in order to do more complete prevention it added the requirement that any Secret that is referenced must exist prior to the KongConsumer. This requirement causes some friction with deployments and goes against the grain of object references in Kubernetes, while also effectively avoiding the data-plane lockup rooted in consumer credential unique constraint violations.

The purpose of this issue is to improve upon the solution to the original problem so that we can still avoid data-plane lockups, but without having to force users to split their manifests into stages as this is a bit unnatural for Kubernetes operations (save perhaps with CRDs, but we want to avoid having our KongConsumers working anything like CRDs).

Expected Behavior

When I deploy a manifest containing KongConsumers which reference Secret credentials, and also contains the Secret objects themselves I should get a successful apply and not have to consider order of operations.

Kong Ingress Controller version

v2.x.x and up

Kubernetes version

any

[shaneutt]

We're currently tracking this issue in #2195 as something that should be resolved by that related feature.

[mflendrich]

Closing in favor of #2195