Policy validation when multiple VS or controllers are in play

[tstraley]

We run two ingress-controller instances in a single cluster. When using certain Policy resources, we encountered situations where the Policy was being validated by both controllers, even if the Policy was not associated with any VS being managed by the controller.

The specific example we faced was a waf policy which was rejected and marked invalid for a controller-level configuration:

  Warning  Rejected        62s   nginx-ingress-controller  Policy main/waf-monitoring-only is invalid and was rejected: spec.waf: Forbidden: App Protect must be enabled via cli argument -enable-app-protect to use WAF policy

This was despite the fact that the VS using this Policy was only part of the other controller instance where -enable-app-protect was true.

The fix was simple once it was finally understood what the issue was (apply ingressClassName to this Policy to limit it to the specific controller instance), but it brings to light that some of the Policy validation that occurs is based on configuration of other resources such as the VS / VSR or controller instance.

Suggestion:
It seems like the Policy validation should not consider the configuration of other resources. If a VS attempts to use a Policy that it cannot apply due to a VS configuration issue, I would expect the VS to produce the error and pick up a Warning status because that Policy may be perfectly functional in some other VS / VSR that is configured correctly to use it.