principles/guidelines for HTTPProxy API design?

[skriss]

In working through the design for rate limiting, I found that it would've been helpful to have some documented principles or guidelines for how to approach adding to the HTTPProxy API. Some topics that could be covered:

• when exposing an Envoy feature, how to think about exposing it ~as-is vs. building a Contour-specific abstraction. Is the use of Envoy a feature, or an implementation detail?
• specifying policies "inline" vs. normalizing them into separate CRDs for reuse (we don't really do the latter currently; is there a place for it?)
• the implications of all non-TLS virtual hosts sharing a single HTTP connection manager (so filter config in general must be shared across them)
• when/how to mirror new HTTPProxy features into Ingress, as annotations?

Would folks find this helpful? If so, could some of the maintainers who have done more work here start to describe some of the guidelines?

[sunjayBhatia]

the implications of all non-TLS virtual hosts sharing a single HTTP connection manager (so filter config in general must be shared across them)

This is interesting as it seems we're in a point where we're exposing Envoy extensions/filters, we currently have slight differences between the RLS proposal and the Ext-Authz feature, and if we add support for ALS we could have another permutation if we don't have a prescribed pattern

[youngnick]

I agree that this is a useful document, I'll start writing down some principles here.

[youngnick]

The first rule is this one: no annotations. All config should be done using fields in HTTPProxy document.

Another very general rule is that we take the HTTPProxy API guarantee seriously. The HTTPProxy API is a stable API, and we can't change the behavior of fields in it without creating a new revision (which would start with a v2alpha1). Many of the rules we use are around making it less likely we will need to deprecate fields, as we will need to carry deprecated fields until we start work on a v2 (which is not impossible, but is a significant effort in change management, and we won't do without a very good reason).

Generally, adding things to change config should be done in HTTPProxy, and any policy around those settings should be added to the Gatekeeper policy template library.

When adding an entirely new type of configuration to HTTPProxy, consider using a SomethingPolicy struct to hold the setting, even if it's the only one for now. This allows us to add fields to that SomethingPolicy struct later, without needing the change the behavior of existing fields (which would violate the GA guarantee). Examples here are TimeoutPolicy and LoadBalancingPolicy. I acknowledge here that we've overloaded the word policy with this, unfortunately.

To address the more specific questions @skriss asked though:

• It was originally intended that the choice of proxy was an implementation detail. That goal has fallen away over time, and I think we could make an argument to retire it.
• However, one thing that is implied if we do say "Contour is Envoy-only" is that maybe we will do less abstracting of the config. I think this would be a bit of a mistake. One of the things that's valuable about Contour is that you don't need to understand everything about Envoy to use it. I don't think that having Contour be a very thin shim between Kubernetes and Envoy is desirable. I think we should strive to set some reasonable default behaviors for a reverse proxy (as distinct from the usual sidecar service proxy use-case for Envoy), and our HTTPProxy document should reflect that perspective.
• With that said, I'm not opposed to creating HTTPProxy document sections that are quite similar to the Envoy when it makes sense (something like the suggestion in Support list of loadBalancerPolicy for HTTPProxy #3044 is what I have in mind here).
• In terms of creating separate CRDs for policy etc, I think that there is a significant overhead to having more CRDs, both in operational terms (as they have their own lifecycle), and in cognitive terms. I haven't seen that the tradeoff of having separate CRDs has been worth it as yet.
• I think we have stumbled into having the insecure traffic share a HTTPConnectionManager, and we haven't really needed to think too much about this in the past. I acknowledge this is a gap, and we need to ensure that we think about this going forward.
• In terms of mirroring things into Ingress as annotations, I'm not sure. I really don't want to end up with lots of projectcontour.io annotations, I'd prefer for people to use HTTPProxy or hopefully, the Service APIs for more advanced use cases. I guess my current case-by-case rule for this is "If people really need it".

So, this is my perspective on some of the HTTPProxy principles. This isn't the final word though, just a place to start.

I'd be interested to hear what everyone else thinks. @stevesloka, @jpeach, what are your thoughts? Am I off the mark here, or have I missed something?

[skriss]

It was originally intended that the choice of proxy was an implementation detail. That goal has fallen away over time, and I think we could make an argument to retire it.

Agree with this based on what I've seen. We're very coupled to Envoy, for better or worse. But, as you said subsequently, that doesn't mean we need to/should map 1:1 with the Envoy API.

I think we have stumbled into having the insecure traffic share a HTTPConnectionManager, and we haven't really needed to think too much about this in the past. I acknowledge this is a gap, and we need to ensure that we think about this going forward.

Yeah, this is just an awkward impedance mismatch between the Contour and Envoy APIs right now. It looks like there is some upstream work going on to allow conditional filter application (see envoyproxy/envoy#12968 and linked google doc), which might be helpful here in the future.

[xaleeks]

One of the things that's valuable about Contour is that you don't need to understand everything about Envoy to use it. I don't think that having Contour be a very thin shim between Kubernetes and Envoy is desirable.

Agree with the general sentiment and we should look to deliver value in Contour operating atop Envoy and not purely as a shim. But seems to be a lot of users asking for config knobs to Envoy, and it'd be interesting to find out how many of our users are picking Contour because they've already invested in an envoy-based stack.

[youngnick]

I agree, that's an interesting question, but it's important to remember that if you're using another set of Envoys, the Contour Envoys aren't much use to you directly, since you can only have one xDS server per Envoy.

I'm also worried about the amount of time we have and will spend adding Envoy knobs, although I think that with the Gatekeeper pattern for policy enforcement, we've made it so we don't have to build that functionality into Contour as well (which we were doing before).

I'm hoping that as the Service APIs mature, we should be able to lean more on what they expose as a standard proxy featureset.