[Enhancement Request] Specify Explicit Controller when HTTPRoute backend is a Service

[DerekTBrown]

What would you like to be added:

HTTPRoute needs a way of explicitly specifying the controllerNames that should be responsible for implementing the route when parentRef is not a Gateway.

I propose this be added to the parentRef schema:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-app-1
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: http-app
    port: 8080
    controllers:
       - "istio.io/gateway-controller"

Alternatively, an annotation could be made on the Service resource to indicate the controller owner.

Why this is needed:

1. Users can define HTTPRoutes that use Service as a parentRef (for instance, Istio).

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-app-1
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: http-app
    port: 8080
  hostnames:
  - "http-app.my-namespace.svc.cluster.local"
  rules:
  - backendRefs:
    - name: http-app-stable
      port: 8080
      weight: 90
    - name: http-app-stable
      port: 8080
      weight: 10

2. Users might have multiple controllers capable of implementing this HTTPRoute.

3. This can lead to conflicts between controllers, that clobber owners references, statuses, etc.

[howardjohn]

Doesn't the status already include a "controller" field to distinguish multiple controllers writing a status about the same parent?

[youngnick]

The status does, but that doesn't help with choosing which implementation should handle a GAMMA route.

This comes down to the same problem we've talked about before - if the current assumption that there is only one GAMMA implementation in a cluster, how do you choose which implementation should handle a HTTPRoute that has a Service parentRef?

(@DerekTBrown the reason why this isn't implemented yet is that assumption I mentioned - we defined that we were not considering having multiple GAMMA implementations in a cluster until later. That was some time ago, so I guess now counts ;) )

We talked a little tiny bit at Kubecon NA 2024 about reconsidering a Mesh resource that would help with this, but it's a tricky problem. Adding a controllers field unfortunately makes no sense for the Ingress use case (as in that case, it's the GatewayClass the HTTPRoute rolls up to that's relevant), so to keep similarity to the Ingress case, we probably need some way to bucket Services as belonging to a particular GAMMA implementation. We talked about a few options previously, but couldn't come to a solution, which is why we have the one-implementation assumption currently.

[howardjohn]

TBH I still don't fully understand why you need to do this?

I feel like there are 2 sub-usecases:

1. Different pods have a different mesh (istio sidecar + linkerd sidecar)
2. Individual pods may have multiple meshes involved (sidecar + cilium, for instance)

(1) seems ok-ish? Why does it need to be scoped to only one mesh? Is it using impl-specific functionality?

(2) also seems ok-ish -- regardless of config, the 'first' mesh in the network path is going to handle Service networking entirely so the Route customization is not the deciding factor in who controls the traffic?

[steveliem]

I have considered the current design of the Gateway API within the GAMMA initiative, particularly regarding the use of parentRef in HTTPRoute resources that reference a Service. I believe that the existing approach is sufficient and does not require additional fields to specify which service mesh product to use, even in environments where multiple service meshes are deployed within a single cluster.

Rationale:

1. Controller Responsibility:

   • The determination of which service mesh manages a particular service should be handled by the respective mesh controllers. These controllers can be configured to monitor specific namespaces or resources, ensuring that each service is managed by the appropriate mesh without the need for explicit specification in the HTTPRoute.

2. Namespace and Resource Isolation:

   • By organizing services and routes within designated namespaces and applying appropriate labels or annotations, administrators can control which mesh controller oversees specific resources. This method maintains clear boundaries and prevents conflicts between multiple meshes operating within the same cluster.

3. Simplicity and Flexibility:

   • Keeping the Gateway API agnostic of specific service mesh implementations enhances its flexibility and simplicity. Introducing fields to specify a particular mesh product could complicate the API and reduce its adaptability to various environments and future technologies.

In conclusion, the current design effectively delegates the responsibility of service-to-mesh association to the mesh controllers and cluster configuration. This approach supports scalability and maintainability, even in complex multi-mesh scenarios, without necessitating changes to the Gateway API specifications.

For further context, similar discussions have taken place in the community, such as the issue on defining namespace boundaries for parent and backend references in mesh configurations .