Skip to content

Commit

Permalink
Allow gatewayProvisioner to create contour that only watch limited na…
Browse files Browse the repository at this point in the history
…mespaces of resources

Signed-off-by: Lubron Zhan <[email protected]>
Signed-off-by: lubronzhan <[email protected]>

Release note

Signed-off-by: Lubron Zhan <[email protected]>
Signed-off-by: lubronzhan <[email protected]>

Unit tests and fix contourDeployment manifest

Signed-off-by: lubronzhan <[email protected]>

Modify the spec

Signed-off-by: Lubron Zhan <[email protected]>
Signed-off-by: lubronzhan <[email protected]>

Move the contain func-

Signed-off-by: lubronzhan <[email protected]>

Refactor contains func

Signed-off-by: lubronzhan <[email protected]>

Inital rbac commit:

Signed-off-by: lubronzhan <[email protected]>

Refactor the role and rolebinding code again

Signed-off-by: Lubron Zhan <[email protected]>
Signed-off-by: lubronzhan <[email protected]>

Fix golint
Signed-off-by: lubronzhan <[email protected]>

Signed-off-by: lubronzhan <[email protected]>

fix spell

Signed-off-by: lubronzhan <[email protected]>

Print git diff change

Signed-off-by: lubronzhan <[email protected]>

Revert "Print git diff change"

This reverts commit 189131c.

Signed-off-by: lubronzhan <[email protected]>

Print out controller-gen version

Signed-off-by: lubronzhan <[email protected]>

Fix logic

Signed-off-by: lubronzhan <[email protected]>
  • Loading branch information
lubronzhan committed Jan 17, 2024
1 parent 572515a commit 2e08f76
Show file tree
Hide file tree
Showing 24 changed files with 455 additions and 76 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/prbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
version: v1.55.2
# TODO: re-enable linting tools package once https://github.com/projectcontour/contour/issues/5077
# is resolved
args: --build-tags=e2e,conformance,gcp,oidc,none
args: --build-tags=e2e,conformance,gcp,oidc,none --out-${NO_FUTURE}format=colored-line-number
- uses: act10ns/slack@ed1309ab9862e57e9e583e51c7889486b9a00b0f # v2.0.0
with:
status: ${{ job.status }}
Expand Down
5 changes: 5 additions & 0 deletions apis/projectcontour/v1alpha1/contourdeployment.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,11 @@ type ContourSettings struct {
// the annotations for Prometheus will be appended or overwritten with predefined value.
// +optional
PodAnnotations map[string]string `json:"podAnnotations,omitempty"`

// WatchNamespaces is an array of namespaces. Setting it will instruct the contour instance
// to only watch these set of namespaces
// +optional
WatchNamespaces []string `json:"watchNamespaces,omitempty"`
}

// DeploymentSettings contains settings for Deployment resources.
Expand Down
5 changes: 5 additions & 0 deletions apis/projectcontour/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions changelogs/unreleased/6073-lubronzhan-small.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Allow gatewayProvisioner to create contour that only watch limited namespaces of resources
16 changes: 8 additions & 8 deletions design/automated-provisioning-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ It will handle the full provisioning lifecycle for these Gateways: creating a Co
![drawing](images/gatewayapi-provisioner-overview.png)

Importantly, this provisioning model allows for (a) any number of GatewayClasses to be controlled by the Gateway provisioner; and (b) any number of Gateways per controlled GatewayClass.
While the exact use cases for many Contour Gateways remain unclear, these one-to-many relationships between controller and GatewayClass, and GatewayClass and Gateway, offer users the full flexibility that the Gateway API spec intends.
While the exact use cases for many Contour Gateways remain unclear, these one-to-many relationships between controller and GatewayClass, and GatewayClass and Gateway, offer users the full flexibility that the Gateway API spec intends.
Additionally, support for this one-to-many relationship is required in order to pass any of the Gateway API conformance tests.

![drawing](images/gatewayapi-resource-relationships.png)
Expand All @@ -77,7 +77,7 @@ Those users can continue to statically provision their Contour + Envoy instances

There will be two ways to configure Contour for Gateway API support in the static provisioning scenario:
- **Controller name** - this is the model implemented today, where Contour is configured with a controller name, and it continuously looks for the oldest GatewayClass with that controller, and the oldest Gateway using that GatewayClass. This model is appropriate for users who expect their GatewayClasses and Gateways to come and go, and who want their Contour instance to dynamically pick up the appropriate Gateway as those changes occur.
- **Gateway name** - Contour can alternately be directly configured with a specific Gateway name, which avoids the multiple levels of indirection of the previous model. This model is appropriate for users who expect their Contour instance to correspond to a single static Gateway; the lifecycle of the Gateway and the lifecycle of the Contour instance are tied together.
- **Gateway name** - Contour can alternately be directly configured with a specific Gateway name, which avoids the multiple levels of indirection of the previous model. This model is appropriate for users who expect their Contour instance to correspond to a single static Gateway; the lifecycle of the Gateway and the lifecycle of the Contour instance are tied together.

Note that the Gateway provisioner will make use of the **Gateway name** mode of configuring Contour, to tie each instance of Contour it provisions directly to a specific Gateway.

Expand All @@ -88,13 +88,13 @@ This custom resource definition embeds a ContourConfiguration spec, as well as a
This ContourDeployment resource serves as a template that defines exactly how to customize each Contour + Envoy instance that is created for this GatewayClass.
When a Gateway is provisioned, the Gateway provisioner will use the configuration options specified to customize the YAML that is applied, and will pass through a copy of the ContourConfiguration data to the Gateway’s Contour instance.

Note that, according to the Gateway API documentation:
Note that, according to the Gateway API documentation:
> It is recommended that [GatewayClass] be used as a template for Gateways.
> This means that a Gateway is based on the state of the GatewayClass at the time it was created and changes to the GatewayClass or associated parameters are not propagated down to existing Gateways.
> This recommendation is intended to limit the blast radius of changes to GatewayClass or associated parameters.
(ref. [GatewayClass API reference documentation](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1alpha2.GatewayClass))
> This recommendation is intended to limit the blast radius of changes to GatewayClass or associated parameters.
(ref. [GatewayClass API reference documentation](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1alpha2.GatewayClass))

For Contour, this means that after a Gateway has been provisioned, the Gateway provisioner will not apply subsequent changes to the GatewayClass/ContourDeployment to it.
For Contour, this means that after a Gateway has been provisioned, the Gateway provisioner will not apply subsequent changes to the GatewayClass/ContourDeployment to it.
This also means that Contour users can modify the ContourConfigurations used by their running Contours after instantiation, without having those changes overwritten by the Gateway provisioner.

Since the Gateway provisioner supports multiple GatewayClasses, each GatewayClass can have a different ContourDeployment reference, corresponding to different sets of Gateway configuration profiles that the infrastructure provider offers (e.g. an external vs. internal profile).
Expand All @@ -109,7 +109,7 @@ This proposal is related to, but separate from, the managed Envoy proposal:
- If we don’t implement managed Envoy, then the Gateway provisioner implements the Envoy provisioning logic.
- Either way, the logic needs to be implemented and live somewhere.

Advantages of doing managed Envoy:
Advantages of doing managed Envoy:
- Users who don’t want automated Gateway provisioning, but do want automated Envoy provisioning, can have it
- Users who don’t want to use Gateway API can still take advantage of automated Envoy provisioning
- Listener programming (combo of Envoy service + Envoy listeners) can be done in one place
Expand All @@ -122,7 +122,7 @@ Disadvantages of doing managed Envoy:
We considered continuing to invest in the Contour Operator, including the Contour CRD, with an eye towards bringing it to beta and eventually GA, and implementing the Gateway provisioner within the Operator (since they would share much of the underlying logic).
Our first challenge with this option is that we have failed to establish a community of contributors around the Operator, so the work would need to be done by the core Contour team of maintainers, which would detract from Contour development.
Our second challenge is that we have seen and heard of only limited usage of the Operator in the wild, so it’s not clear to us that continuing to develop it is an important priority for our users.
Finally, continuing to maintain the Operator in a separate repository creates development overhead, since various bits of code and configuration must be manually kept in sync between Contour and the Operator.
Finally, continuing to maintain the Operator in a separate repository creates development overhead, since various bits of code and configuration must be manually kept in sync between Contour and the Operator.

### Alternative 2
We also considered converting the existing Contour Operator into a Gateway provisioner, dropping the Contour CRD, and only supporting a Gateway API-based dynamic provisioning workflow.
Expand Down
7 changes: 7 additions & 0 deletions examples/contour/01-crds.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1620,6 +1620,13 @@ spec:
More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
type: object
type: object
watchNamespaces:
description: |-
WatchNamespaces is an array of namespaces. Setting it will instruct the contour instance
to only watch these set of namespaces
items:
type: string
type: array
type: object
envoy:
description: |-
Expand Down
34 changes: 34 additions & 0 deletions examples/namespaced-gatewayapi/gatewayclass-clusterrole.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# GatewayClass is cluster-scope. So split its necessary cluster role into a single file
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: contour-gatewayclass
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: contour-gatewayclass
subjects:
- kind: ServiceAccount
name: contour
namespace: projectcontour
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: contour-gatewayclass
rules:
- apiGroups:
- gateway.networking.k8s.io
resources:
- gatewayclasses
verbs:
- get
- list
- watch
- apiGroups:
- gateway.networking.k8s.io
resources:
- gatewayclasses/status
verbs:
- update
79 changes: 79 additions & 0 deletions examples/namespaced-gatewayapi/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# This kustomize file can be used as an example for deploying Contour with custom ClusterRole and Role that could
# change Contour to watch namespaced resources plus cluster-scoped GatewayClass
# It changes the current cluster-wide RBAC rules in the example deployment manifest to namespaced RBAC rules.
# It is meant to be used together with contour serve --watch-namespaces=<ns> option to restrict
# Contour to a certain namespace.
# It also add a ClusterRole and ClusterRoleBinding to Role that only watches GatwayClass
# Run with:
# kubectl kustomize examples/namespaced-gatewayapi/
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../render/
- gatewayclass-clusterrole.yaml
patches:
- patch: |-
- op: replace
path: /kind
value: RoleBinding
- op: replace
path: /metadata/name
value: contour-resources
- op: replace
path: /roleRef/kind
value: Role
- op: replace
path: /roleRef/name
value: contour-resources
- op: add
path: /metadata/namespace
value: projectcontour
target:
group: rbac.authorization.k8s.io
kind: ClusterRoleBinding
name: contour
version: v1
- patch: |-
- op: replace
path: /kind
value: Role
- op: replace
path: /metadata/name
value: contour-resources
- op: add
path: /metadata/namespace
value: projectcontour
- op: replace
path: /rules/1
value: |-
- apiGroups:
- gateway.networking.k8s.io
resources:
- gateways
- grpcroutes
- httproutes
- referencegrants
- tcproutes
- tlsroutes
verbs:
- get
- list
- watch
- op: replace
path: /rules/2
value: |-
- apiGroups:
- gateway.networking.k8s.io
resources:
- gateways/status
- grpcroutes/status
- httproutes/status
- tcproutes/status
- tlsroutes/status
verbs:
- update
target:
group: rbac.authorization.k8s.io
kind: ClusterRole
name: contour
version: v1
7 changes: 7 additions & 0 deletions examples/render/contour-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1839,6 +1839,13 @@ spec:
More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
type: object
type: object
watchNamespaces:
description: |-
WatchNamespaces is an array of namespaces. Setting it will instruct the contour instance
to only watch these set of namespaces
items:
type: string
type: array
type: object
envoy:
description: |-
Expand Down
7 changes: 7 additions & 0 deletions examples/render/contour-gateway-provisioner.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1631,6 +1631,13 @@ spec:
More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
type: object
type: object
watchNamespaces:
description: |-
WatchNamespaces is an array of namespaces. Setting it will instruct the contour instance
to only watch these set of namespaces
items:
type: string
type: array
type: object
envoy:
description: |-
Expand Down
7 changes: 7 additions & 0 deletions examples/render/contour-gateway.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1842,6 +1842,13 @@ spec:
More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
type: object
type: object
watchNamespaces:
description: |-
WatchNamespaces is an array of namespaces. Setting it will instruct the contour instance
to only watch these set of namespaces
items:
type: string
type: array
type: object
envoy:
description: |-
Expand Down
7 changes: 7 additions & 0 deletions examples/render/contour.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1839,6 +1839,13 @@ spec:
More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
type: object
type: object
watchNamespaces:
description: |-
WatchNamespaces is an array of namespaces. Setting it will instruct the contour instance
to only watch these set of namespaces
items:
type: string
type: array
type: object
envoy:
description: |-
Expand Down
2 changes: 2 additions & 0 deletions internal/provisioner/controller/gateway.go
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,8 @@ func (r *gatewayReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct

contourModel.Spec.KubernetesLogLevel = contourParams.KubernetesLogLevel

contourModel.Spec.WatchNamespaces = contourParams.WatchNamespaces

if contourParams.Deployment != nil &&
contourParams.Deployment.Strategy != nil {
contourModel.Spec.ContourDeploymentStrategy = *contourParams.Deployment.Strategy
Expand Down
11 changes: 11 additions & 0 deletions internal/provisioner/model/model.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
package model

import (
"slices"

contourv1alpha1 "github.com/projectcontour/contour/apis/projectcontour/v1alpha1"
"github.com/projectcontour/contour/internal/ref"

Expand Down Expand Up @@ -134,6 +136,10 @@ func (c *Contour) EnvoyTolerationsExist() bool {
return false
}

func (c *Contour) WatchAllNamespaces() bool {
return c.Spec.WatchNamespaces == nil || slices.Contains(c.Spec.WatchNamespaces, corev1.NamespaceAll)
}

// ContourSpec defines the desired state of Contour.
type ContourSpec struct {
// ContourReplicas is the desired number of Contour replicas. If unset,
Expand Down Expand Up @@ -245,6 +251,11 @@ type ContourSpec struct {
// If the value is 0, the overload manager is disabled.
// defaults to 0.
EnvoyMaxHeapSizeBytes uint64

// WatchNamespaces is an array of namespaces. Setting it will instruct the contour instance
// to only watch these set of namespaces
// default is nil, contour will watch resource of all namespaces
WatchNamespaces []string
}

// WorkloadType is the type of Kubernetes workload to use for a component.
Expand Down
5 changes: 5 additions & 0 deletions internal/provisioner/objects/deployment/deployment.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
"context"
"fmt"
"path/filepath"
"strings"

"github.com/projectcontour/contour/apis/projectcontour/v1alpha1"
"github.com/projectcontour/contour/internal/provisioner/equality"
Expand Down Expand Up @@ -100,6 +101,10 @@ func DesiredDeployment(contour *model.Contour, image string) *appsv1.Deployment
args = append(args, "--debug")
}

if !contour.WatchAllNamespaces() {
args = append(args, fmt.Sprintf("--watch-namespaces=%s", strings.Join(contour.Spec.WatchNamespaces, ",")))
}

// Pass the insecure/secure flags to Contour if using non-default ports.
for _, port := range contour.Spec.NetworkPublishing.Envoy.Ports {
switch {
Expand Down
Loading

0 comments on commit 2e08f76

Please sign in to comment.