diff --git a/enhancements/sig-architecture/21-cluster-gateway/README.md b/enhancements/sig-architecture/21-cluster-gateway/README.md new file mode 100644 index 00000000..1ca7995e --- /dev/null +++ b/enhancements/sig-architecture/21-cluster-gateway/README.md @@ -0,0 +1,189 @@ +# Multi-Cluster API Gateway + +## Release Signoff Checklist + +- [ ] Enhancement is `implementable` +- [ ] Design details are appropriately documented from clear requirements +- [ ] Test plan is defined +- [ ] Graduation criteria for dev preview, tech preview, GA +- [ ] User-facing documentation is created in [website](https://github.com/open-cluster-management-io/open-cluster-management-io.github.io/) + +## Summary + +After solving the issues of (1) hub -> spoke network connectivity [#19](https://github.com/open-cluster-management-io/enhancements/pull/19) +and (2) hub -> spoke authentication via token projection [#24](https://github.com/open-cluster-management-io/enhancements/pull/24) +the last missing puzzle for dynamic multi-cluster API invocation will be an +API gateway for helping switching contexts between clusters. This API gateway +will be proxying the requests to any managed cluster similar to the mechanism +of existing `serivce/proxy` or `pod/proxy`, and the gateway is supposed to be +plugging into the kubernetes cluster as an aggregated apiserver so that the +hub cluster can discover a new proxy subresource named `clustergateways/proxy`. + +## Motivation + +### Dynamically Switching between Managed Clusters + +An old-fashion approach of reading resources from multiple managed clusters +will be looping over the clusters and instantiate new client instances then +send the actual requests. While w/ the help of gateway, switching client context +between the managed clusters will be as simple as replacing the target resource +name of the `clustergateway` resource, e.g. for accessing a cluster named `foo`, +all the client required to do is simply requesting the `proxy` subresource for +the new resource `clustergateway` named `foo`. Additionally, accessing arbitrary +non-resource paths will also be supported, e.g. for probing the overall health +of the managed cluster, we wil be call an API path of `/apis/open-cluster-management.io/clustergateways/foo/proxy/healthz` +against the hub kube-apiserver. Note that the API gateway is supposed to use the +projected service-account token [#24](https://github.com/open-cluster-management-io/enhancements/pull/24) +to identify itself when proxying requests to the managed cluster's kube-apiserver. + +### Keep KubeConfig/Tokens Invisible From Clients + +In most cases, we hope to grant the accessibility of managed clusters' +kube-apiserver to hub components w/o exposing the plain tokens. For the gateway, +it can be easily done by restricting the proxy clients RBAC rights to the +`clustergateways/proxy` subresource, also the verbs can be tightened so that we +can for example ban some proxy clients from deleting or listing resources. +To secure the tokens from the both the gateway users and the other tenants +sharing the hub cluster, all the projected tokens are supposed to be maintained +under a single high-secured namespace named `managed-cluster-credentials`. + +### Network Connectivity + +If the managed cluster is running in an isolated network, we will have to rely on +the work proposed by KEP [#19](https://github.com/open-cluster-management-io/enhancements/pull/19) +which establishes reverse proxy tunnels between the hub and spoke clusters so the +those requests proxied by the gateway can finally reach the managed cluster's +apiserver. + +## Goals + +- Having a new resource `clustergateway` and its subresource `proxy` based on + apiserver aggregation. +- Adding a new optional component named `cluster-gateway` for delegating requests + to the spoke clusters. + +## Proposal + +An initial POC implementation now goes publicly available under the repo [cluster-gateway](https://github.com/oam-dev/cluster-gateway). + + +### API + +By installing the gateway to the hub cluster via applying an `APIService`, the new +resource should be visible by the `kubectl api-resources` command. A sample will +be: + +```yaml +apiVersion: "proxy.open-cluster-management.io/v1alpha1" +kind: "ClusterGateway" +metadata: + name: <..> +spec: + provider: "" + access: + endpoint: "https://127.0.0.1:9443" + caBundle: "..." + credential: + type: [X509Certificate|ServiceAccountToken|ManagedServiceAccount] + x509Certificate: + certificate: "..." + privateKey: "..." + serviceAccountToken: + token: "..." + managedServiceAccount: + namespace: ".." + name: "..." +status: { } +``` + +Note that the credential being used by the gateway can be from 3 diffrent +sources: + +- X509Certificate: PEM encoded certificates and keys. +- ServiceAccountToken: The JWT token signed and copied from the managed cluster. +- ManagedServiceAccount: Referencing an existing managed service-account, and read + the corresponding projected tokens to authenticate the proxying requests. + +### Client Usage + + +There can be two major kinds of clients using the api gateway: + +- Kubectl Users: In some cases, the hub administrators need a one-time read/write + to a managed cluster e.g. when trouble-shooting or operating certain cluster. The + only thing we will need to do to make the kubectl works with the gatewa is just + simply adding a suffix `/apis/open-cluster-management.io/clustergateways//proxy/` + to the original hub kubeconfig, this will magically turn the hub kubeconfig to a + kubeconfig dedicated for any managed cluster. + +- Hub System: Components in hub cluster can also do the similar things as appending + a suffix to the request manually. Also we will be providing a set of client library + by "client expansion" to make the integration easier: + +A sample of the client library's interface will be: + +```go +type ClusterGatewayExpansion interface { + RESTClient(clusterName string) rest.Interface + + GetKubernetesClient(clusterName string) kubernetes.Interface + GetControllerRuntimeClient(clusterName string, options client.Options) (client.Client, error) + + RoundTripperForCluster(clusterName string) http.RoundTripper + RoundTripperForClusterFromContext() http.RoundTripper + RoundTripperForClusterFromContextWrapper(http.RoundTripper) http.RoundTripper +} +// See also: https://github.com/oam-dev/cluster-gateway/blob/master/pkg/generated/clientset/versioned/typed/cluster/v1alpha1/clustergateway_expansion.go#L34-L35 +``` + +For example to read a namespace from cluster "foo" via the library will be easy as: + +```go +kubernetes.New(gatewayClient. + ClusterV1alpha1(). + ClusterGateways(). + RESTClient("foo")). + CoreV1(). + Namespaces(). + Get(....) +``` + + +### Authorizing the Proxying Requests + + +Due to the fact that all the outbound requests from the gateway will be sharing a +common identity when reaching a managed cluster, we are not able to control the +granularity of authorization to discriminate different end users working behind +the proxy. However, practically it can be difficult to the control the RBAC scope +of the outbound requests from the gateway dynamically b/c for now in OCM there's +no such framework as "multi-cluster authorization". But one feasible solution is to +have an additional authorization check before actually proxying the requests via the +`rbacv1.SubjectAccessReview` (SAR) api. The subject of the SAR should keep the +original client identity, but the verb can be virtual such as `/`. +For example, when a client "user1" is attempting to read a namespace "default" from +cluster "foo", then the following SAR will be asserted before proxying: + +```yaml +apiVersion: authorization.k8s.io/v1 +kind: SubjectAccessReview +spec: + resourceAttributes: + group: "" + resource: "namespaces" + verb: "foo/get" + name: "default" +``` + +Alternatively in the future if there's a native central authorization engine in OCM, +we can simply switch from manual SAR calls to the engine. + + +### Trimming Off Persisting Layer + +Usually an aggregated apiserver requires ETCD backends, but this can be trimmed off if +we can restrict the `ManagedServiceAccount` to be the only source of the authentication +credentials. By customizing the REST storage implementation of the aggregated +`ClusterGateway` resource, we can transform the secrets projected from managed cluster +to a virtual `ClusterGateway` instance. The virtual instance should be completely read-only +to avoid conflicts with the underlying secret resource.