Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minimize the RBAC permissions for the pull mode cluster #5793

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

zhzhuang-zju
Copy link
Contributor

@zhzhuang-zju zhzhuang-zju commented Nov 7, 2024

What type of PR is this?
/kind feature

What this PR does / why we need it:
image
Currently, pull-mode clusters interact with karmada-apiserver on the control plane via a mounted karmada-kubeconfig, where the certificate has Organization as system:nodes and CN as system:node:{cluser_name}, so karmada -agent has the RBAC permissions on the control plane of a user with Group as system:nodes and username as system:node:{cluser_name}.
Since the control plane binds the clusterrole to Group system:nodes, all karmada-agents actually have the same set of RBAC permissions.

The clusterrole bound to Group system:nodes has too many permissions, e.g. secret permissions, cluster permissions.

image
So in order to minimise RBAC permissions, each karmada-agent needs to be assigned separate RBAC permissions

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
Installation methods involved:

  • local-up
  • karmadactl init

Does this PR introduce a user-facing change?:

`Security`: minimize the RBAC permissions for the pull mode cluster

@karmada-bot karmada-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 7, 2024
@karmada-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chaunceyjiang for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@karmada-bot karmada-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 7, 2024
@zhzhuang-zju
Copy link
Contributor Author

Local validation:

$ karmadactl register 172.18.0.6:32543 --token ak0bng.1mde96apjyi98dwz --discovery-token-ca-cert-hash sha256:c31ef70196f7b28545526c94a3c62ce74ad9c476971a818e16f49396f50b9430 --kubeconfig ~/.kube/member4.config --context member4  --karmada-agent-image docker.io/karmada/karmada-agent:latest --cluster-name member4 -v 4
I1107 20:23:59.964757 3098581 register.go:331] Registering cluster. cluster name: member4
I1107 20:23:59.964796 3098581 register.go:332] Registering cluster. cluster namespace: karmada-cluster
[preflight] Running pre-flight checks
I1107 20:23:59.964814 3098581 register.go:491] Validating the existence of file /etc/karmada/bootstrap-karmada-agent.conf
I1107 20:23:59.964827 3098581 register.go:491] Validating the existence of file /etc/karmada/karmada-agent.conf
I1107 20:23:59.964836 3098581 register.go:491] Validating the existence of file /etc/karmada/pki/ca.crt
[preflight] All pre-flight checks were passed
[karmada-agent-start] Waiting to perform the TLS Bootstrap
I1107 20:23:59.976730 3098581 register.go:1155] [discovery] Created cluster-info discovery client, requesting info from "172.18.0.6:32543"
I1107 20:23:59.988138 3098581 register.go:1193] [discovery] Requesting info from "172.18.0.6:32543" again to validate TLS against the pinned public key
I1107 20:23:59.997904 3098581 register.go:1210] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.18.0.6:32543"
I1107 20:23:59.997940 3098581 register.go:506] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I1107 20:23:59.997958 3098581 register.go:517] [discovery] writing bootstrap karmada-agent config file at /etc/karmada/bootstrap-karmada-agent.conf
I1107 20:23:59.998535 3098581 register.go:526] [discovery] writing CA certificate at /etc/karmada/pki/ca.crt
I1107 20:24:00.461989 3098581 register.go:866] Waiting for the client certificate agent-rbac-generator-member4wcq4v to be issued
I1107 20:24:01.465729 3098581 register.go:874] Signing certificate of csr agent-rbac-generator-member4wcq4v successfully
[karmada-agent-start] Waiting to check cluster exists
[karmada-agent-start] Assign the necessary RBAC permissions to the agent
[karmada-agent-start] Waiting to construct karmada-agent kubeconfig
I1107 20:24:03.665759 3098581 register.go:866] Waiting for the client certificate member4-nn9h2 to be issued
I1107 20:24:04.669051 3098581 register.go:874] Signing certificate of csr member4-nn9h2 successfully
I1107 20:24:04.669084 3098581 register.go:908] writing bootstrap karmada-agent config file at /etc/karmada/karmada-agent.conf
[karmada-agent-start] Waiting the necessary secret and RBAC
I1107 20:24:04.680320 3098581 idempotency.go:68] Secret karmada-system/karmada-kubeconfig has been created or updated.
I1107 20:24:04.683534 3098581 idempotency.go:188] ClusterRole karmada-agent has been created or updated.
I1107 20:24:04.691452 3098581 idempotency.go:212] ClusterRolebinding karmada-agent has been created or updated.
[karmada-agent-start] Waiting karmada-agent Deployment

cluster(member4) is joined successfully

$ kubectl get cluster
NAME      VERSION   MODE   READY   AGE
member1   v1.31.0   Push   True    3d10h
member2   v1.31.0   Push   True    6d6h
member3   v1.31.0   Pull   True    32h
member4   v1.31.0   Pull   True    34s

$ karmadactl apply -f samples/nginx/deployment.yaml --cluster member4
deployment.apps/nginx created
propagationpolicy.policy.karmada.io/nginx-6d7f8d5f5b unchanged

$ kubectl --kubeconfig ~/.kube/member4.config get deployments.apps 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   2/2     2            2           11s

$ karmadactl unregister member4  --cluster-kubeconfig ~/.kube/member4.config -v=4                                                                           
I1107 20:26:53.269394 3100053 unregister.go:186] Unregistering cluster. cluster name: member4
I1107 20:26:53.269597 3100053 unregister.go:187] Unregistering cluster. karmada-agent deployed in namespace: karmada-system
I1107 20:26:53.269608 3100053 unregister.go:188] Unregistering cluster. member cluster secrets stored in namespace: karmada-cluster
I1107 20:26:56.314363 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:26:57.317515 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:26:58.310547 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:26:59.310182 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:00.309665 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:01.314053 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:02.311988 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:03.309908 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:04.310080 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:05.309887 3100053 cluster.go:64] Waiting for the cluster object member4 to be deleted
I1107 20:27:06.312824 3100053 unregister.go:333] Successfully delete cluster object (member4) from control plane.
W1107 20:27:06.312863 3100053 unregister.go:342] The RBAC resources on the control plane need to be manually cleaned up, including the following resources:
ClusterRole/system:karmada:member4:agent
ClusterRoleBinding/system:karmada:member4:agent
Role/karmada-cluster/system:karmada:member4:agent-secret
Role/karmada-es-member4/system:karmada:member4:agent-work
RoleBinding/karmada-cluster/system:karmada:member4:agent-secret
RoleBinding/karmada-es-member4/system:karmada:member4:agent-work
I1107 20:27:06.316944 3100053 unregister.go:366] Successfully delete resource ({ServiceAccount karmada-member4 karmada-cluster}) from member cluster (member4).
I1107 20:27:06.320891 3100053 unregister.go:366] Successfully delete resource ({ClusterRole karmada-controller-manager:karmada-member4 }) from member cluster (member4).
I1107 20:27:06.323882 3100053 unregister.go:366] Successfully delete resource ({ClusterRoleBinding karmada-controller-manager:karmada-member4 }) from member cluster (member4).
I1107 20:27:06.325671 3100053 unregister.go:366] Successfully delete resource ({Secret karmada-member4 karmada-cluster}) from member cluster (member4).
I1107 20:27:06.328792 3100053 unregister.go:366] Successfully delete resource ({ServiceAccount karmada-impersonator karmada-cluster}) from member cluster (member4).
I1107 20:27:06.332187 3100053 unregister.go:366] Successfully delete resource ({Secret karmada-impersonator karmada-cluster}) from member cluster (member4).
I1107 20:27:06.335267 3100053 unregister.go:366] Successfully delete resource ({Namespace karmada-cluster }) from member cluster (member4).
I1107 20:27:06.339246 3100053 unregister.go:366] Successfully delete resource ({Deployment karmada-agent karmada-system}) from member cluster (member4).
I1107 20:27:06.342516 3100053 unregister.go:366] Successfully delete resource ({ServiceAccount karmada-agent-sa karmada-system}) from member cluster (member4).
I1107 20:27:06.346048 3100053 unregister.go:366] Successfully delete resource ({ClusterRole karmada-agent }) from member cluster (member4).
I1107 20:27:06.350296 3100053 unregister.go:366] Successfully delete resource ({ClusterRoleBinding karmada-agent }) from member cluster (member4).
I1107 20:27:06.354519 3100053 unregister.go:366] Successfully delete resource ({Secret karmada-kubeconfig karmada-system}) from member cluster (member4).
I1107 20:27:06.354622 3100053 unregister.go:382] Successfully delete local file (/etc/karmada/karmada-agent.conf) in current node.
I1107 20:27:06.354666 3100053 unregister.go:382] Successfully delete local file (/etc/karmada/pki/ca.crt) in current node.

@codecov-commenter
Copy link

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 57.51295% with 164 lines in your changes missing coverage. Please review.

Project coverage is 42.48%. Comparing base (0cc294f) to head (5f781cd).

Files with missing lines Patch % Lines
pkg/karmadactl/register/register.go 64.80% 107 Missing ⚠️
pkg/util/rbac.go 0.00% 28 Missing ⚠️
pkg/karmadactl/util/work.go 0.00% 18 Missing ⚠️
pkg/karmadactl/unregister/unregister.go 61.53% 8 Missing and 2 partials ⚠️
pkg/karmadactl/cmdinit/karmada/deploy.go 0.00% 1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5793      +/-   ##
==========================================
+ Coverage   42.41%   42.48%   +0.06%     
==========================================
  Files         656      657       +1     
  Lines       55884    56178     +294     
==========================================
+ Hits        23705    23867     +162     
- Misses      30659    30786     +127     
- Partials     1520     1525       +5     
Flag Coverage Δ
unittests 42.48% <57.51%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants