From c04529b4876b077cf3add7d1c6499f857b500d1b Mon Sep 17 00:00:00 2001 From: CrescentP <56831307+CrescentP@users.noreply.github.com> Date: Wed, 15 Sep 2021 16:59:06 +0800 Subject: [PATCH 1/5] =?UTF-8?q?Create=20TKEStack=20=E9=9B=86=E7=BE=A4CIS?= =?UTF-8?q?=E5=AE=89=E5=85=A8=E5=9F=BA=E7=BA=BF=E6=89=AB=E6=8F=8F=E5=88=86?= =?UTF-8?q?=E6=9E=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...1\253\346\217\217\345\210\206\346\236\220" | 409 ++++++++++++++++++ 1 file changed, 409 insertions(+) create mode 100644 "TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220" diff --git "a/TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220" "b/TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220" new file mode 100644 index 000000000..f4fe91b1b --- /dev/null +++ "b/TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220" @@ -0,0 +1,409 @@ +# TKEStack 集群CIS安全基线扫描分析 + +## *任务描述*： + +为了提供更安全的K8s集群，有必要使集群遵循安全的配置基线。 + +## *建议方案*： + +建议使用Center for Internet Security(CIS) benchmarks 来评估和分析集群的安全性。 + +## 具体流程 + +### 搭建TKEStack集群 + +1. 在`Installer `节点执行脚本 + + ```shell + arch=amd64 version=v1.3.1 && wget https://tke-release-1251707795.cos.ap-guangzhou.myqcloud.com/tke-installer-linux-$arch-$version.run{,.sha256} && sha256sum --check --status tke-installer-linux-$arch-$version.run.sha256 && chmod +x tke-installer-linux-$arch-$version.run && ./tke-installer-linux-$arch-$version.run + ``` + +2. 控制台安装 + + 按照提示好global集群 + + 最终可以得到这个 + + ![image-20210915153621127](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153621127.png) + +3. 访问控制台 + + ![image-20210915153800794](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153800794.png) + +### 创建业务集群 + +在集群管理中选择新建独立集群，填写相关信息即可 + +![image-20210915153919020](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153919020.png) + +### 使用 kube-bencht 扫描集群 + +Kube-Bench是一款针对Kubernete的安全检测工具，从本质上来说，Kube-Bench是一个基于Go开发的应用程序，它可以帮助研究人员对部署的Kubernete进行安全检测，安全检测原则遵循CIS Kubernetes Benchmark。 + +#### 安装部署 + +- 二进制安装 + + ```shell + wget https://github.com/aquasecurity/kube-bench/releases/download/v0.4.0/kube-bench_0.4.0_linux_amd64.tar.gz + tar -zxvf kube-bench_0.4.0_linux_amd64.tar.gz + mv kube-bench /usr/bin/ + ``` + +- + + ```shell + docker run --rm -v `pwd`:/host aquasec/kube-bench:latest install[root@VM-0-11-centos cis-1.6]# ls /etc/kube-bench/cfg/ + ack-1.0 aks-1.0 cis-1.20 cis-1.5 cis-1.6 config.yaml eks-1.0 gke-1.0 rh-0.7 rh-1.0 + # 根据这些文件对集群当中的组件扫描已有的配置 + [root@VM-0-11-centos cis-1.6]# cd /etc/kube-bench/cfg/cis-1.6 + [root@VM-0-11-centos cis-1.6]# ls + config.yaml controlplane.yaml etcd.yaml master.yaml node.yaml policies.yaml + ``` + +### 扫描结果 + +#### master + +```shell +[root@VM-0-11-centos cis-1.6]# kube-bench run --targets=master +[INFO] 1 Master Node Security Configuration +[INFO] 1.1 Master Node Configuration Files +[PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) +[PASS] 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) +[PASS] 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) +[PASS] 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) +[WARN] 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) +[PASS] 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) +[PASS] 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) +[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) +[PASS] 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) +[PASS] 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) +[PASS] 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) +[PASS] 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) +[PASS] 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) +[WARN] 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) +[INFO] 1.2 API Server +[WARN] 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) +[PASS] 1.2.2 Ensure that the --basic-auth-file argument is not set (Automated) +[FAIL] 1.2.3 Ensure that the --token-auth-file parameter is not set (Automated) +[PASS] 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) +[PASS] 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) +[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) +[PASS] 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) +[PASS] 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) +[PASS] 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) +[WARN] 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) +[PASS] 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) +[WARN] 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) +[WARN] 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) +[PASS] 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) +[PASS] 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) +[FAIL] 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) +[PASS] 1.2.17 Ensure that the admission control plugin NodeRestriction is set (Automated) +[PASS] 1.2.18 Ensure that the --insecure-bind-address argument is not set (Automated) +[PASS] 1.2.19 Ensure that the --insecure-port argument is set to 0 (Automated) +[PASS] 1.2.20 Ensure that the --secure-port argument is not set to 0 (Automated) +[FAIL] 1.2.21 Ensure that the --profiling argument is set to false (Automated) +[FAIL] 1.2.22 Ensure that the --audit-log-path argument is set (Automated) +[FAIL] 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) +[FAIL] 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) +[FAIL] 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) +[WARN] 1.2.26 Ensure that the --request-timeout argument is set as appropriate (Automated) +[PASS] 1.2.27 Ensure that the --service-account-lookup argument is set to true (Automated) +[PASS] 1.2.28 Ensure that the --service-account-key-file argument is set as appropriate (Automated) +[PASS] 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) +[PASS] 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) +[PASS] 1.2.31 Ensure that the --client-ca-file argument is set as appropriate (Automated) +[PASS] 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate (Automated) +[WARN] 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) +[WARN] 1.2.34 Ensure that encryption providers are appropriately configured (Manual) +[WARN] 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) +[INFO] 1.3 Controller Manager +[WARN] 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) +[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated) +[PASS] 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) +[PASS] 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) +[PASS] 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) +[PASS] 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) +[PASS] 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) +[INFO] 1.4 Scheduler +[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated) +[PASS] 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + +== Remediations == +1.1.9 Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 + +1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the below command: +ps -ef | grep etcd +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +1.1.21 Run the below command (based on the file location on your system) on the master node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +1.2.1 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--anonymous-auth=false + +1.2.3 Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --token-auth-file= parameter. + +1.2.6 Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +1.2.10 Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +1.2.12 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +1.2.13 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +1.2.16 Follow the documentation and create Pod Security Policy objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to a +value that includes PodSecurityPolicy: +--enable-admission-plugins=...,PodSecurityPolicy,... +Then restart the API Server. + +1.2.21 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--profiling=false + +1.2.22 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example: +--audit-log-path=/var/log/apiserver/audit.log + +1.2.23 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxage parameter to 30 or as an appropriate number of days: +--audit-log-maxage=30 + +1.2.24 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. +--audit-log-maxbackup=10 + +1.2.25 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB: +--audit-log-maxsize=100 + +1.2.26 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, +--request-timeout=300s + +1.2.33 Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --encryption-provider-config parameter to the path of that file: --encryption-provider-config= + +1.2.34 Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +1.2.35 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM +_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM +_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM +_SHA384 + +1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example: +--terminated-pod-gc-threshold=10 + +1.3.2 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the below parameter. +--profiling=false + +1.4.1 Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the master node and set the below parameter. +--profiling=false + + +== Summary == +43 checks PASS +11 checks FAIL +11 checks WARN +0 checks INFO + +``` + +#### node + +```shell +[root@VM-0-11-centos cis-1.6]# kube-bench run --targets=node +[INFO] 4 Worker Node Security Configuration +[INFO] 4.1 Worker Node Configuration Files +[PASS] 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) +[PASS] 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) +[PASS] 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) +[PASS] 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Manual) +[PASS] 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) +[PASS] 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Manual) +[PASS] 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) +[PASS] 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) +[PASS] 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) +[PASS] 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) +[INFO] 4.2 Kubelet +[PASS] 4.2.1 Ensure that the anonymous-auth argument is set to false (Automated) +[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) +[PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) +[PASS] 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) +[PASS] 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) +[FAIL] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) +[PASS] 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) +[WARN] 4.2.8 Ensure that the --hostname-override argument is not set (Manual) +[WARN] 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) +[WARN] 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) +[PASS] 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) +[PASS] 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) +[WARN] 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + +== Remediations == +4.2.6 If using a Kubelet config file, edit the file to set protectKernelDefaults: true. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +4.2.8 Edit the kubelet service file /etc/systemd/system/kubelet.service +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +4.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +4.2.10 If using a Kubelet config file, edit the file to set tlsCertFile to the location +of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +4.2.13 If using a Kubelet config file, edit the file to set TLSCipherSuites: to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + + +== Summary == +18 checks PASS +1 checks FAIL +4 checks WARN +0 checks INFO + +``` + +### 结果分析 + +我们按照CIS Kubernetes Benchmark进行分析 + +#### 1.1.12 + +![image-20210915162536350](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162536350.png) + +#### 1.2.3 + +![image-20210915162614233](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162614233.png) + +#### 1.2.6 + +![image-20210915162642312](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162642312.png) + +#### 1.2.16 + +![image-20210915162714613](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162714613.png) + +#### 1.2.21 + +![image-20210915162757078](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162757078.png) + +#### 1.2.22 + +![image-20210915162841148](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162841148.png) + +#### 1.2.23 + +![image-20210915162848073](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162848073.png) + +#### 1.2.24 + +![image-20210915162855325](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162855325.png) + +#### 1.2.25 + +#### ![image-20210915162902354](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162902354.png) + +#### 1.3.2 + +![image-20210915162933815](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162933815.png) + +#### 1.4.1 + +![image-20210915162955259](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162955259.png) + +#### 4.2.6 + +![image-20210915163028273](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915163028273.png) + +## 结语 + +以上资料参考 + +https://tkestack.github.io/docs/ + +https://cloud.tencent.com/developer/article/1680433 + +https://www.cisecurity.org/benchmark/kubernetes/ + +欢迎大家参与到TKEStack的建设中：https://github.com/tkestack + +**在这里十分感谢TKEStack的各位导师给我提供的资源和帮助** From 8d8642ec0a699934205e21b5ff80891a17aef5d6 Mon Sep 17 00:00:00 2001 From: CrescentP <56831307+CrescentP@users.noreply.github.com> Date: Wed, 15 Sep 2021 16:59:33 +0800 Subject: [PATCH 2/5] =?UTF-8?q?Rename=20TKEStack=20=E9=9B=86=E7=BE=A4CIS?= =?UTF-8?q?=E5=AE=89=E5=85=A8=E5=9F=BA=E7=BA=BF=E6=89=AB=E6=8F=8F=E5=88=86?= =?UTF-8?q?=E6=9E=90=20to=20TKEStack=20=E9=9B=86=E7=BE=A4CIS=E5=AE=89?= =?UTF-8?q?=E5=85=A8=E5=9F=BA=E7=BA=BF=E6=89=AB=E6=8F=8F=E5=88=86=E6=9E=90?= =?UTF-8?q?.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...47\272\277\346\211\253\346\217\217\345\210\206\346\236\220.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename "TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220" => "TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220.md" (100%) diff --git "a/TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220" "b/TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220.md" similarity index 100% rename from "TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220" rename to "TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220.md" From fec20a168408da5578aa7697c9741899606e9975 Mon Sep 17 00:00:00 2001 From: CrescentP <56831307+CrescentP@users.noreply.github.com> Date: Wed, 15 Sep 2021 17:45:21 +0800 Subject: [PATCH 3/5] =?UTF-8?q?Update=20and=20rename=20TKEStack=20?= =?UTF-8?q?=E9=9B=86=E7=BE=A4CIS=E5=AE=89=E5=85=A8=E5=9F=BA=E7=BA=BF?= =?UTF-8?q?=E6=89=AB=E6=8F=8F=E5=88=86=E6=9E=90.md=20to=20Center=20for=20I?= =?UTF-8?q?nternet=20Security=20(CIS)=20benchmarks=20to=20evaluate=20and?= =?UTF-8?q?=20analyze=20the=20security=20of=20cluster?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...) benchmarks to evaluate and analyze the security of cluster | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename "TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220.md" => Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster (99%) diff --git "a/TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220.md" b/Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster similarity index 99% rename from "TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220.md" rename to Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster index f4fe91b1b..e91fda666 100644 --- "a/TKEStack \351\233\206\347\276\244CIS\345\256\211\345\205\250\345\237\272\347\272\277\346\211\253\346\217\217\345\210\206\346\236\220.md" +++ b/Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster @@ -1,4 +1,4 @@ -# TKEStack 集群CIS安全基线扫描分析 +# Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster ## *任务描述*： From 8dd48b82b76353003f1a36c0b1e8f20984263f51 Mon Sep 17 00:00:00 2001 From: CrescentP <56831307+CrescentP@users.noreply.github.com> Date: Thu, 16 Sep 2021 14:52:46 +0800 Subject: [PATCH 4/5] Delete Test --- ...aluate and analyze the security of cluster | 409 ------------------ 1 file changed, 409 deletions(-) delete mode 100644 Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster diff --git a/Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster b/Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster deleted file mode 100644 index e91fda666..000000000 --- a/Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster +++ /dev/null @@ -1,409 +0,0 @@ -# Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster - -## *任务描述*： - -为了提供更安全的K8s集群，有必要使集群遵循安全的配置基线。 - -## *建议方案*： - -建议使用Center for Internet Security(CIS) benchmarks 来评估和分析集群的安全性。 - -## 具体流程 - -### 搭建TKEStack集群 - -1. 在`Installer `节点执行脚本 - - ```shell - arch=amd64 version=v1.3.1 && wget https://tke-release-1251707795.cos.ap-guangzhou.myqcloud.com/tke-installer-linux-$arch-$version.run{,.sha256} && sha256sum --check --status tke-installer-linux-$arch-$version.run.sha256 && chmod +x tke-installer-linux-$arch-$version.run && ./tke-installer-linux-$arch-$version.run - ``` - -2. 控制台安装 - - 按照提示好global集群 - - 最终可以得到这个 - - ![image-20210915153621127](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153621127.png) - -3. 访问控制台 - - ![image-20210915153800794](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153800794.png) - -### 创建业务集群 - -在集群管理中选择新建独立集群，填写相关信息即可 - -![image-20210915153919020](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153919020.png) - -### 使用 kube-bencht 扫描集群 - -Kube-Bench是一款针对Kubernete的安全检测工具，从本质上来说，Kube-Bench是一个基于Go开发的应用程序，它可以帮助研究人员对部署的Kubernete进行安全检测，安全检测原则遵循CIS Kubernetes Benchmark。 - -#### 安装部署 - -- 二进制安装 - - ```shell - wget https://github.com/aquasecurity/kube-bench/releases/download/v0.4.0/kube-bench_0.4.0_linux_amd64.tar.gz - tar -zxvf kube-bench_0.4.0_linux_amd64.tar.gz - mv kube-bench /usr/bin/ - ``` - -- - - ```shell - docker run --rm -v `pwd`:/host aquasec/kube-bench:latest install[root@VM-0-11-centos cis-1.6]# ls /etc/kube-bench/cfg/ - ack-1.0 aks-1.0 cis-1.20 cis-1.5 cis-1.6 config.yaml eks-1.0 gke-1.0 rh-0.7 rh-1.0 - # 根据这些文件对集群当中的组件扫描已有的配置 - [root@VM-0-11-centos cis-1.6]# cd /etc/kube-bench/cfg/cis-1.6 - [root@VM-0-11-centos cis-1.6]# ls - config.yaml controlplane.yaml etcd.yaml master.yaml node.yaml policies.yaml - ``` - -### 扫描结果 - -#### master - -```shell -[root@VM-0-11-centos cis-1.6]# kube-bench run --targets=master -[INFO] 1 Master Node Security Configuration -[INFO] 1.1 Master Node Configuration Files -[PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) -[PASS] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) -[PASS] 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) -[PASS] 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) -[PASS] 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) -[PASS] 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) -[PASS] 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) -[PASS] 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) -[WARN] 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) -[PASS] 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) -[PASS] 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) -[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) -[PASS] 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) -[PASS] 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) -[PASS] 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) -[PASS] 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) -[PASS] 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) -[PASS] 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) -[PASS] 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) -[PASS] 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) -[WARN] 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) -[INFO] 1.2 API Server -[WARN] 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) -[PASS] 1.2.2 Ensure that the --basic-auth-file argument is not set (Automated) -[FAIL] 1.2.3 Ensure that the --token-auth-file parameter is not set (Automated) -[PASS] 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) -[PASS] 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) -[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) -[PASS] 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) -[PASS] 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) -[PASS] 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) -[WARN] 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) -[PASS] 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) -[WARN] 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) -[WARN] 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) -[PASS] 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) -[PASS] 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) -[FAIL] 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) -[PASS] 1.2.17 Ensure that the admission control plugin NodeRestriction is set (Automated) -[PASS] 1.2.18 Ensure that the --insecure-bind-address argument is not set (Automated) -[PASS] 1.2.19 Ensure that the --insecure-port argument is set to 0 (Automated) -[PASS] 1.2.20 Ensure that the --secure-port argument is not set to 0 (Automated) -[FAIL] 1.2.21 Ensure that the --profiling argument is set to false (Automated) -[FAIL] 1.2.22 Ensure that the --audit-log-path argument is set (Automated) -[FAIL] 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) -[FAIL] 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) -[FAIL] 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) -[WARN] 1.2.26 Ensure that the --request-timeout argument is set as appropriate (Automated) -[PASS] 1.2.27 Ensure that the --service-account-lookup argument is set to true (Automated) -[PASS] 1.2.28 Ensure that the --service-account-key-file argument is set as appropriate (Automated) -[PASS] 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) -[PASS] 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) -[PASS] 1.2.31 Ensure that the --client-ca-file argument is set as appropriate (Automated) -[PASS] 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate (Automated) -[WARN] 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) -[WARN] 1.2.34 Ensure that encryption providers are appropriately configured (Manual) -[WARN] 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) -[INFO] 1.3 Controller Manager -[WARN] 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) -[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated) -[PASS] 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) -[PASS] 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) -[PASS] 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) -[PASS] 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) -[PASS] 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) -[INFO] 1.4 Scheduler -[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated) -[PASS] 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - -== Remediations == -1.1.9 Run the below command (based on the file location on your system) on the master node. -For example, -chmod 644 - -1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the below command: -ps -ef | grep etcd -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -1.1.21 Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 600 /etc/kubernetes/pki/*.key - -1.2.1 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the below parameter. ---anonymous-auth=false - -1.2.3 Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and remove the --token-auth-file= parameter. - -1.2.6 Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -1.2.10 Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -1.2.12 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -1.2.13 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -1.2.16 Follow the documentation and create Pod Security Policy objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --enable-admission-plugins parameter to a -value that includes PodSecurityPolicy: ---enable-admission-plugins=...,PodSecurityPolicy,... -Then restart the API Server. - -1.2.21 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the below parameter. ---profiling=false - -1.2.22 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example: ---audit-log-path=/var/log/apiserver/audit.log - -1.2.23 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --audit-log-maxage parameter to 30 or as an appropriate number of days: ---audit-log-maxage=30 - -1.2.24 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. ---audit-log-maxbackup=10 - -1.2.25 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB: ---audit-log-maxsize=100 - -1.2.26 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameter as appropriate and if needed. -For example, ---request-timeout=300s - -1.2.33 Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the --encryption-provider-config parameter to the path of that file: --encryption-provider-config= - -1.2.34 Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -1.2.35 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the master node and set the below parameter. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM -_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM -_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM -_SHA384 - -1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example: ---terminated-pod-gc-threshold=10 - -1.3.2 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the master node and set the below parameter. ---profiling=false - -1.4.1 Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the master node and set the below parameter. ---profiling=false - - -== Summary == -43 checks PASS -11 checks FAIL -11 checks WARN -0 checks INFO - -``` - -#### node - -```shell -[root@VM-0-11-centos cis-1.6]# kube-bench run --targets=node -[INFO] 4 Worker Node Security Configuration -[INFO] 4.1 Worker Node Configuration Files -[PASS] 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) -[PASS] 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) -[PASS] 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) -[PASS] 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Manual) -[PASS] 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) -[PASS] 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Manual) -[PASS] 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) -[PASS] 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) -[PASS] 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) -[PASS] 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) -[INFO] 4.2 Kubelet -[PASS] 4.2.1 Ensure that the anonymous-auth argument is set to false (Automated) -[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) -[PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) -[PASS] 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) -[PASS] 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) -[FAIL] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) -[PASS] 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) -[WARN] 4.2.8 Ensure that the --hostname-override argument is not set (Manual) -[WARN] 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) -[WARN] 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) -[PASS] 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) -[PASS] 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) -[WARN] 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - -== Remediations == -4.2.6 If using a Kubelet config file, edit the file to set protectKernelDefaults: true. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -4.2.8 Edit the kubelet service file /etc/systemd/system/kubelet.service -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -4.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -4.2.10 If using a Kubelet config file, edit the file to set tlsCertFile to the location -of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -4.2.13 If using a Kubelet config file, edit the file to set TLSCipherSuites: to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - - -== Summary == -18 checks PASS -1 checks FAIL -4 checks WARN -0 checks INFO - -``` - -### 结果分析 - -我们按照CIS Kubernetes Benchmark进行分析 - -#### 1.1.12 - -![image-20210915162536350](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162536350.png) - -#### 1.2.3 - -![image-20210915162614233](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162614233.png) - -#### 1.2.6 - -![image-20210915162642312](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162642312.png) - -#### 1.2.16 - -![image-20210915162714613](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162714613.png) - -#### 1.2.21 - -![image-20210915162757078](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162757078.png) - -#### 1.2.22 - -![image-20210915162841148](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162841148.png) - -#### 1.2.23 - -![image-20210915162848073](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162848073.png) - -#### 1.2.24 - -![image-20210915162855325](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162855325.png) - -#### 1.2.25 - -#### ![image-20210915162902354](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162902354.png) - -#### 1.3.2 - -![image-20210915162933815](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162933815.png) - -#### 1.4.1 - -![image-20210915162955259](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162955259.png) - -#### 4.2.6 - -![image-20210915163028273](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915163028273.png) - -## 结语 - -以上资料参考 - -https://tkestack.github.io/docs/ - -https://cloud.tencent.com/developer/article/1680433 - -https://www.cisecurity.org/benchmark/kubernetes/ - -欢迎大家参与到TKEStack的建设中：https://github.com/tkestack - -**在这里十分感谢TKEStack的各位导师给我提供的资源和帮助** From 0f8d019d107b438ce1d4d13c6ff709c2b01de0c6 Mon Sep 17 00:00:00 2001 From: CrescentP <56831307+CrescentP@users.noreply.github.com> Date: Thu, 16 Sep 2021 14:58:46 +0800 Subject: [PATCH 5/5] CIS Center for Internet Security (CIS) benchmarks to evaluate and analyze the security of cluster --- CISbenchmarks.md | 409 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 409 insertions(+) create mode 100644 CISbenchmarks.md diff --git a/CISbenchmarks.md b/CISbenchmarks.md new file mode 100644 index 000000000..f4fe91b1b --- /dev/null +++ b/CISbenchmarks.md @@ -0,0 +1,409 @@ +# TKEStack 集群CIS安全基线扫描分析 + +## *任务描述*： + +为了提供更安全的K8s集群，有必要使集群遵循安全的配置基线。 + +## *建议方案*： + +建议使用Center for Internet Security(CIS) benchmarks 来评估和分析集群的安全性。 + +## 具体流程 + +### 搭建TKEStack集群 + +1. 在`Installer `节点执行脚本 + + ```shell + arch=amd64 version=v1.3.1 && wget https://tke-release-1251707795.cos.ap-guangzhou.myqcloud.com/tke-installer-linux-$arch-$version.run{,.sha256} && sha256sum --check --status tke-installer-linux-$arch-$version.run.sha256 && chmod +x tke-installer-linux-$arch-$version.run && ./tke-installer-linux-$arch-$version.run + ``` + +2. 控制台安装 + + 按照提示好global集群 + + 最终可以得到这个 + + ![image-20210915153621127](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153621127.png) + +3. 访问控制台 + + ![image-20210915153800794](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153800794.png) + +### 创建业务集群 + +在集群管理中选择新建独立集群，填写相关信息即可 + +![image-20210915153919020](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915153919020.png) + +### 使用 kube-bencht 扫描集群 + +Kube-Bench是一款针对Kubernete的安全检测工具，从本质上来说，Kube-Bench是一个基于Go开发的应用程序，它可以帮助研究人员对部署的Kubernete进行安全检测，安全检测原则遵循CIS Kubernetes Benchmark。 + +#### 安装部署 + +- 二进制安装 + + ```shell + wget https://github.com/aquasecurity/kube-bench/releases/download/v0.4.0/kube-bench_0.4.0_linux_amd64.tar.gz + tar -zxvf kube-bench_0.4.0_linux_amd64.tar.gz + mv kube-bench /usr/bin/ + ``` + +- + + ```shell + docker run --rm -v `pwd`:/host aquasec/kube-bench:latest install[root@VM-0-11-centos cis-1.6]# ls /etc/kube-bench/cfg/ + ack-1.0 aks-1.0 cis-1.20 cis-1.5 cis-1.6 config.yaml eks-1.0 gke-1.0 rh-0.7 rh-1.0 + # 根据这些文件对集群当中的组件扫描已有的配置 + [root@VM-0-11-centos cis-1.6]# cd /etc/kube-bench/cfg/cis-1.6 + [root@VM-0-11-centos cis-1.6]# ls + config.yaml controlplane.yaml etcd.yaml master.yaml node.yaml policies.yaml + ``` + +### 扫描结果 + +#### master + +```shell +[root@VM-0-11-centos cis-1.6]# kube-bench run --targets=master +[INFO] 1 Master Node Security Configuration +[INFO] 1.1 Master Node Configuration Files +[PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) +[PASS] 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) +[PASS] 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) +[PASS] 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) +[WARN] 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) +[PASS] 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) +[PASS] 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) +[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) +[PASS] 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) +[PASS] 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) +[PASS] 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) +[PASS] 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) +[PASS] 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) +[PASS] 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) +[WARN] 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) +[INFO] 1.2 API Server +[WARN] 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) +[PASS] 1.2.2 Ensure that the --basic-auth-file argument is not set (Automated) +[FAIL] 1.2.3 Ensure that the --token-auth-file parameter is not set (Automated) +[PASS] 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) +[PASS] 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) +[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) +[PASS] 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) +[PASS] 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) +[PASS] 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) +[WARN] 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) +[PASS] 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) +[WARN] 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) +[WARN] 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) +[PASS] 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) +[PASS] 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) +[FAIL] 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) +[PASS] 1.2.17 Ensure that the admission control plugin NodeRestriction is set (Automated) +[PASS] 1.2.18 Ensure that the --insecure-bind-address argument is not set (Automated) +[PASS] 1.2.19 Ensure that the --insecure-port argument is set to 0 (Automated) +[PASS] 1.2.20 Ensure that the --secure-port argument is not set to 0 (Automated) +[FAIL] 1.2.21 Ensure that the --profiling argument is set to false (Automated) +[FAIL] 1.2.22 Ensure that the --audit-log-path argument is set (Automated) +[FAIL] 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) +[FAIL] 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) +[FAIL] 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) +[WARN] 1.2.26 Ensure that the --request-timeout argument is set as appropriate (Automated) +[PASS] 1.2.27 Ensure that the --service-account-lookup argument is set to true (Automated) +[PASS] 1.2.28 Ensure that the --service-account-key-file argument is set as appropriate (Automated) +[PASS] 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) +[PASS] 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) +[PASS] 1.2.31 Ensure that the --client-ca-file argument is set as appropriate (Automated) +[PASS] 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate (Automated) +[WARN] 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) +[WARN] 1.2.34 Ensure that encryption providers are appropriately configured (Manual) +[WARN] 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) +[INFO] 1.3 Controller Manager +[WARN] 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) +[FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated) +[PASS] 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) +[PASS] 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) +[PASS] 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) +[PASS] 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) +[PASS] 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) +[INFO] 1.4 Scheduler +[FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated) +[PASS] 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + +== Remediations == +1.1.9 Run the below command (based on the file location on your system) on the master node. +For example, +chmod 644 + +1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the below command: +ps -ef | grep etcd +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +1.1.21 Run the below command (based on the file location on your system) on the master node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +1.2.1 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--anonymous-auth=false + +1.2.3 Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and remove the --token-auth-file= parameter. + +1.2.6 Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +1.2.10 Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +1.2.12 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +1.2.13 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +1.2.16 Follow the documentation and create Pod Security Policy objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --enable-admission-plugins parameter to a +value that includes PodSecurityPolicy: +--enable-admission-plugins=...,PodSecurityPolicy,... +Then restart the API Server. + +1.2.21 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--profiling=false + +1.2.22 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example: +--audit-log-path=/var/log/apiserver/audit.log + +1.2.23 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxage parameter to 30 or as an appropriate number of days: +--audit-log-maxage=30 + +1.2.24 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. +--audit-log-maxbackup=10 + +1.2.25 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB: +--audit-log-maxsize=100 + +1.2.26 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, +--request-timeout=300s + +1.2.33 Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the --encryption-provider-config parameter to the path of that file: --encryption-provider-config= + +1.2.34 Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +1.2.35 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the master node and set the below parameter. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM +_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM +_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM +_SHA384 + +1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example: +--terminated-pod-gc-threshold=10 + +1.3.2 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the master node and set the below parameter. +--profiling=false + +1.4.1 Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the master node and set the below parameter. +--profiling=false + + +== Summary == +43 checks PASS +11 checks FAIL +11 checks WARN +0 checks INFO + +``` + +#### node + +```shell +[root@VM-0-11-centos cis-1.6]# kube-bench run --targets=node +[INFO] 4 Worker Node Security Configuration +[INFO] 4.1 Worker Node Configuration Files +[PASS] 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) +[PASS] 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) +[PASS] 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) +[PASS] 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Manual) +[PASS] 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) +[PASS] 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Manual) +[PASS] 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) +[PASS] 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) +[PASS] 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) +[PASS] 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) +[INFO] 4.2 Kubelet +[PASS] 4.2.1 Ensure that the anonymous-auth argument is set to false (Automated) +[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) +[PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) +[PASS] 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) +[PASS] 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) +[FAIL] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) +[PASS] 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) +[WARN] 4.2.8 Ensure that the --hostname-override argument is not set (Manual) +[WARN] 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) +[WARN] 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) +[PASS] 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) +[PASS] 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) +[WARN] 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + +== Remediations == +4.2.6 If using a Kubelet config file, edit the file to set protectKernelDefaults: true. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +4.2.8 Edit the kubelet service file /etc/systemd/system/kubelet.service +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +4.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +4.2.10 If using a Kubelet config file, edit the file to set tlsCertFile to the location +of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +4.2.13 If using a Kubelet config file, edit the file to set TLSCipherSuites: to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + + +== Summary == +18 checks PASS +1 checks FAIL +4 checks WARN +0 checks INFO + +``` + +### 结果分析 + +我们按照CIS Kubernetes Benchmark进行分析 + +#### 1.1.12 + +![image-20210915162536350](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162536350.png) + +#### 1.2.3 + +![image-20210915162614233](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162614233.png) + +#### 1.2.6 + +![image-20210915162642312](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162642312.png) + +#### 1.2.16 + +![image-20210915162714613](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162714613.png) + +#### 1.2.21 + +![image-20210915162757078](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162757078.png) + +#### 1.2.22 + +![image-20210915162841148](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162841148.png) + +#### 1.2.23 + +![image-20210915162848073](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162848073.png) + +#### 1.2.24 + +![image-20210915162855325](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162855325.png) + +#### 1.2.25 + +#### ![image-20210915162902354](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162902354.png) + +#### 1.3.2 + +![image-20210915162933815](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162933815.png) + +#### 1.4.1 + +![image-20210915162955259](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915162955259.png) + +#### 4.2.6 + +![image-20210915163028273](https://gitee.com/Crescent_P/picture-bed/raw/master/image-20210915163028273.png) + +## 结语 + +以上资料参考 + +https://tkestack.github.io/docs/ + +https://cloud.tencent.com/developer/article/1680433 + +https://www.cisecurity.org/benchmark/kubernetes/ + +欢迎大家参与到TKEStack的建设中：https://github.com/tkestack + +**在这里十分感谢TKEStack的各位导师给我提供的资源和帮助**