From b09766ae403aec12757bc23178b5e688eddaa255 Mon Sep 17 00:00:00 2001 From: dazmc <87435100+dazmc@users.noreply.github.com> Date: Wed, 22 Jan 2025 16:45:06 +0000 Subject: [PATCH] kcli - eks workflow (#773) --- docs/EKS-kcli.md | 369 +++++++++++++++++++++++++++++++++++++++++++++++ docs/index.md | 9 +- docs/index.rst | 8 +- 3 files changed, 384 insertions(+), 2 deletions(-) create mode 100644 docs/EKS-kcli.md diff --git a/docs/EKS-kcli.md b/docs/EKS-kcli.md new file mode 100644 index 000000000..0134b394f --- /dev/null +++ b/docs/EKS-kcli.md @@ -0,0 +1,369 @@ +# EKS Workflow + +This page highlights an example workflow using kcli with AWS EKS. In the workflow it uses `myeks` as the clustername and AWS keypair name. It was deployed and tested in `us-east-1` region (but should work in other regions). + +This workflow also introduces concepts around EKS that you might not be familiar with. + +- EKS Auto Mode - released 2024 +- EKS byocni (Bring Your Own CNI) - released 2024 +- Pod identity - release Nov 2023. +- cilium CNI (instead of AWS)* + +*other CNIs can be used + +## Pre-Requisite tools and Requirements + +1. AWS cli +2. helm cli +3. cilium cli (optional) +4. kcli +5. boto3 and dependencies +6. kubectl cli +7. An AWS account + +## AWS Role Creation (optional) + +Specific role(s) can be created in AWS which can be used by kcli in the provisioning process for the control plane and/or worker nodes if you have specific requirements. + +If you don't create role(s) default ones will be created by kcli in the cluster creation step. + +- Example Roles and attached policies for control plane and worker nodes + + + 1. ROLE name = eks-ctlplane (with eks.amazonaws.com trust relationship) + + - AmazonEBSCSIDriverPolicy + - AmazonEC2ContainerRegistryReadOnly + - AmazonEKS_CNI_Policy + - AmazonEKSBlockStoragePolicy + - AmazonEKSClusterPolicy + + 2. ROLE name = eks-worker (with ec2.amazonaws.com trust relationship) + + - AmazonEBSCSIDriverPolicy + - AmazonEC2ContainerRegistryReadOnly + - AmazonEKS_CNI_Policy + - AmazonEKSBlockStoragePolicy + - AmazonEKSWorkerNodePolicy + +## Create Cluster + +OS distributions aren't always up to date with the latest/newer boto3 and botocore versions from AWS. This means that at times kcli will fail due to missing features because of the installed OS version of boto3/botocore. + +So this example workflow uses a virtual environment to ensure you have the latest versions. + +- Create virtual environment and install kcli/boto3 +```bash +mkdir kcli +cd kcli +python3 -m venv kcli-boto3 +source kcli-boto3/bin/activate +pip list +pip install setuptools wheel +pip3 install kcli +pip install boto3 +pip list +kcli version +#when finished with the virtual environment +#deactivate +``` + +- Update aws credentials in kcli config file and variables +```bash +vim ~/.kcli/config.yml + aws: + type: aws + access_key_id: XXXXXXXXXXXX + access_key_secret: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY + region: us-east-1 + keypair: myekskey + +# you might choose to update your AWS profile +#aws configure --profile kcli +#export AWS_PROFILE=kcli + +export AWS_ACCESS_KEY_ID=$(grep access_key_id ~/.kcli/config.yml| awk -F: '{print $2}'| sed 's/ //g') +export AWS_SECRET_ACCESS_KEY=$(grep access_key_secret ~/.kcli/config.yml| awk -F: '{print $2}'| sed 's/ //g') +``` + +- Test AWS credentials + +Make sure AWS is functioning - this command should either return nothing or your buckets, but shouldn't fail if the environment variables have been set correctly. + +```bash +aws s3 ls + ``` + +- Import keypair into AWS + +In the ~/.kcli/config.yaml you specified a key that will be injected into the deployed cluster. So here import the key or if you already have a key in AWS then you can skip this, just make sure you reference the existing AWS key name in the config.yaml file. + +```bash +# ssh-keygen -b 4096 -t rsa or ssh-keygen -b 4096 -t ed25519 etc... +aws ec2 import-key-pair --key-name "myekskey" --public-key-material fileb://~/.ssh/id_rsa.pub +``` + +- Create cluster (BYOCNI) + +In this workflow we pass the parameters as arguments to the kcli create command. Alternatively you could add them to a parameter file and call the parameter file with the kcli create command. + +Alternative kcli create cluster command examples (ymmv):- + +> *You can create a cluster with AWS-CNI like this to have kcli deploy standard set of addons this will also deploy the cluster in Auto Mode* + +> `kcli create kube eks myeks` + +> *You could also deploy AWS-CNI without Auto Mode and some addons like this* + +> `kcli create kube eks -P default_addons=false -P apps=[coredns,vpc-cni,kube-proxy,aws-ebs-csi-driver,eks-pod-identity-agent] myeks` + +> *Here you add additional arguments to specify separate roles for control plane and workers* + +> `kcli create kube eks -P subnet=subnet-0f4e6564cd0052731 -P extra_subnets=[subnet-0ea77dff7cd0dda6d] -P default_addons=False -P apps=[coredns,vpc-cni,kube-proxy,aws-ebs-csi-driver,eks-pod-identity-agent] -P ctlplane_role=eks-ctlplane -P worker_role=eks-worker myeks` + +For this workflow we will create a BYOCNI cluster (no aws-cni or kube-proxy), disable Auto Mode and install cilium on the cluster. + +```bash +# list available subnets. Choose from the list and use them as arguments to kcli create or let kcli autodetect when creating. + +kcli list subnets + +# you will see something like this and can pick the subnets you want to use in the create command + +Listing Subnets... ++--------------------------+------------+----------------+--------------------------+-----------------------+ +| Subnet | Zone | Cidr | Id | Network | ++--------------------------+------------+----------------+--------------------------+-----------------------+ +| subnet-00455945f5834b4f6 | us-east-1d | 172.31.32.0/20 | subnet-00455945f5834b4f6 | vpc-0a577c7b378354b02 | +| subnet-01a28a456b7587e91 | us-east-1a | 172.31.0.0/20 | subnet-01a28a456b7587e91 | vpc-0a577c7b378354b02 | +| subnet-01bf43e04191244d4 | us-east-1b | 172.31.80.0/20 | subnet-01bf43e04191244d4 | vpc-0a577c7b378354b02 | +| subnet-031627ea5f56a0fda | us-east-1f | 172.31.64.0/20 | subnet-031627ea5f56a0fda | vpc-0a577c7b378354b02 | +| subnet-0707e63fab3c1eec4 | us-east-1c | 172.31.16.0/20 | subnet-0707e63fab3c1eec4 | vpc-0a577c7b378354b02 | +| subnet-0d69ee6b2a02fb637 | us-east-1e | 172.31.48.0/20 | subnet-0d69ee6b2a02fb637 | vpc-0a577c7b378354b02 | ++--------------------------+------------+----------------+--------------------------+-----------------------+ + +# to list cluster parameters you can then choose which ones you want to alter in create command using a parameter file or pass as an arguments (-P) + +kcli info cluster eks + +# you should see something like this + +Auto mode is used by default +default add-ons can be disabled if a custom CNI is required +OIDC configuration can be achieved by setting the relevant oidc_* variables +ami_type: None +auto_mode: True +capacity_type: None +ctlplane_role: None +default_addons: True +disk_size: None +extended_support: True +extra_networks: [] +extra_subnets: None +flavor: None +logging: False +logging_types: ['api'] +network: default +oidc_client_id: None +oidc_group_claim: cognito:groups +oidc_issuer_url: None +oidc_name: oidc-config +oidc_username_claim: email +role: None +security_group: None +subnet: None +version: None +worker_role: None +workers: 2 +zonal_shift: False + +# get the instance-types or flavors - then you can use that in the create command if required + +kcli get instance-type + +# this kcli create example disables all addons, specifies subnets, enables coredns, ebs-csi and pod identity addons. It will deploy the default instance type in EKS as it isn't specified + +kcli create kube eks -P subnet=subnet-049d0829d2985b950 -P extra_subnets=[subnet-076237bed7ca7bd5f] -P default_addons=False -P apps=[coredns,aws-ebs-csi-driver,eks-pod-identity-agent] myeks + +# eventually you will see this - notice it has created the needed AWS roles for control plane and workers + +Disabling network add-ons (and automode) +Creating ctlplane role kcli-eks-ctlplane +Using ctlplane role kcli-eks-ctlplane +Creating worker role kcli-eks-worker +Using worker role kcli-eks-worker +Using subnet subnet-00455945f5834b4f6 as default +Using subnet subnet-031627ea5f56a0fda as extra subnet +Waiting for cluster myeks to be created +Creating nodegroup myeks +Kubernetes cluster myeks deployed!!! +INFO export KUBECONFIG=(kcli-boto3)$HOME/.kcli/clusters/myeks/auth/kubeconfig +INFO export PATH=(kcli-boto3)$PWD:(kcli-boto3)$PATH +Adding app coredns +Issue adding app coredns +Adding app aws-ebs-csi-driver +Issue adding app aws-ebs-csi-driver +Adding app eks-pod-identity-agent +``` + +- Export the kubeconfig and check all pods are in a pending state. + +```bash +export KUBECONFIG=(kcli-boto3)$HOME/.kcli/clusters/myeks/auth/kubeconfig +# after the create command has finished give things 2mins to complete - things are still rolling out +kubectl get pods -A +``` + +- Install cilium CNI - HELM or CLI + +If you have deployed using the byocni kcli create command in the previous sections then you will need to deploy a CNI - this example uses cilium. if you have deployed AWS CNI or are using another CNI then skip this step. + +```bash +# get the control plane API address +export API_SERVER=$(kubectl cluster-info | grep Kubernetes | awk -F/ '{print $3}') + +# this will automatically get the API address in one command and install cilium using helm + +helm upgrade --install --namespace kube-system --repo https://helm.cilium.io cilium cilium --set cluster.name=myeks --set k8sServiceHost=${API_SERVER} --set k8sServicePort=443 + +# ALTERNATIVE via cilium CLI - Install cilium and enables Ingress Controller +cilium install --set cluster.name=myeks --set k8sServiceHost=${API_SERVER} --set k8sServicePort=443 --set ingressController.enabled=true --set ingressController.loadbalancerMode=dedicated +``` + +- Check to see that pods are now running and deploy a test pod + +```bash +#all namespaces +kubectl get pods -A +# in default namespace +kubectl run nginx --image=nginx +kubectl get pods +``` + +## Pod Identity + +Clusters deployed by kcli won't have access to AWS services like S3 or EBS etc... + +Pod identity enables us to tie kubernetes Service Accounts to AWS roles with attached policies to allow access to AWS services. *It is the replacement process for IRSA* + +> Pod identity doesn't require OpenID provider + +*The process described below can be used to tie other kubernetes service accounts to AWS roles which have access to aws services* + +Using EBS as an example we can tie the EBS controller service account to a AWS role with an attached policy for managing EBS. + +- EBS storage +```bash +#Create podidentity-trust-relationship.json +cat <podidentity-trust-relationship.json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AllowEksAuthToAssumeRoleForPodIdentity", + "Effect": "Allow", + "Principal": { + "Service": "pods.eks.amazonaws.com" + }, + "Action": [ + "sts:AssumeRole", + "sts:TagSession" + ] + } + ] +} +EOF +# create a role with the trust policy for Pod identity attached +aws iam create-role --role-name ebs-role --assume-role-policy-document file://podidentity-trust-relationship.json --description "EBS role" + +# you can list the policies available +#aws iam list-policies | grep CSI + +# You could create a policy document +# example of one for EBS - https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/docs/example-iam-policy.json + +# and then create the policy in AWS with POLICY_NAME +# aws iam create-policy --policy-name POLICY_NAME --policy-document file://POLICY_DOC --query Policy.Arn --output text + +# and then attach the custom policy to the custom role +#aws iam attach-role-policy --role-name ebs-role --policy-arn=arn:aws:iam::XXXXXX:policy/POLICY_NAME + +# In this case we are using an existing AWS managed policy and attaching it to the role +aws iam attach-role-policy --role-name ebs-role --policy-arn=arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy +# list policies attached to the role +aws iam list-attached-role-policies --role-name ebs-role --output text +# list the pod identities in your cluster - should be empty (unless you have created other associations) +aws eks list-pod-identity-associations --cluster-name myeks + +# Get the account id +export ACCTID=$(aws sts get-caller-identity --query "Account" --output text) + +# associate the role with the kubernetes service account - we automatically get the account ID +aws eks create-pod-identity-association --cluster-name myeks --role-arn arn:aws:iam::${ACCTID}:role/ebs-role --namespace kube-system --service-account ebs-csi-controller-sa + +# restart the pods for ebs-csi so it picks up the new AWS credentials that are injected by pod identity +kubectl rollout restart -n kube-system deployment ebs-csi-controller +# check the AWS credentials are in the pod +kubectl describe pod -n kube-system ebs-csi-controller-7c77d8cc97-8hgg2 | grep AWS + +# you will also need to set the default storage class as it is no longer set by default. In this case only gp2 is setup, so make it the default. +kubectl annotate sc gp2 storageclass.kubernetes.io/is-default-class=true + +# optionally test creation of a EBS backed pv/pvc +cat < test-ebs-pvc.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ebs-storage +spec: + accessModes: + - ReadWriteOnce + storageClassName: gp2 + resources: + requests: + storage: 2Gi +EOF + +# create pvc + +kubectl apply -f test-ebs-pvc.yaml + +# create deployment using pvc which will trigger pv and EBS allocation +cat < test-ebs-pod.yaml +apiVersion: v1 +kind: Pod +metadata: + labels: + run: test-ebs + name: test-ebs +spec: + containers: + - image: nginx + name: test-ebs + volumeMounts: + - mountPath: /var/log/nginx + name: nginx-logs + volumes: + - name: nginx-logs + persistentVolumeClaim: + claimName: ebs-storage +EOF + +# create pod with pvc +kubectl apply -f test-ebs-pod.yaml +``` + +## Useful commands + +```bash +# changes depending on what type of cluster eks, generic etc...) +kcli list app +kcli list subnets +kcli list flavors +kcli info app coredns +kcli info app aws-ebs-csi-driver +kcli info cluster eks +kcli create bucket myeks +kcli create app aws-ebs-csi-driver +kcli delete cluster eks myeks +``` diff --git a/docs/index.md b/docs/index.md index db7fbd505..115718a71 100644 --- a/docs/index.md +++ b/docs/index.md @@ -307,6 +307,8 @@ To use this provider with kcli rpm, you'll need to install dnf -y install python3-boto3 ``` +see [AWS EKS workflow](https://github.com/karmab/kcli/blob/main/docs/EKS-kcli.md) for an example process + ## Azure ``` @@ -319,7 +321,7 @@ azure: location: westus ``` -The following parameters are specific to aws: +The following parameters are specific to azure: - `subscription_id` - `app_id` @@ -1645,6 +1647,9 @@ For all the platforms, the workflow is the following: - launch the specific subcommand. For instance, to deploy a generic Kubernetes cluster, one would use `kcli create cluster generic --pf my_parameters.yml $cluster`. Parameter files can be repeated and combined with specific parameters on the command line, which always take precedence. - Once the installation finishes, set the following environment variable in order to interact with the csluter `export KUBECONFIG=$HOME/.kcli/clusters/$cluster/auth/kubeconfig` +see [AWS EKS workflow](https://github.com/karmab/kcli/blob/main/docs/EKS-kcli.md) for an example EKS process + + ## Getting information on available parameters For each supported platform, you can use `kcli info cluster $clustertype` @@ -1922,6 +1927,8 @@ kcli create cluster gke mygke Note that on those platforms, we rely more on default values provided by the Platform +see [AWS EKS workflow](https://github.com/karmab/kcli/blob/main/docs/EKS-kcli.md) for an example EKS process. + ### Deploying applications on top of Kubernetes/OpenShift You can use kcli to deploy applications on your Kubernetes/OpenShift (regardless of whether it was deployed with kcli) diff --git a/docs/index.rst b/docs/index.rst index d1de60fbc..8d0c3780b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -326,6 +326,8 @@ To use this provider with kcli rpm, you’ll need to install dnf -y install python3-boto3 +see `AWS EKS workflow `__ for an example process + Azure ----- @@ -339,7 +341,7 @@ Azure secret: xxxxxxxxxxyyyyyyyy location: westus -The following parameters are specific to aws: +The following parameters are specific to azure: - ``subscription_id`` - ``app_id`` @@ -1771,6 +1773,8 @@ For all the platforms, the workflow is the following: - launch the specific subcommand. For instance, to deploy a generic Kubernetes cluster, one would use ``kcli create cluster generic --pf my_parameters.yml $cluster``. Parameter files can be repeated and combined with specific parameters on the command line, which always take precedence. - Once the installation finishes, set the following environment variable in order to interact with the csluter ``export KUBECONFIG=$HOME/.kcli/clusters/$cluster/auth/kubeconfig`` +see `AWS EKS workflow `__ for an example EKS process + Getting information on available parameters ------------------------------------------- @@ -2072,6 +2076,8 @@ For instance, to deploy a GKE cluster, you would use Note that on those platforms, we rely more on default values provided by the Platform +see `AWS EKS workflow `__ for an example EKS process. + Deploying applications on top of Kubernetes/OpenShift ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~