AKS support in kuberentes

[johrstrom]

This ticket is to support AKS credential initialization in kuberentes.

┆Issue is synchronized with this Asana task by Unito

[farassadek]

Hi Jeff

in "gems/ood_core-0.18.1/lib/ood_core/job/adapters/kubernetes/batch.rb" we have

• gcloud auth activate-service-account --key-file=#{cred_file}
• gcloud container clusters get-credentials #{locale} #{cluster}

gcloud auth activate-service-account is equivalent to aws configure
aws configure to configure the the credentials on the server, it creates ~/.aws folder with two files, one for the credentials and another for other configuration like the region.

To get the credentials (given awscli is installed and ~/.aws folder exist:
aws configure get aws_access_key_id to get the access key given .aws exists
aws configure get aws_secret_access_key
aws configure get region

for gcloud container clusters get-credentials #{locale} #{cluster}

Get the cluster name (assume one cluster available only).
cluster_name=$(aws eks list-clusters | awk '{print $2}')

Get endpoint
endpoint=$(aws eks describe-cluster --name $cluster_name --query "cluster.endpoint")

Get the certificate
certificate_data=$(aws eks describe-cluster --name $cluster_name --query "cluster.certificateAuthority.data")

mkdir -p ~/.kube
cat > ~/.kube/config << EOF

apiVersion: v1
clusters:
- cluster:
    server: $endpoint
    certificate-authority-data: $certificate_data
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws
      args:
        - "eks"
        - "get-token"
        - "--cluster-name"
        - $cluster_name

EOF

cluster=$(kubectl get svc | grep -v NAME | awk '{print $1}')
aws eks get-token --cluster-name $cluster

[farassadek]

Hi Jeff,
Here is the code to setup a user to use EKS. This to be added to the pre-hook to setup user credentials.

#!/bin/bash
# 
#
# setupUserAccessCredentials.sh can be added to the openondemand hooks to:
#	Create an IAM user in AWS
#	Create access and secret access keys for the IAM user
#	Download and setup the credentials in .aws folder in the user home directory
#	Add the user to the kube-system configmap/aws-auth
#
# Note:
#	AWS IAM user credentials will be used for authentication only. No policy is 
#	attached to the IAM users that authorize that user to trigger api call to AWS.
#
# Requirements:
#	Credentials with permission to:
#		- create IAM users
#		- configure EKS cluster. 
# 	awscli and kubectl in /bin folder
# How to:
#	setupUserAccessCredentials.sh <USERNAME>
#
# Example:
#	createAwsEksIamUser.sh fadel
#		- Create IAM user "fadel" in AWS
#		- Create access and secret access keys for IAM user "fadel"
#		- Setup the access/secret keys in /path-to-fadel-home/.aws
#		- Add fadel IAM user to configmap/aws-auth
#			- userarn: arn:aws:iam::${AWS_ACCOUNT}:user/fadel
#			  username: fadel

USERNAME=$1

# Get user home path
USERHOME="$(getent passwd $USERNAME | awk -F ':' '{print $6}')"

# Exit if user home does not exist.
if [ -z "$USERHOME" ]; then
	echo ""
	echo "User home not found on the system"
	echo ""
	exit 255;
fi

# Exit if the IAM user already exists on AWS.
CHECK_USER=$(aws iam get-user --user-name ${USERNAME} 2>/dev/null)
if [ "$?" -eq "0" ]; then
	echo ""
	echo "User \"${USERNAME}\" Exists"
	echo "Commands for user manipulation:"
	echo "List \"${USERNAME}\" credentials:    aws iam list-access-keys --user-name ${USERNAME}"
	echo "Delete \"${USERNAME}\" credentials:  aws iam delete-access-key --access-key-id  ACCESS-KEY-ID  --user-name  ${USERNAME}"
	echo "Delete \"${USERNAME}\":              aws iam delete-user  --user-name  ${USERNAME}"
	exit 255;
fi

# Create AWS IAM user, exit if creation failed.
IAMUSER=$(aws iam create-user --user-name ${USERNAME})
if [ "$?" -ne "0" ]; then
	echo "Error creating user ${USERNAME}"
	exit 255;
fi

# Get the IAM ARN of the recently created user.
USERARN=$(aws iam get-user --user-name  ${USERNAME} --query "User.Arn" | xargs)

# Create access key and secret access keys for the IAM user.
CREDENTIALS=$(aws iam create-access-key --user-name ${USERNAME} --output text)
ACCESS_KEY=$(echo $CREDENTIALS | awk '{print $2}')
SECRET_KEY=$(echo $CREDENTIALS | awk '{print $4}')


# Setup user aws credentials into a temp folder
tmpDir=/tmp/${USERNAME}_$(date +"%F_%T")
mkdir -p $tmpDir

mkdir -p $tmpDir/.aws
echo [default] > $tmpDir/.aws/config
echo region = us-east-1 >> $tmpDir/.aws/config

echo [default] > $tmpDir/.aws/credentials
echo aws_access_key_id = $ACCESS_KEY >> $tmpDir/.aws/credentials
echo aws_secret_access_key = $SECRET_KEY >> $tmpDir/.aws/credentials
chmod 600 $tmpDir/.aws/*

# Check if the folder .aws exists on the user home.
DOTAWS=$(su - ${USERNAME} -c "stat ${USERHOME}/.aws" 2>/dev/null 1>/dev/null; echo $?)

if [ "$DOTAWS" -eq 0 ]; then
	# If .aws existed, backup the folder to "DOT_AWS_archive_{DATE}"
	su - ${USERNAME} -c "mv ${USERHOME}/.aws  ${USERHOME}/DOT_AWS_archive_$(date +"%F_%T")"
fi

# Copy the created .aws in temp into the user home folder.
chown -R ${USERNAME} ${tmpDir}
su - ${USERNAME} -c "cp -r $tmpDir/.aws  ${USERHOME}/"


# Update kube system aws-auth configmap
K8SUSER="    - userarn: ${USERARN}\n      username: ${USERNAME}\n"
kubectl get -n kube-system configmap/aws-auth -o yaml | awk "/mapUsers: \|/{print;print \"$K8SUSER\";next}1" > /tmp/aws-auth-patch.yml
kubectl patch configmap/aws-auth -n kube-system --patch "$(cat /tmp/aws-auth-patch.yml)"
# IMPORTANT , If [] is in aws-auth then it should be removed (also dangerous) 
kubectl get -n kube-system configmap/aws-auth -o yaml | grep -v '\[\]' > /tmp/aws-auth-patch.yml
kubectl patch configmap/aws-auth -n kube-system --patch "$(cat /tmp/aws-auth-patch.yml)"

# Remove the temp folder(s) and file(s)
rm -fr $tmpDir
rm -f  /tmp/aws-auth-patch.yml

[johrstrom]

I'm not sure if I like bootstrapping users in ondemand itself. I'd prefer if the credentials were out of scope for us - that is, that they exist already in .aws/credentials. In an on prem OIDC cluster, we copy OIDC tokens, but we don't create OIDC users.

We're given the privilege to bootstrap a user to a kubernetes cluster - yes, but extending that privilege to create IAM users in AWS? I'll have to think on. In any case, if this works for you, then that's great!

I'm not familiar with the pattern of using an auth confimap - is that common for AWS?

[farassadek]

The aws credentials for authentication only, users cannot trigger api call to aws using those credentials other than getting a token. that mean we can have a token out of those user credentials and populate the token to the user but since the code run in user space to. get and populate the token then a smart user can get the credentials. It is somehow complicated in AWS and my work based on AWS documentation. I guess it would be better if we can meet so I can explain my approach.
As for the auth confimap then yes, it is from AWS documentation, the user will not be seen by the EKS cluster unless there is an entry for the user (or for a role) in configmap/aws-auth .

Here is the docs of aws eks:

https://docs.aws.amazon.com/eks/latest/userguide/cluster-auth.html

• Enabling IAM user and role access to your cluster: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
• Authenticating users for your cluster from an OpenID Connect identity provider: https://docs.aws.amazon.com/eks/latest/userguide/authenticate-oidc-identity-provider.html
• Create a kubeconfig for Amazon EKS: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
• Installing aws-iam-authenticator: https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html
• Default Amazon EKS Kubernetes roles and users: https://docs.aws.amazon.com/eks/latest/userguide/default-roles-users.html