Kubernetes in Cloud

General Usage

build secret pod for mounting

kubectl create secret generic secrets \
--from-file=$HOME/.cloudvolume/secrets/google-secret.json \
--from-file=$HOME/.cloudvolume/secrets/aws-secret.json \
--from-file=$HOME/.cloudvolume/secrets/boss-secret.json

delete secret kubectl delete secret --all

monitor and watch

get the pod id

kubectl get pods

watch the logs

watch kubectl logs pod-id

config dns servers

kubectl edit configmap kube-dns-autoscaler --namespace=kube-system

We can monitor the traffic operations per second in Kubernetes deployment monitor interface. There is no bandwidth usage in Kubernetes, we can only monitor it in instance group monitor interface.

deployment

• create: kubectl apply -f deploy.yml
• check: kubectl get deployments
• delete: kubectl delete deployment inference
• scale: kubectl scale --replicas=85 -f deploy.yml

Google Cloud

The complete documentation.

easy to use web console

resize cluster

gcloud container clusters resize my-cluster --region us-central1 --node-pool gpu-pool-1 --num-nodes 1

Reconnect

gcloud container clusters get-credentials my-cluster

Install NVIDIA drivers

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/stable/nvidia-driver-installer/cos/daemonset-preloaded.yaml

AWS

we use kops to create clusters in AWS. It is easy to use and manage. kops is the kubectl of cluster. AWS was officially supported and it can also work with other cloud vendors.

Setup the key and roles

follow the setup here.

Create cluster

create-cluster.sh

#!/bin/sh


# use k8x.local in cluster name to enable gossip DNS setup
export NAME=chunkflow.k8s.local
export KOPS_STATE_STORE=s3://chunkflow-cluster-state-store

kops create cluster $NAME \
    --zones us-east-1c \
    --master-count 1 \
    --master-zones us-east-1c \
    --master-size m3.medium  \
    --master-volume-size 10 \
    --node-count 1 \
    --node-size p2.xlarge \
    --node-volume-size 20 \
    --cloud-labels "Team=MyTeam,Owner=MyName" \
    --dry-run \
    -o yaml > $NAME.yaml

run this script and it will generate yml file for kops

./create-cluster.sh 

edite the yaml file and add maxPrice: "your_bidding_price" under the worker section. The maxPrice item will make kops allocate spot instances rather than default on-demand instances. This feature was not supported in command line interface yet. That's why we need the yaml file here.

kops create -f chunkflow.k8s.local.yaml 
kops create secret --name chunkflow.k8s.local sshpublickey admin -i ~/.ssh/id_rsa.pub
kops update cluster chunkflow.k8s.local --yes
kubectl get nodes

delete cluster

kops delete -f chunkflow.k8s.local.yaml --yes

GPU suport

in kops documentation.

Note that the setup only support CUDA9.1. K80 GPUs needs cuda9 and T4 needs cuda10. This is required for both Google Cloud and AWS!

Connect with kubectl

In default, kubectl will be set up after using kops. We'll lost the kubectl connection to AWS once we connect it to other cloud, such as Google Cloud. We can reconnect kubectl to AWS following the steps here.