diff --git a/README.md b/README.md index bc85b41..6ef8730 100644 --- a/README.md +++ b/README.md @@ -143,3 +143,309 @@ spec: claimName: secondary-filestore-pvc ``` + + +## Pre-Requisites + +In a production evironment, we expect these pre-requisites are already created. If not, create it accordingly. + +I use google cloud shell for running the below commands. If you are not using the cloud shell make sure that you have installed necessary tools before running these. + +### Set the variables + +``` +project=my-project-123456 + +vpcname=gke-vpc + +subnet1=subnet1 + +subnet2=subnet2 + +storagebucket=$project-filestore-backup$RANDOM + +primaryfilestore=filestore-primary + +secondaryfilestore=filestore-secondary + +primaryfileshare=vol1 + +secondaryfileshare=vol1 + +gkecluster=gke-cluster1 + +serviceaccount=filestore-backup-storage-sa +``` + +### 1. Create VPC & Subnets + +**Create VPC** + +``` +gcloud compute networks create $vpcname --subnet-mode=custom --bgp-routing-mode=regional +``` + +**Create Firewall Rules** + + +``` +gcloud compute firewall-rules create allow-all-access-gke --network $vpcname --allow all +``` + +**Create Subnet1 with secondary ip range for the gke pods** + +``` +gcloud compute networks subnets create $subnet1 --network=$vpcname --range=10.128.0.0/19 --region=europe-north1 --secondary-range=pods-$subnet1=10.130.0.0/19 +``` + +**Create Subnet2 with secondary ip range for the gke pods** + +``` +gcloud compute networks subnets create $subnet2 --network=$vpcname --range=10.132.0.0/19 --region=europe-west4 --secondary-range=pods-$subnet2=10.134.0.0/19 +``` + + +### 2. Create Storage Bucket for filestore backup + +``` +gsutil mb -p $project -c STANDARD -l eu gs://$storagebucket +``` + +### 3. Create Primary and Secondary Filestore instances + +**Create Primary Filestore Instance** + +``` +gcloud filestore instances create $primaryfilestore --project=$project --zone=europe-north1-b --tier=STANDARD --file-share=name=$primaryfileshare,capacity=1TB --network=name=$vpcname +``` + +**Create Primary Filestore Instance** + + +``` +gcloud filestore instances create $secondaryfilestore --project=$project --zone=europe-west4-a --tier=STANDARD --file-share=name=$secondaryfileshare,capacity=1TB --network=name=$vpcname +``` + +### 4. Create GKE Cluster + + +``` +gcloud container clusters create $gkecluster \ + --region europe-north1 --node-locations europe-north1-a,europe-north1-b --enable-master-authorized-networks \ + --network $vpcname \ + --subnetwork $subnet1 \ + --cluster-secondary-range-name pods-$subnet1 \ + --services-ipv4-cidr 10.131.128.0/24 \ + --enable-private-nodes \ + --enable-ip-alias \ + --master-ipv4-cidr 10.131.248.0/28 \ + --num-nodes 1 \ + --default-max-pods-per-node 64 \ + --no-enable-basic-auth \ + --no-issue-client-certificate \ +--enable-master-authorized-networks \ +--master-authorized-networks=35.201.7.129/32 + +``` + +For master-authorized-networks, replace the value with your public ip address where you will use kubectl commands. + +If you are using google cloud shell, you can run the below command to get the public ip address of your cloud shell: + +``` +curl icanhazip.com +``` + +### 5. Create Container Registry for storing container images + +Refer the document to enable container registry and authenticate to it : https://cloud.google.com/container-registry/docs/quickstart + + +### 6. Create Service Account for filestore backup & set permissions + +**Create Service Account** + +``` +gcloud iam service-accounts create $serviceaccount --description="sa for filestore backup gcs storage" --display-name="filestore-backup-storage-sa" +``` + +**Set ACL for Servicve Account in filestore backup storage bucket** + +``` +gsutil iam ch serviceAccount:$serviceaccount@$project.iam.gserviceaccount.com:objectAdmin gs://$storagebucket/ +``` + +**Create JSON Key for the Service Account** + +``` +gcloud iam service-accounts keys create ~/filestore-backup-storage-sa-key.json --iam-account $serviceaccount@$project.iam.gserviceaccount.com +``` + + +**Convert the Json key to base64 encoded format for using in kubernetes secret** + +``` +base64 ~/filestore-backup-storage-sa-key.json | tr -d '\n' | tr -d '\r' > ~/filestore-backup-storage-sa-key-base64.txt +``` + + +This output should be used as the value for GCP_GCLOUD_AUTH in the filestore-backups-cronjob-sample.yaml + +Note: Make sure that there is no newline while you copy paste this value to the yaml. + + +## Using the Solution + +**Clone the repository to the cloud shell** + +``` +git clone https://github.com/sreesanpd/google-filestore-backup-kubernetes-cronjobs +``` + +**Change Directory to repository folder** + +``` +cd google-filestore-backup-kubernetes-cronjobs +``` + +**Change Directory to Dockerfil foldere** + +``` +cd docker-resources +``` + +**Docker build and push to container registry** + +``` +docker build . -t gcr.io/$project/gcp-filestore-k8s-backup + +docker push gcr.io/$project/gcp-filestore-k8s-backup + +``` + +Note: Make sure that you have followed the steps to enable container registry and authenticated to it as per the step 5 in pre-requisites + +**Change directory to kubernetes folder** + +``` +cd ../kubernetes-resources +``` + +**Modify the yaml with correct values as per your requirement** + +Refer to GCS - Example Kubernetes Cronjob in this document + +**Create the cronjob in the GKE cluster** + +``` +kubectl -f kubernetes-resources/filestore-backups-cronjob-sample.yaml +``` + +## Troubleshooting + +If there is a problem with the cronjob container, you can inspect it by: + +1. Add sleep timer to the container by editing google-filestore-backup-kubernetes-cronjobs/docker-resources/resources/filestore-backup.sh + +``` +#!/bin/bash + +## Create the GCloud Authentication file if set ## +if [ ! -z "$GCP_GCLOUD_AUTH" ] +then + echo "$GCP_GCLOUD_AUTH" > "$HOME"/gcloud.json + gcloud auth activate-service-account --key-file="$HOME"/gcloud.json +fi + +## backup filestore to GCS ## +DATE=$(date +"%m-%d-%Y-%T") +gsutil rsync -r /mnt/$FILESHARE_MOUNT_PRIMARY/ gs://$GCP_BUCKET_NAME/$DATE/ + + +## rsync primary filestore to secondary filestore ## +rsync -avz --delete /mnt/$FILESHARE_MOUNT_PRIMARY/ /mnt/$FILESHARE_MOUNT_SECONDARY/ + +sleep 1000 + +``` + +2. Build and push the container image to container repository + +3. Deploy the cronjob + +4. Login to the cronjob container shell by running the below command: + +``` +kubectl exec -it podname -- /bin/bash +``` + +Replace the podname with your cronjob's pod name. + +5. Run the bash script manually using: + + +``` +bash -x /filestore-backup.sh +``` + +6. You can also check whether the filestore has been properly mounted by running the command: + +``` +df -h +``` + +7. If any problem with deploying the cronjob, run: + +``` +kubectl describe cronjob filestore-backup +``` + +## Clean Up + +**Delete GKE Cluster** + +``` +gcloud container clusters delete $gkecluster --region europe-north1 --quiet +``` + +**Delete Filestore Instances** + +``` +gcloud filestore instances delete $primaryfilestore --zone europe-north1-b --quiet + +gcloud filestore instances delete $secondaryfilestore --zone europe-west4-a --quiet +``` + +**Delete Storage Bucket** + +``` +gsutil rm -r gs://$storagebucket + +gsutil rb -f gs://$storagebucket +``` + +**Delete Subnets** + +``` +gcloud compute networks subnets delete $subnet1 --region europe-north1 --quiet + +gcloud compute networks subnets delete $subnet2 --region europe-west4 --quiet +``` + +**Delete Firewall Rules** + +``` +gcloud compute firewall-rules delete allow-all-access-gke --quiet +``` + +**Delete VPC** + +``` +gcloud compute networks delete $vpcname --quiet +``` + +**Delete Service Account** + +``` +gcloud iam service-accounts delete $serviceaccount@$project.iam.gserviceaccount.com --quiet +```