Skip to content

Latest commit

 

History

History
135 lines (90 loc) · 5.47 KB

cockroach-setup.md

File metadata and controls

135 lines (90 loc) · 5.47 KB
Raw

Deploy CockroachDB to Kubernetes Clusters

  • Download and Configure the Scripts to deploy cockroachdb
  1. Create a directory and download the required script and configuration files into it:
mkdir multiregion
cd multiregion
curl -OOOOOOOOO \
https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/{README.md,client-secure.yaml,cluster-init-secure.yaml,cockroachdb-statefulset-secure.yaml,dns-lb.yaml,example-app-secure.yaml,external-name-svc.yaml,setup.py,teardown.py}
  1. Retrieve the kubectl "contexts" for your clusters:
kubectl config get-contexts

At the top of the setup.py script, fill in the contexts map with the zones of your clusters and their "context" names, e.g.:

contexts = { 'eastus': 'crdb-k3s-eastus', 'westus': 'crdb-k3s-westus', 'northeurpoe': 'crdb-k3s-northeurope',}

  1. In the setup.py script, fill in the regions map with the zones and corresponding regions of your clusters, for example:

regions = { 'eastus': 'eastus', 'westus': 'westus', 'northeurope': 'northeurope',}

Setting regions is optional, but recommended, because it improves CockroachDB's ability to diversify data placement if you use more than one zone in the same region. If you aren't specifying regions, just leave the map empty.

  1. If you haven't already, install CockroachDB locally and add it to your PATH. The cockroach binary will be used to generate certificates.

If the cockroach binary is not on your PATH, in the setup.py script, set the cockroach_path variable to the path to the binary.

  1. Optionally, to optimize your deployment for better performance, review CockroachDB Performance on Kubernetes and make the desired modifications to the cockroachdb-statefulset-secure.yaml file.
  2. Run the setup.py script:
python setup.py

As the script creates various resources and creates and initializes the CockroachDB cluster, you'll see a lot of output, eventually ending with job "cluster-init-secure" created.

  1. Configure Core DNS

Each Kubernetes cluster has a CoreDNS service that responds to DNS requests for pods in its region. CoreDNS can also forward DNS requests to pods in other regions.

To enable traffic forwarding to CockroachDB pods in all 3 regions, you need to modify the ConfigMap for the CoreDNS Corefile in each region.

  1. Create a copy of the existing ConfigMap from each region. We will update these to forward DNS quires to the correct regions.
kubectl config use-context $clus1
kubectl -n kube-system get configmap coredns -o yaml > eastus.yaml
kubectl config use-context $clus2
kubectl -n kube-system get configmap coredns -o yaml > westus.yaml
kubectl config use-context $clus3
kubectl -n kube-system get configmap coredns -o yaml > northeurope.yaml
  1. After [obtaining the IP addresses of the ingress load balancers in all 3 regions aks clusters, you can use this information to define a separate ConfigMap for each region. Each unique ConfigMap lists the forwarding addresses for the pods in the 2 other regions

For each region, modify configmap.yaml by replacing:

  • region2 and region3 with the namespaces in which the CockroachDB pods will run in the other 2 regions.
  • ip1, ip2, and ip3 with the IP addresses of the Network Load Balancers in the region, which you looked up in the previous step.

You will end up with 3 different ConfigMaps. Give each ConfigMap a unique filename like configmap-1.yaml. An example of which can be found in this repository

  1. Then apply the new ConfigMap:
kubectl replace -f eastus.yaml --context $clus1 --force
kubectl replace -f westus.yaml --context $clus2 --force
kubectl replace -f northeurope.yaml --context $clus3 --force
  1. For each region, check that your CoreDNS settings were applied:
kubectl get -n kube-system cm/coredns --export -o yaml --context <cluster-context>
  1. Confirm that the CockroachDB pods in each cluster say 1/1 in the READY column - This could take a couple of minutes to propagate, indicating that they've successfully joined the cluster:
kubectl get pods --selector app=cockroachdb --all-namespaces --context $clus1

NAMESPACE NAME READY STATUS RESTARTS AGE eastus cockroachdb-0 1/1 Running 0 14m eastus cockroachdb-1 1/1 Running 0 14m eastus cockroachdb-2 1/1 Running 0 14m

kubectl get pods --selector app=cockroachdb --all-namespaces --context $clus2

NAMESPACE NAME READY STATUS RESTARTS AGE westus cockroachdb-0 1/1 Running 0 14m westus cockroachdb-1 1/1 Running 0 14m westus cockroachdb-2 1/1 Running 0 14m

  1. Create secure clients
kubectl config use-context $clus1
kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/multiregion/client-secure.yaml --namespace $loc1
kubectl exec -it cockroachdb-client-secure -n $loc1 -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public
CREATE USER cockroach WITH PASSWORD 'cockroach';
GRANT admin TO cockroach;
  1. Port forward the admin ui
kubectl port-forward cockroachdb-0 8080 --context $clus1 --namespace $loc1

You will then be able to access the Admin UI via your browser. http://localhost:8080

Back