GITBOOK-920: ED-2838 release-6.0.0 documentation CSP changes

SUMMARY.md

@@ -94,6 +94,13 @@
     * [Ansible Variables](use/installing-ed-platform/release-5.2.0/ansible-variables.md)
     * [Build](use/installing-ed-platform/release-5.2.0/build.md)
     * [Post Install Steps](use/installing-ed-platform/release-5.1.0/post-install-steps.md)
+  * [release-6.0.0 (Draft)](use/installing-ed-platform/release-5.2.0-1/README.md)
+    * [Prerequisites for your own Sunbird ED Instance](use/installing-ed-platform/release-5.2.0-1/prerequisites-for-your-own-sunbird-ed-instance.md)
+    * [Infra Provisioning](use/installing-ed-platform/release-5.2.0-1/infra-provisioning.md)
+    * [Jenkins Setup](use/installing-ed-platform/release-5.2.0-1/jenkins-setup.md)
+    * [Ansible Variables](use/installing-ed-platform/release-5.2.0-1/ansible-variables.md)
+    * [Build](use/installing-ed-platform/release-5.2.0-1/build.md)
+    * [Post Install Steps](use/installing-ed-platform/release-5.2.0-1/post-install-steps.md)
   * [Sunbird ED Portal](use/installing-ed-platform/sunbird-ed-portal.md)
   * [Sunbird ED Mobile](use/installing-ed-platform/sunbird-ed-mobile.md)
   * [Easy installer (Draft)](use/installing-ed-platform/easy-installer-draft.md)

use/installing-ed-platform/release-5.2.0-1/README.md

@@ -0,0 +1,2 @@
+# release-6.0.0 (Draft)
+

use/installing-ed-platform/release-5.2.0-1/ansible-variables.md

@@ -0,0 +1,58 @@
+# Ansible Variables
+
+### Overview <a href="#overview" id="overview"></a>
+
+Ansible is the configuration management system used in Sunbird. The infrastructure setup, deployment of services and other configurations is handled primarily through ansible.
+
+#### Updating the Private Repository with Hosts and Variables <a href="#updating-the-private-repository-with-hosts-and-variables" id="updating-the-private-repository-with-hosts-and-variables"></a>
+
+Execute the following commands in your local to clone the private repo template and update your private GitHub repository -
+
+```bash
+git clone https://github.com/project-sunbird/sunbird-devops
+cd sunbird-devops
+git checkout tags/release-6.0.0 -b release-6.0.0
+```
+
+* Copy the directory `sunbird-devops/private_repo/ansible` to your private repo local workspace
+* Run the `key-generate.sh` script located under `ansible/inventory/dev` folder. The script will ask for the ansible vault password. Enter the same vault password that you used in the Jenkins setup stage. The script will generate many private key files and they will be encrypted with the vault password. Deployments will fail if keys are not encrypted.
+* Update the files **common.yml**, **hosts**, and **secrets.yml** under **Core**, **KnowledgePlatform** and **DataPipeline** directories. After updating, push them to your private repo branch
+* Your private repo structure starting from the root path should be as shown below
+
+```bash
+  ansible
+└── inventory
+    └── dev
+        ├── Core
+        │   ├── common.yml
+        │   ├── hosts
+        │   └── secrets.yml
+        ├── Kubernetes
+        │   ├── common.yml   (soft link to Core directory files)
+        │   ├── hosts.       (soft link to Core directory files)
+        │   └── secrets.yml  (soft link to Core directory files)
+        ├── DataPipeline
+        │   ├── common.yml
+        │   ├── hosts
+        │   └── secrets.yml
+        └── KnowledgePlatform
+            ├── common.yml
+            ├── hosts
+            └── secrets.yml
+```
+
+**Neo4j download and upload to object storage**
+
+* Neo4j community artifact needs to be downloaded from neo4j official website [http://dist.neo4j.org/neo4j-community-3.3.9-unix.tar.gz](http://dist.neo4j.org/neo4j-community-3.3.9-unix.tar.gz) (only Neo4j 3.4 and below is supported)&#x20;
+* Upload downloaded artifact to `cloud_storage_artifacts_bucketname`&#x20;
+* Update `neo4j_zip` variable with artifact name `Eg: neo4j-community-3.3.9-unix.tar.gz`
+
+> Note:
+>
+> * The ansible inventory setup is a must before we can start to run jobs from the **Provision**, **ArtifactUpload** and **Deploy** directory in Jenkins. The **Build** directory on Jenkins does not depend on the ansible variables
+> * It is highly recommended that you complete the ansible inventory updates before proceeding further
+
+#### List of Servers with their Ansible Group Names <a href="#list-of-servers-with-their-ansible-group-names" id="list-of-servers-with-their-ansible-group-names"></a>
+
+<table><thead><tr><th width="178">Module</th><th width="139">Servers</th><th width="151">Service</th><th>Ansible Group Names</th></tr></thead><tbody><tr><td>Build and Deploy</td><td>Server-1</td><td>Jenkins Master</td><td></td></tr><tr><td>Databases</td><td>Server-2</td><td>Cassandra</td><td>cassandra-1, lp-cassandra, dp-cassandra, core-cassandra, report-cassandra, cassandra-node-1, cassandra</td></tr><tr><td></td><td></td><td>Postgres</td><td>postgresql-master-1, postgresql-master, postgres</td></tr><tr><td></td><td></td><td>Application Elasticsearch</td><td>es-1, composite-search-cluster, core-es-1, core-es, es-backup, es</td></tr><tr><td></td><td></td><td>Neo4j</td><td>learning-neo4j-node1, learning-neo4j-cluster</td></tr><tr><td></td><td></td><td>Mongo</td><td>mongo_master, mongo</td></tr><tr><td>Knowledge Platform</td><td>Server-3</td><td>Redis</td><td>redis1, redis, redis-ps, redisall, lp-redis, lp-redis-ps, dp-redis, lms-redis, redis-exporter-targets</td></tr><tr><td></td><td></td><td>Zookeeper</td><td>processing-cluster-zookeepers, ingestion-cluster-zookeeper, raw-zookeeper, zookeeper</td></tr><tr><td></td><td></td><td>Kafka</td><td>processing-cluster-kafka, ingestion-cluster-kafka, kafka-1, kafka</td></tr><tr><td>Data Pipeline</td><td>Server-4</td><td>Spark</td><td>spark</td></tr><tr><td></td><td></td><td>Kafka Indexer (Logstash)</td><td>kafka-indexer</td></tr><tr><td></td><td></td><td>InfluxDB</td><td>influxdb</td></tr><tr><td></td><td></td><td>Keycloak</td><td>keycloak-1, keycloak</td></tr><tr><td>Learning</td><td>Server-5</td><td>Learning</td><td>learning1, learning, learningall</td></tr><tr><td></td><td></td><td>Graylog</td><td>graylog-1, graylog</td></tr><tr><td>Druid</td><td>Server-6</td><td>Druid</td><td>druid-postgres, raw-coordinator, raw-overlord, raw-broker, raw-historical, raw-middlemanager, raw-graphite, rollup-coordinator, dp-druid-broker, broker, coordinator, druid-raw</td></tr><tr><td></td><td></td><td>Logs Elasticsearch</td><td>log-es-1, log-es-backup, log-es</td></tr><tr><td></td><td></td><td></td><td></td></tr></tbody></table>
+

use/installing-ed-platform/release-5.2.0-1/infra-provisioning.md

@@ -0,0 +1,70 @@
+# Infra Provisioning
+
+> **Note:** We have automated infra provision scripts available for azure, if you using any other cloud service provider other than azure provision infrastructure manually.
+
+**Provisioning infrastructure on other cloud service providers (except Azure)**
+
+* Object storage with CORS enabled
+* Virtual network with 2 subnets, one for VM's and other for kubernetes cluster
+* Managed Kubernetes cluster with 4 worker nodes each node with 4 Core, 16GB RAM configuration
+* Create VM's as mentioned in prerequisites
+* Enable communication between above created two subnets, so that kubernetes cluster and VM'S created can communicate with each other
+
+**Provisioning infrastructure on Azure**
+
+You can run the following steps to create azure infrastructure using ansible.
+
+Easiest way to use the script will be to use azure cloud shell, as the cloud shell comes with all prerequisites bundled.
+
+* login to portal.azure.com
+* click on the cloudshell -> select bash ( if you’re using it for the first time )
+
+If you want to run this on your local machine, Follow this [guide](https://docs.microsoft.com/en-us/azure/developer/ansible/install-on-linux-vm?tabs=azure-cli#install-ansible-on-the-virtual-machine).
+
+```
+  git clone https://github.com/project-sunbird/sunbird-devops -b release-6.0.0
+cd sunbird-devops/deploy
+# Update the necessary variables in playbook
+ansible-playbook -c local azure-provision.yaml
+# Resulting infrastructure infromation will be stored in sunbird-devops/deploy/azure-resources.txt file.
+```
+
+**Creating the AKS cluster**
+
+> **Note** Follow the steps given below to create the Kubernetes cluster in Azure. For other clouds, please visit the respective cloud provider website The AKS cluster and VM’s should be in same vnet. If they are in different vnet, you have to peer the vnets. To successfully peer, the IP address of the vnets should not overlap.
+
+* Create a service principal and assign contributor role to service principal
+* Get the secrets and client id of service principal
+* Click [here](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) for more details
+* Create the AKS cluster either via Azure portal or using `az aks` command line
+* Refer to Azure documentation for all the available options
+* Below is a sample command which you can use -
+
+```bash
+  az aks create --resource-group <resouse-group-name> --node-resource-group <k8s-resource-group-name> --name <cluster name>  --node-count 4 --admin-username deployer --kubernetes-version 1.19.9 --service-principal "<service principal id>" --node-vm-size Standard_D4s_v3 --client-secret "<client id>" --network-plugin azure --ssh-key-value @deployer.pub -l <region> --vm-set-type VirtualMachineScaleSets --vnet-subnet-id /subscriptions/<subscription id>/resourceGroups/<resouse-group-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>/subnets/<subnet name>
+```
+
+> Note: Ensure you have allocated at least 1024 IP’s for your Kubernetes subnet (CIDR notation as x.x.x.x/22)
+
+Get the kubeconfig file for your cluster with the below command -
+
+```bash
+  az aks get-credentials --resource-group <resource group name> --name <cluster name> --file  k8s.yaml
+```
+
+#### Configuring the Azure storage account <a href="#configuring-the-azure-storage-account" id="configuring-the-azure-storage-account"></a>
+
+* Update the CORS rule for the storage account as follows:
+
+```bash
+    Allowed Origins: *
+    Allowed Methods: GET,HEAD,OPTIONS, PUT
+    Allowed Headers: Access-Control-Allow-Origin,Access-Control-Allow-Method,Origin,x-ms-meta-qq,x-ms-blob-type,x-ms-blob-content-type,Content-Type
+    Exposed Headers: Access-Control-Allow-Origin,Access-Control-Allow-Methods
+    Max Age: 200
+
+```
+
+* Disable **Secure transfer required** in storage account configuration
+
+***

use/installing-ed-platform/release-5.2.0-1/jenkins-setup.md

@@ -0,0 +1,77 @@
+# Jenkins Setup
+
+Jenkins is used to build, deploy and setup the infrastructure for Sunbird. Almost everything in Sunbird is automated using Jenkins pipelines which integrates with ansible and other tools.
+
+*   SSH to the Jenkins server and enter the following commands -
+
+    ```bash
+       git clone https://github.com/project-sunbird/sunbird-devops.git
+       cd sunbird-devops && git checkout tags/release-6.0.0 -b release-6.0.0
+       cd deploy/jenkins
+       sudo bash jenkins-server-setup.sh
+    ```
+* Open Jenkins UI in a browser by visiting **JENKINS\_IP:8080**
+* Enter the initial password and follow the on-screen instructions. Choose **Install suggested plugin** and create an admin user
+* Go to **http://JENKINS\_IP:8080/pluginManager/available** -> Search for ‘Configuration as Code Plugin’ and install the plugin without restart.
+
+> Don’t run the following until above steps finished
+
+*   Run the below commands on Jenkins server to install external plugins
+
+    ```bash
+      sudo bash jenkins-plugins-setup.sh
+      
+    ```
+*   Setup environment specific Jenkins jobs. Update the environment list as per your requirement in ascending order. For example, if you want to have dev, staging and production environments, follow the below instructions
+
+    ```bash
+      cp envOrder.txt.sample envOrder.txt
+      vi envOrder.txt
+      dev=0
+      staging=1
+      production=2
+      sudo bash jenkins-jobs-setup.sh
+    ```
+*   Restart Jenkins
+
+    ```bash
+      sudo service jenkins restart
+    ```
+*   Configure Jenkins
+
+    ```
+    sudo su jenkins
+    curl -SsL -o ~/jenkins.yaml https://raw.githubusercontent.com/project-sunbird/sunbird-devops/release-5.1.0/deploy/jenkins/jenkins.yaml
+      
+    # Replace all placeholders ${VALUE} with below mentioned values
+    ${DOCKER_REGISTRY_URL} to the container registry URL which you would like to store the container images
+    ${GH_USERNAME} to the github username which has access to the private repository
+    ${GH_PASSWORD} to the github access token which has access to the private repository
+    ${GH_PRIVATE_REPO_URL} to the private github repository URL where the environment variables and servers information is updated
+    ${GH_PRIVATE_REPO_BRANCH} to the name of the private repo branch where the environment variables and servers information is updated
+    ${GH_PUBLIC_REPO_BRANCH} to "release-6.0.0"
+    ${KP_PUBLIC_REPO_BRANCH} to "release-5.7.0"
+    ${DP_PUBLIC_REPO_BRANCH} to "release-5.1.2"
+
+    vim ~/jenkins.yaml
+    ```
+*   Run the below commands on Jenkins server to establish connectivity between Jenkins to kubernetes cluster and servers -
+
+    ```bash
+      mkdir -p /var/lib/jenkins/secrets
+      cd /var/lib/jenkins/secrets
+      touch deployer_ssh_key vault-pass k8s.yaml
+      chmod 600 deployer_ssh_key vault-pass k8s.yaml
+    ```
+
+    * Copy the contents of your server’s private key into `/var/lib/jenkins/secrets/deployer_ssh_key`
+    * Copy the kubernetes config file contents into `/var/lib/jenkins/secrets/k8s.yaml`
+    * Update `/var/lib/jenkins/secrets/vault-pass` file with the ansible vault password which will be used to encrypt the ansible secrets file
+* Run `sudo visudo` on jenkins server and add the below line -
+
+```bash
+      jenkins ALL=(ALL) NOPASSWD:ALL
+```
+
+* Reboot the Jenkins VM (`sudo reboot`)
+

use/installing-ed-platform/release-5.2.0-1/post-install-steps.md

@@ -0,0 +1,10 @@
+# Post Install Steps
+
+**Post Installation Steps**
+
+| JENKINS JOB TO RUN                                              | GITHUB TAG                                                                               | GITHUB REPO                                                                                                                  | COMMENTS                                                                                                                                                                                                                                                                                                                                                                          |
+| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| OpsAdministration/Core/PostInstallScript                        | branch\_or\_tag: release-5.1.0\_RC3                                                      | [https://github.com/project-sunbird/sunbird-devops.git](https://github.com/project-sunbird/sunbird-devops.git)               | Creates the default forms, framework, users, channel, licenses etc. Please ensure you provide all the values that the job requires. You need to also ensure the script is successful by closely inspecting the output line by line on the Jenkins console log. You can also take a look at the script and API’s and create your own data if you don’t require the default values. |
+| Deploy/DataPipeline/AnalyticsPopulatePSQLConsumerChannelMapping | release-5.1.0, channel\_id: your sunbird organisation id, consumer\_id: kong consumer id | [https://github.com/project-sunbird/sunbird-data-pipeline.git](https://github.com/project-sunbird/sunbird-data-pipeline.git) | Adds kong consumer in postgres Analytics DB to whitelist some of the API’s. You can get the kong cosumer id by querying in postgres on kong db `select * from consumers where username = 'api-admin';`                                                                                                                                                                            |
+
+***

use/installing-ed-platform/release-5.2.0-1/prerequisites-for-your-own-sunbird-ed-instance.md

@@ -0,0 +1,54 @@
+# Prerequisites for your own Sunbird ED Instance
+
+#### Recommended Permissions and Experience <a href="#recommended-permissions-and-experience" id="recommended-permissions-and-experience"></a>
+
+To successfully complete Sunbird installation, you need to have:
+
+* System administrator permissions on and all servers
+* Hands-on experience in administering and debugging Linux systems
+* Hands-on experience using Docker and Kubernetes to run containerized services
+
+> **Note:** Sunbird is tested only on cloud hosted Ubuntu servers. We do not provide any support for installing Sunbird on other operating systems.
+
+#### Infra Requirements <a href="#infra-requirements" id="infra-requirements"></a>
+
+* Kubernetes Cluster with 4 worker nodes each of 4 Core 16 GB RAM
+* Total 6 VM's are required each of 4 Core 16 GB RAM
+* Private GitHub repository to store ansible inventory
+* Fully Qualified Domain Name (FQDN) with SSL
+* Object Storage
+* Docker hub account or Any docker registry
+* Public IP
+* Google OAuth Credentials
+
+> **Steps to create:** https://developers.google.com/workspace/guides/create-credentials#oauth-client-id
+
+* Google V3 ReCaptcha Credentials
+
+> **Steps to create:** Login to https://www.google.com/recaptcha/admin and create one for domain
+
+* Maxmind city database (free or paid)
+* A SMTP account (any email provider works, except Gmail for now)
+* All ports must be open for internal communication between the VM's/Servers provisioned for the installation
+
+> **Note:** If resources are isolated by subnets, then subnet-to-subnet communication needs to be enabled by security rules
+
+* Enable internet and outbound access from VMs.&#x20;
+
+> **Note:** This is required to install dependent packages from the ubuntu package manager and other open source repositories
+
+* A sms service provider API Token (optional)
+
+> **Note:** This is required to get OTP's to registered email address when user register or reset
+
+* YouTube API Token (optional)
+
+> **Note:** This is required to upload video content directly using youtube URL
+
+* Slack account and slack bot with API Token (optional)
+
+#### Provisioning Servers <a href="#provisioning-servers" id="provisioning-servers"></a>
+
+Before you start the installation process, ensure that you have the required infrastructure mentioned above is provisioned. Every component in Sunbird can scale horizontally / vertically by adding additional resources. Please find the mapping between server and the application components below.
+
+<table><thead><tr><th>APPLICATION COMPONENT</th><th>SERVER CONFIG</th><th width="192.33333333333331">COUNT</th></tr></thead><tbody><tr><td>Jenkins</td><td>4core 16GB RAM 250G HDD</td><td>1</td></tr><tr><td>KP</td><td>4core 16GB RAM 60G HDD</td><td>1</td></tr><tr><td>DP</td><td>4core 16GB RAM 60G HDD</td><td>1</td></tr><tr><td>DB</td><td>4core 16GB RAM 60G HDD</td><td>1</td></tr><tr><td>Learning</td><td>4core 16GB RAM 60G HDD</td><td>1</td></tr><tr><td>Druid</td><td>4core 16GB RAM 60G HDD</td><td>1</td></tr><tr><td>Basic Load Balancers</td><td>Layer 4</td><td>2 (Optional)</td></tr></tbody></table>