[WIP] Added the dashboard installer script with Keystone. Updated the Makefile to create a installer dist #676

anvithks · 2022-11-22T04:10:20Z

What type of PR is this?
/kind enhancement

What this PR does / why we need it:
This PR adds an independent installer script for the following:

SODA Dashboard
SODA Authentication using Keystone. (Required for Strato and Terra projects)
SRM Toolchain with Prometheus, Grafana and Alertmanager. (Required for Delfin visualization)

The SRM toolchain will be added in a subsequent commit or PR.
Which issue(s) this PR fixes:
Fixes #675

Test Report Added?:
/kind TESTED

/kind NOT-TESTED

Test Report:

Install

Uninstall

Uninstall & Purge

Special notes for your reviewer:

…ile to create a installer dist

joseph-v · 2022-11-22T04:25:38Z

install/readme.md

+- Export the desired SODA Release version to the environment variable
+
+```sh
+export SODA_RELEASE_VERSION=v1.7.0 # Replace with SODA release version or docker tag for dashboard.


Rename SODA_RELEASE_VERSION to DASHBOARD_RELEASE_VERSION

joseph-v · 2022-11-22T04:29:20Z

install/readme.md

+./install.sh
+
+Usage: install.sh <install|uninstall|uninstall_purge>
+./install install : Install Auth and Dashboard.


Please use consistent comment for Install and uninstall,
Use # to for comment

I will update the instructions.
Since they are not comments but instructions to be shown to the user if they run the script without providing the necessary parameter these should not be # comments.

I will add some comments in the script to let the users know that these are the usage instructions.

… instructions.

joseph-v · 2022-11-22T04:42:29Z

install/readme.md

+
+- This installer is currently tested and works on **Ubuntu 20.04**. 
+
+- Python  >= 3.8.10 ***(This installer may not work on Ubuntu 18.04 since the python3 version may not be supported.)***


Please check delfin PR #942 which works for the 18.04. (You can try to use ministone.py from that PR that contain some fixes)

@joseph-v PR #942 in the installer repository?

I used the ministone.py from that PR and tried on a Ubuntu 19=8.04 setup. I am getting the following exception:

Execption: HTTPConnectionPool(host='192.168.56.1', port=80): Max retries exceeded with url: /identity/v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd0243a5588>: Failed to establish a new connection: [Errno 111] Connection refused',))

But the same script worked in Ubuntu 20.04.

I checked online and there seem to be some dependencies that may be deprecated.

…he config files

joseph-v · 2022-11-25T03:40:41Z

install/install.sh

+
+    # Deleting the SRM Toolchain work folder
+    echo -e "\e[1;31m Deleting the SRM toolchain workfolder. \e[0m" 
+    rm -Rf /opt/srm-toolchain/


/opt/srm-toolchain/ needs to be configurable. All parameters in SODA installer ansible/group_vars/srm_toolschain.yml needs to be taken as input for the script

Made the work directory configurable. User needs to export this as an environment variable or the default value is picked.

install/install.sh

joseph-v · 2022-11-25T03:44:44Z

install/install.sh

+
+install_srm_toolchain() {
+    install $@
+    mkdir -p /opt/srm-toolchain


/opt/srm-toolchain needs to be configurable

It is configurable and is picked from the variable

joseph-v · 2022-11-25T03:47:49Z

install/readme.md

+export ALERTMANAGER_IMAGE_TAG=v0.21.0
+export SODA_ALERTMANAGER_PORT=9093
+export GRAFANA_IMAGE_TAG=7.3.5
+export SODA_GRAFANA_PORT=3000


All parameters in SODA installer ansible/group_vars/srm_toolschain.yml needs to be taken as input for the script

Yes. These are to be exported as environment variable. The env variable is then exposed to the docker container as well.

skdwriting · 2022-11-25T04:01:50Z

install/conf/srm-toolchain/alertmanager/config.yml

@@ -0,0 +1,6 @@
+route:
+ receiver: 'delfin'


Why dont we keep the project specific configuration files with separate folders? Same for related files below as well.

like ....dashboard/install/conf/delfin/.......
.......conf/strato/....

This is done.
currently the structure is changed to

conf/ - keystone/ ... - delfin ...

skdwriting · 2022-11-25T04:03:34Z

install/conf/srm-toolchain/prometheus/alerts.rules

+- name: Throughput
+  rules:
+
+  - alert: high_throughput


Some of these are very specific to Delfin right?! Why these confs should be added separately under the dashboard?! Can we not source it from the respective project?!

These configurations are for configuring Prometheus and alertmanager to be viewed using SODA Dashboard. User may not use this if they are not using SODA Dashboard. Hence the configurations are to be kept in the Dashboard or SRM toolchain conf.

I think it should be the installer configuration either from SODA installer or Delfin installer where the user decides to have dashboard or not.

We can consider this as an enhancement. Currently we are creating independent installer for Dashboard, which can be run from the Dashboard release.
Since most of these are environment variables the SODA Installer can just configure them at the time of installation. (This is how it currently works in the common installer as of v1.7.0)

skdwriting · 2022-11-25T04:05:09Z

install/install.sh

+    -e OPENSDS_AUTH_URL=http://$HOST_IP/identity \
+    -e OPENSDS_HOTPOT_URL=http://$HOST_IP:50040 \
+    -e OPENSDS_GELATO_URL=http://$HOST_IP:8089 \
+    -e SODA_DELFIN_URL=http://$HOST_IP:8190 \


Why these are hardcoded? The project selection is not available?! Or take such info from the soda conf information. Why this needs to hardcoded under dashboard? Should be part of SODA installer right?

These are being exposed as environment variable to the dashboard docker container. Project has never been available for the Dashboard.
There is no SODA conf information that is exported that gives these information on the env variables. These ports are internally mapped to the nginx conf within the SODA Dashboard docker container. This is the same command that was used as part of the SODA installer.

I think this is not correct. We need to discuss and clean it. Currently can you please add FIXME and we can merge if it is a blocker.

Sure we can do that. But this is how it works in the SODA installer currently.
We do not have any project / module specific configuration for Dashboard.

Again, this is now an independent installer that needs to be run on the system without the common installer. In this case we will have to export all the required env variables on the command line and expose the same to the docker containers.

…tructure

joseph-v · 2022-11-26T10:28:43Z

How do we handle the case where delfin wants prometheus (SRM Toolchain) but no dashboard. It may looks to odd to configure SRM toolschain from Dashboard in that case

anvithks · 2022-11-27T11:49:25Z

How do we handle the case where delfin wants prometheus (SRM Toolchain) but no dashboard. It may looks to odd to configure SRM toolschain from Dashboard in that case

Did we have this scenario in the previous release ?

AFAIK the SRM toolchain that we use was specifically for use with SODA Dashboard .

If you need to provide it separately then we can take it up as a new requirement .

In which case we need to figure out how delfin can configure and install the srm toolchain .

Added the dashboard installer script with Keystone. Updated the Makef…

75f97b5

…ile to create a installer dist

anvithks added the enhancement New feature or request label Nov 22, 2022

anvithks requested review from skdwriting and joseph-v November 22, 2022 04:10

anvithks self-assigned this Nov 22, 2022

joseph-v reviewed Nov 22, 2022

View reviewed changes

Updated as per review comments. changed the variable name and updated…

c3f4bee

… instructions.

joseph-v reviewed Nov 22, 2022

View reviewed changes

anvithks added 5 commits November 23, 2022 17:34

Added installer for the SRM toolchain. Updated the readme and added t…

349c5f4

…he config files

Updated the install, readme and Makefile

1a0771b

Updated the Makefile. Fixed incorrect version

1dfa4c8

Updated the readme. Added prerequisite of root

2a099ce

Updated the license info

0f36b23

joseph-v reviewed Nov 25, 2022

View reviewed changes

skdwriting reviewed Nov 25, 2022

View reviewed changes

anvithks added 2 commits November 26, 2022 11:18

Updated as per review comments. Made path configurable, changed dir s…

d07f1e6

…tructure

Updated as per review comments. Added FIXME

dfc7af0

anvithks changed the title ~~Added the dashboard installer script with Keystone. Updated the Makefile to create a installer dist~~ [WIP] Added the dashboard installer script with Keystone. Updated the Makefile to create a installer dist Nov 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Added the dashboard installer script with Keystone. Updated the Makefile to create a installer dist #676

[WIP] Added the dashboard installer script with Keystone. Updated the Makefile to create a installer dist #676

anvithks commented Nov 22, 2022 •

edited

Loading

joseph-v Nov 22, 2022

anvithks Nov 22, 2022

joseph-v Nov 22, 2022

anvithks Nov 22, 2022

anvithks Nov 22, 2022

joseph-v Nov 22, 2022

anvithks Nov 22, 2022

anvithks Nov 22, 2022

joseph-v Nov 25, 2022

anvithks Nov 26, 2022

joseph-v Nov 25, 2022

anvithks Nov 26, 2022

joseph-v Nov 25, 2022

anvithks Nov 26, 2022

skdwriting Nov 25, 2022

anvithks Nov 26, 2022

skdwriting Nov 25, 2022

anvithks Nov 26, 2022

skdwriting Nov 26, 2022

anvithks Nov 26, 2022

skdwriting Nov 25, 2022

anvithks Nov 26, 2022

skdwriting Nov 26, 2022

anvithks Nov 26, 2022

anvithks Nov 26, 2022

joseph-v commented Nov 26, 2022

anvithks commented Nov 27, 2022


		- This installer is currently tested and works on Ubuntu 20.04.

		- Python >= 3.8.10 *(This installer may not work on Ubuntu 18.04 since the python3 version may not be supported.)*

[WIP] Added the dashboard installer script with Keystone. Updated the Makefile to create a installer dist #676

Are you sure you want to change the base?

[WIP] Added the dashboard installer script with Keystone. Updated the Makefile to create a installer dist #676

Conversation

anvithks commented Nov 22, 2022 • edited Loading

Install

Uninstall

Uninstall & Purge

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joseph-v commented Nov 26, 2022

anvithks commented Nov 27, 2022

anvithks commented Nov 22, 2022 •

edited

Loading