From f4448988a2b5c51d0b77616409fff3371239a7ee Mon Sep 17 00:00:00 2001 From: prasadtalasila Date: Fri, 30 Jun 2023 08:16:45 +0200 Subject: [PATCH] Completes the documentation --- deploy/vagrant/single-machine/README.md | 3 +- deploy/vagrant/two-machine/README.md | 2 +- docs/FAQ.md | 5 + docs/admin/client/CLIENT.md | 52 +++++- docs/admin/host.md | 92 +++++++++++ docs/admin/servers/lib/LIB-MS.md | 71 +++++++-- docs/admin/trial.md | 13 ++ docs/admin/vagrant/base-box.md | 66 ++++++++ docs/admin/vagrant/single-machine.md | 30 ++++ docs/admin/vagrant/two-machines.md | 74 +++++++++ docs/thanks.md | 2 +- docs/user/servers/lib/LIB-MS.md | 156 ------------------ mkdocs.yml | 12 +- mkdocs_offline.yml | 12 +- packages/client/README.md | 53 +++++++ packages/lib/README.md | 201 ++++++++++++++++++++++++ servers/config/gateway/README.md | 4 - 17 files changed, 659 insertions(+), 189 deletions(-) create mode 100644 docs/admin/host.md create mode 100644 docs/admin/trial.md create mode 100644 docs/admin/vagrant/base-box.md create mode 100644 docs/admin/vagrant/single-machine.md create mode 100644 docs/admin/vagrant/two-machines.md create mode 100644 packages/client/README.md create mode 100644 packages/lib/README.md diff --git a/deploy/vagrant/single-machine/README.md b/deploy/vagrant/single-machine/README.md index 400623592..9481ccb11 100644 --- a/deploy/vagrant/single-machine/README.md +++ b/deploy/vagrant/single-machine/README.md @@ -6,12 +6,11 @@ A dummy **foo.com** URL has been used for illustration. Please change this to yo Please follow these steps to make this work in your local environment. -1. Create [**dtaas** Vagrant box](../make_boxes/dtaas/README.md). Copy _vagrant_ SSH private key here. This shall be useful for logging into the vagrant machine create for single-machine deployment. You would have created an SSH key pair - _vagrant_ and _vagrant.pub_. The _vagrant_ is the private SSH key and is needed for the next steps. +1. Create [**dtaas** Vagrant box](../make_boxes/dtaas/README.md). Copy _vagrant_ SSH private key here. This shall be useful for logging into the vagrant machine created for single-machine deployment. You would have created an SSH key pair - _vagrant_ and _vagrant.pub_. The _vagrant_ is the private SSH key and is needed for the next steps. 1. Update the **Vagrantfile**. Fields to update are: 1. Hostname (`node.vm.hostname = "foo.com"`) 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. 1. Other adjustments are optional. - 1. Copy _vagrant_ SSH private key into the current directory (`deploy/vagrant/single-machine`). 1. Execute the following commands from terminal ```bash diff --git a/deploy/vagrant/two-machine/README.md b/deploy/vagrant/two-machine/README.md index e7362f952..9ddbcbf39 100644 --- a/deploy/vagrant/two-machine/README.md +++ b/deploy/vagrant/two-machine/README.md @@ -26,7 +26,7 @@ The first step is to define the network identity of the two VMs. For that, you n Update the **boxes.json**. There are entries one for each server. The fields to update are: - 1. `name` - name of server1 (`"name" = "workspaces"`) + 1. `name` - name of server1 (`"name" = "dtaas"`) 1. `hostname` - hostname of server1 (`"name" = "foo.com"`) 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. 1. `name` - name of server2 (`"name" = "services"`) diff --git a/docs/FAQ.md b/docs/FAQ.md index a3ea18785..061371278 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -13,6 +13,7 @@ DTaaS is software platform on which you can create and run digital twins. Please see the [features](user/features.md) page to get a sense of the things you can do in DaaS. + ??? Question "Are there any Key Performance / Capability Indicators for DTaaS?" | Key Performance Indicator | Value | @@ -28,6 +29,10 @@ | Availability | It is a matter of human resources. If you have human resources to maintain DTaaS round the clock, upwards 95% is easily possible. | +??? Question "Do you provide licensed software like Matlab?" + + The licensed software are not available on the software platform. But users have private workspaces which are based on Linux-based xfce Desktop environment. Users can install software in their workspaces. The licensed software installed by one user is not available to another user. + ## Digital Twin Models diff --git a/docs/admin/client/CLIENT.md b/docs/admin/client/CLIENT.md index f9fc9d601..6f8aa8f95 100644 --- a/docs/admin/client/CLIENT.md +++ b/docs/admin/client/CLIENT.md @@ -2,9 +2,10 @@ To host DTaaS client website on your server, follow these steps: -- The `build` folder contains all the optimized static files that are ready for deployment. +- Download the **DTaaS-client.zip** from the [releases page](https://github.com/INTO-CPS-Association/DTaaS/releases). +- Inside the `DTaaS-client` directory, there is `site` directory. The `site` directory contains all the optimized static files that are ready for deployment. -- Locate the file `build/env.js` and replace the example values to match your infrastructure. The constructed links will be "`REACT_APP_URL`/`REACT_APP_URL_BASENAME`/`{username}`/`{Endpoint}`". See the definitions below: +- Locate the file `site/env.js` and replace the example values to match your infrastructure. See the definitions below: ```js window.env = { @@ -36,7 +37,6 @@ To host DTaaS client website on your server, follow these steps: }; ``` - - Copy the entire contents of the build folder to the root directory of your server where you want to deploy the app. You can use FTP, SFTP, or any other file transfer protocol to transfer the files. - Make sure your server is configured to serve static files. This can vary depending on the server technology you are using, but typically you will need to configure your server to serve files from a specific directory. @@ -44,4 +44,48 @@ To host DTaaS client website on your server, follow these steps: - Once the files are on your server, you should be able to access your app by visiting your server's IP address or domain name in a web browser. -:fontawesome-solid-circle-info: The website depends on Traefik and ML Workspace components to be available. Otherwise, you only get a skeleton non-functional website. \ No newline at end of file +:fontawesome-solid-circle-info: The website depends on **Traefik gateway** and **ML Workspace** components to be available. Otherwise, you only get a skeleton non-functional website. + +## Complementary Components + +The website requires background services for providing actual functionality. The minimum background service required is atleast one [ML Workspace](https://github.com/ml-tooling/ml-workspace) serving the following routes. + +```js +https:foo.com//lab +https:foo.com//terminals/main' +https:foo.com//tools/vnc/?password=vncpassword +https:foo.com//tools/vscode/ +``` + +The `username` is the user workspace created using ML Workspace docker container. Please follow the instructions in [README](https://github.com/ml-tooling/ml-workspace/blob/main/README.md). You can create as many user workspaces as you want. If you have two users - alice and bob - on your system, then the following the commands in will instantiate the required user workspaces. + +```bash +mkdir -p files/alice files/bob files/common + +printf "\n\n start the user workspaces" +docker run -d \ + -p 8090:8080 \ + --name "ml-workspace-alice" \ + -v "$(pwd)/files/alice:/workspace" \ + -v "$(pwd)/files/common:/workspace/common" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="alice" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 + + + +docker run -d \ + -p 8091:8080 \ + --name "ml-workspace-bob" \ + -v "$(pwd)/files/bob:/workspace" \ + -v "$(pwd)/files/common:/workspace/common" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="bob" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 +``` + +Given that multiple services are running at different routes, a reverse proxy is needed to map the background services to external routes. You can use Apache, NGINX, Traefik or any other software to work as reverse proxy. diff --git a/docs/admin/host.md b/docs/admin/host.md new file mode 100644 index 000000000..b083c029d --- /dev/null +++ b/docs/admin/host.md @@ -0,0 +1,92 @@ +# DTaaS on Linux Operating System + +These are installation instructions for running DTaaS application on a Ubuntu Server 20.04 Operating System. The setup requires a machine which can spare 16GB RAM, 8 vCPUs and 50GB Hard Disk space. + +A dummy **foo.com** URL has been used for illustration. Please change this to your unique website URL. It is assumed that you are going to serve the application in only HTTPS mode. + +Please follow these steps to make this work in your local environment. Download the **DTaaS.zip** from the [releases page](https://github.com/INTO-CPS-Association/DTaaS/releases). Unzip the same into a directory named **DTaaS**. The rest of the instructions assume that your working directory is **DTaaS**. + +!!! note + If you only want to test the application and are not setting up a production instance, you can follow the instructions of [trial installation](trial.md). + +## Configuration + +You need to configure the Traefik gateway, library microservice and react client website. + +The first step is to decide on the number of users and their usenames. The traefik gateway configuration has a template for two users. You can modify the usernames in the template to the usernames chosen by you. + +### The traefik gateway server + +You can run the Run the Traefik gateway server in both and HTTPS HTTPS mode to experience the DTaaS application. The installation guide assumes that you can run the application in HTTPS mode. + +The Traefik gateway configuration is at _deploy/config/gateway/fileConfig.yml_. Change `foo.com` to your local hostname and user1/user2 to the usernames chosen by you. + +!!! tip + + Do not use `http://` or `https://` in _deploy/config/gateway/fileConfig.yml_. + +#### Authentication + +The dummy username is `foo` and the password is `bar`. +Please change this before starting the gateway. + +```bash +rm deploy/config/gateway/auth +touch deploy/config/gateway/auth +htpasswd deploy/config/gateway/auth +password: +``` + +The user credentials added in _deploy/config/gateway/auth_ should match the usernames in _deploy/config/gateway/fileConfig.yml_. + +## Configure lib microservice + +The first step in this configuration is to prepare the a filesystem for users. An example file system in `files/` directory. You can rename the top-level user1/user2 to the usernames chosen by you. + +Update the _deploy/config/lib_ of the library microservice. The simplest possibility is to use `local` mode with the following example. The filepath is the absolute filepath to `files/` directory. + +```env +PORT='4001' +MODE='local' +LOCAL_PATH ='filepath' +LOG_LEVEL='debug' +APOLLO_PATH='/lib' +GRAPHQL_PLAYGROUND='true' +``` + +## Configure react website + +Change the React website configuration in _deploy/config/client/env.js_. + +```js +window.env = { + REACT_APP_ENVIRONMENT: 'prod', + REACT_APP_URL: 'https://foo.com/', + REACT_APP_URL_BASENAME: '', + REACT_APP_URL_DTLINK: '/lab', + REACT_APP_URL_LIBLINK: '', + REACT_APP_WORKBENCHLINK_TERMINAL: '/terminals/main', + REACT_APP_WORKBENCHLINK_VNCDESKTOP: '/tools/vnc/?password=vncpassword', + REACT_APP_WORKBENCHLINK_VSCODE: '/tools/vscode/', + REACT_APP_WORKBENCHLINK_JUPYTERLAB: '/lab', + REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK: '', +}; +``` + +## Update the installation script + +Open `deploy/install.sh` and update user1/user2 to usernames chosen by you. + +## Perform the Installation + +Go to the DTaaS directory and execute + +```sh +source deploy/install.sh +``` + +You can run this script multiple times until the installation is successful. + +## Access the application + +Now you should be able to access the DTaaS application at: _https://foo.com_ diff --git a/docs/admin/servers/lib/LIB-MS.md b/docs/admin/servers/lib/LIB-MS.md index 1e475e8d2..96fa44db7 100644 --- a/docs/admin/servers/lib/LIB-MS.md +++ b/docs/admin/servers/lib/LIB-MS.md @@ -1,8 +1,14 @@ # Host Library Microservice +The **lib microservice** is a simplified file manager providing graphQL API. It has three features: + +* provide a listing of directory contents. +* transfer a file to user. +* Source files can either come from local file system or from a gitlab instance. + The library microservice is designed to manage and serve files, functions, and models to users, allowing them to access and interact with various resources. -This document provides instructions for running the library microservice. +This document provides instructions for running a stand alone library microservice. ## Setup the File System @@ -11,31 +17,45 @@ The users expect the following file system structure for their reusable assets. ![File System Layout](file-system-layout.png) There is a skeleton file structure in [DTaaS codebase](https://github.com/INTO-CPS-Association/DTaaS/tree/feature/distributed-demo/files). You can copy and create file system for your users. + +## Gitlab setup (optional) + +For this microserivce to be functional, a certain directory or gitlab project structure is expected. The microservice expects that the gitlab consisting of one group, DTaaS, and within that group, all of the projects be located, **user1**, **user2**, ... , as well as a **commons** project. Each project corresponds to files of one user. +A sample file structure can be seen in [gitlab dtaas group](https://gitlab.com/dtaas). You can visit the gitlab documentation on [groups](https://docs.gitlab.com/ee/user/group/) for help on the management of gitlab groups. + +You can clone the git repositories from the `dtaas` group to get a sample file system structure for the lib microservice. + ## Setup Microservice To set up the lib microservice, follow these steps: -```sh -git clone https://github.com/INTO-CPS-Association/DTaaS.git -cd DTaaS/server/lib -yarn install # Install the required dependencies -``` +Download the **lib-microservice.zip** from the [releases page](https://github.com/INTO-CPS-Association/DTaaS/releases). -### Environment Variables +## Configuration setup -To set up the environment variables for the lib microservice, create a new file named _.env_ in the `servers/lib` folder. Then, add the following variables and their respective values. Below you can see and how, with included examples: +The microservices uses `.env` environment files to receive configuration. + +To set up the environment variables for the lib microservice, create a new file named _.env_ in the `lib-ms` directory. Then, add the following variables and their respective values. Below you can see and how, with included examples: ```ini PORT='4001' MODE='local' or 'gitlab' -LOCAL_PATH='/Users//DTaaS/files' -GITLAB_GROUP='dtaas' +LOCAL_PATH ='/Users//DTaaS/files' +GITLAB_GROUP ='dtaas' GITLAB_URL='https://gitlab.com/api/graphql' TOKEN='123-sample-token' +LOG_LEVEL='debug' +TEST_PATH='/Users//DTaaS/servers/lib/test/data/test_assets' APOLLO_PATH='/lib' or '' GRAPHQL_PLAYGROUND='false' or 'true' ``` +The `LOCAL_PATH` variable is the absolute filepath to the location of the local directory which will be served to users by the Library microservice. + +The `GITLAB_URL`, `GITLAB_GROUP` and `TOKEN` are only relevant for `gitlab` mode. The `TOKEN` should be set to your GitLab Group access API token. For more information on how to create and use your access token, [gitlab page](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html). + +Once you've generated a token, copy it and replace the value of `TOKEN` with your token for the gitlab group, can be found. + Replace the default values the appropriate values for your setup. **NOTE**: @@ -46,19 +66,36 @@ Replace the default values the appropriate values for your setup. ### Start Microservice ```bash -yarn install -yarn build -yarn start +yarn install # Install dependencies for the microservice +yarn build # build the application +yarn start # start the application +``` + +You can press `Ctl+C` to halt the application. If you wish to run the microservice in the background, use + +```bash +nohup yarn start & disown ``` The lib microservice is now running and ready to serve files, functions, and models. Users can access the library microservice at URL: `http://localhost:/lib`. +## Developer Commands + +```bash +yarn install # Install dependencies for the microservice +yarn syntax # analyzes source code for potential errors, style violations, and other issues, +yarn build # compile ES6 files into ES5 javascript files and copy all JS files into build/ directory +yarn test -a # run all tests +yarn test -e # run end-to-end tests +yarn test -i # run integration tests +yarn test -u # run unit tests +yarn start # start the application +yarn clean # deletes directories "build", "coverage", and "dist" +``` -### Modes of Operation +## Service Endpoint -The library microservice hides the mode of operation from the users. The files can come from two sources: +The URL endpoint for this microservice is located at: `localhost:PORT/lib` -1. A directory on the server machine hosting DTaaS (`mode=local`) -1. A gitlab group on a gitlab instance (`mode=gitlab`). The gitlab can either be a community [gitlab](https://gitlab.com) or your own private gitlab instance. \ No newline at end of file diff --git a/docs/admin/trial.md b/docs/admin/trial.md new file mode 100644 index 000000000..93fcb1ff6 --- /dev/null +++ b/docs/admin/trial.md @@ -0,0 +1,13 @@ + +# Trial Installation + +A single step install script is helpful in performing a trial run of the software. This script installs DTaaS software with default credentials and users on a Ubuntu server OS. You can use it to check a test installation of DTaaS. + +```bash +wget https://github.com/INTO-CPS-Association/DTaaS/releases/download/v0.2.0/single-script-install.sh +bash single-script-install.sh +``` + +!!! warning + + This test installation has default credentials and is thus highly insecure. diff --git a/docs/admin/vagrant/base-box.md b/docs/admin/vagrant/base-box.md new file mode 100644 index 000000000..68ccd319d --- /dev/null +++ b/docs/admin/vagrant/base-box.md @@ -0,0 +1,66 @@ +# Vagrant Box for DTaaS + +There are some good vagrant boxes on [vagrant website](https://app.vagrantup.com/boxes/search). But these boxes require too many installations that take a long time and network bandwidth. So it is efficient to create one local vagrant box for DTaaS application and reuse the same in all installations. + +## Installed Software + +This base DTaaS vagrant box, when it is successfully created, has the following software: + +* docker +* nodejs and yarn +* jupyter +* microk8s +* containers + * mltooling/ml-workspace:0.13.2 + * traefik2.5 + * influxdb2.4 + * grafana + * telegraf + * gitlab + +## Create the vagrant box + +Publish a base virtualbox package to be used by +vagrant to publish all other virtualbox packages + +```bash +#create a key pair +ssh-keygen -b 4096 -t rsa -f key -q -N "" +mv key vagrant +mv key.pub vagrant.pub + +vagrant up + +# let the provisioning be complete +# replace the vagrant ssh key-pair with personal one +vagrant ssh + +# install the oh-my-zsh +sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" +# install plugins: history, autosuggestions, +git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions + +# inside ~/.zshrc, modify the following line +plugins=(git zsh-autosuggestions history cp tmux) + +# remove the vagrant default public key - first line of +# /home/vagrant/.ssh/authorized_keys + +# exit vagrant guest machine and then +# copy own private key to vagrant private key location +cp vagrant .vagrant/machines/default/virtualbox/private_key + +# check +vagrant ssh #should work + +vagrant halt + +vagrant package --base dtaas \ +--info "info.json" --output dtaas.vagrant + +# Add box to the vagrant cache in ~/.vagrant.d/boxes directory +vagrant box add --name dtaas ./dtaas.vagrant + +# You can use this box in other vagrant boxes using +#config.vm.box = "dtaas" +``` diff --git a/docs/admin/vagrant/single-machine.md b/docs/admin/vagrant/single-machine.md new file mode 100644 index 000000000..b92b80a05 --- /dev/null +++ b/docs/admin/vagrant/single-machine.md @@ -0,0 +1,30 @@ +# DTaaS on Single Vagrant Machine + +These are installation instructions for running DTaaS application inside one vagrant Virtual Machine. The setup requires a machine which can spare 16GB RAM, 8 vCPUs and 50GB Hard Disk space to the vagrant box. + +A dummy **foo.com** URL has been used for illustration. Please change this to your unique website URL. + +Please follow these steps to make this work in your local environment. + +1. Create [**dtaas** Vagrant box](./base-box.md). Copy _vagrant_ SSH private key into _deploy/vagrant/single-machine_. This shall be useful for logging into the vagrant machine created for single-machine deployment. You would have created an SSH key pair - _vagrant_ and _vagrant.pub_. The _vagrant_ is the private SSH key and is needed for the next steps. +1. Update the **Vagrantfile**. Fields to update are: + 1. Hostname (`node.vm.hostname = "foo.com"`) + 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. + 1. Other adjustments are optional. +1. Execute the following commands from terminal + +```bash +vagrant up +vagrant ssh +``` + +Set a cronjob inside the vagrant virtual machine to remote the conflicting default route. + +```bash +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/release-v0.2/deploy/vagrant/route.sh +sudo bash route.sh +``` + +If you only want to test the application and are not setting up a production instance, you can follow the instructions of [single script install](../trial.md). + +If you are not in a hurry and would rather have a production instance, follow the instructions of [regular server installation](../host.md) setup to complete the installation. diff --git a/docs/admin/vagrant/two-machines.md b/docs/admin/vagrant/two-machines.md new file mode 100644 index 000000000..9ae9fd237 --- /dev/null +++ b/docs/admin/vagrant/two-machines.md @@ -0,0 +1,74 @@ +# DTaaS on Two Vagrant Machines + +These are installation instructions for running DTaaS application in two vagrant virtual machines (VMs). In this setup, all the user workspaces shall be run on server1 while all the platform services will be run on server2. + +The setup requires two server VMs with the following hardware configuration: + +**server1**: 16GB RAM, 8 x64 vCPUs and 50GB Hard Disk space + +**server2**: 6GB RAM, 3 x64 vCPUs and 50GB Hard Disk space + +Under the default configuration, two user workspaces are provisioned on server1. The default installation setup also installs InfluxDB, Grafana and RabbitMQ services on server2. If you would like to install more services, you can create shell scripts to install the same on server2. + +## Create Base Vagrant Box + +Create [**dtaas** Vagrant box](./base-box.md). You would have created an SSH key pair - _vagrant_ and _vagrant.pub_. The _vagrant_ is the private SSH key and is needed for the next steps. Copy _vagrant_ SSH private key into the current directory (`deploy/vagrant/single-machine`). This shall be useful for logging into the vagrant machines created for two-machine deployment. + +## Configure Server Settings + +**NOTE**: A dummy **foo.com** and **services.foo.com** URLs has been used for illustration. Please change these to your unique website URLs. + +The first step is to define the network identity of the two VMs. For that, you need _server name_, _hostname_ and _MAC address_. The hostname is the network URL at which the server can be accessed on the web. Please follow these steps to make this work in your local environment. + +Update the **boxes.json**. There are entries one for each server. The fields to update are: + + 1. `name` - name of server1 (`"name" = "dtaas"`) + 1. `hostname` - hostname of server1 (`"name" = "foo.com"`) + 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. + 1. `name` - name of server2 (`"name" = "services"`) + 1. `hostname` - hostname of server2 (`"name" = "services.foo.com"`) + 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. + 1. Other adjustments are optional. + +## Launch platform default services + +RabbitMQ, Grafana and InfluxDB services are provisioned on this server. +InfluxDB webUI will be available at: _services.foo.com_. The RabbitMQ service and its management interface shall be available at 5672 and 15672 TCP ports respectively. The Grafana service shall be available at TCP port 3000. + +The firewall and network access settings of corporate / cloud network need to be configured to allow external access to the services. Otherwise the users of DTaaS will not be able to utilize these services from their user workspaces. + +Execute the following commands from terminal to start the machine. + +```bash +vagrant up --provision services +vagrant ssh services +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/release-v0.2/deploy/vagrant/two-machine/services.sh +bash services.sh +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/release-v0.2/deploy/vagrant/route.sh +sudo bash route.sh +``` + +After the server is up and running, you can see the following services active within server2. + +| service | external url | +|:---|:---| +| Influx visualization service | services.foo.com | +| Grafana visualization service | services.foo.com:3000 | +| RabbitMQ communication service | services.foo.com:5672 | +| RabbitMQ management service | services.foo.com:15672 | +|| + +## Launch DTaaS application + +Execute the following commands from terminal + +```bash +vagrant up --provision dtaas +vagrant ssh dtaas +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/release-v0.2/deploy/vagrant/route.sh +sudo bash route.sh +``` + +If you only want to test the application and are not setting up a production instance, you can follow the instructions of [single script install](../trial.md). + +If you are not in a hurry and would rather have a production instance, follow the instructions of [regular server installation](../host.md) setup to complete the installation. diff --git a/docs/thanks.md b/docs/thanks.md index 30a715b7c..f3330f347 100644 --- a/docs/thanks.md +++ b/docs/thanks.md @@ -5,7 +5,7 @@ ## Users -Cláudio Ângelo Gonçalves Gomes, Hao Feng, Tanusree Roy +Cláudio Ângelo Gonçalves Gomes, Dmitri Tcherniak, Elif Ecem Bas, Giuseppe Abbiati, Hao Feng, Henrik Ejersbo, Tanusree Roy ## Documentation diff --git a/docs/user/servers/lib/LIB-MS.md b/docs/user/servers/lib/LIB-MS.md index c1f524901..c130224ef 100644 --- a/docs/user/servers/lib/LIB-MS.md +++ b/docs/user/servers/lib/LIB-MS.md @@ -113,103 +113,8 @@ send requests to: https://foo.com/lib } ``` -=== "HTTP Request :warning:" - - ``` http-request - POST /lib - Host: foo.com: - Content-Type:application/json - User-Agent:Mozilla - Accept:_/_ - - query { - listDirectory(path: "user2") { - repository { - tree { - blobs { - edges { - node { - name - - } - } - } - trees { - edges { - node { - name - } - } - } - } - } - } - } - ``` - -=== "HTTP Response :warning:" - - ``` http-response - 200 OK - access-control-allow-origin: \* - connection: keep-alive - content-length: 76 - content-type: application/json; charset=utf-8 - date: Mon, 15 May 2023 10:13:37 GMT - etag: ................ - keep-alive: timeout=5 - x-powered-by: Express - - { - "data": { - "listDirectory": { - "repository": { - "tree": { - "blobs": { - "edges": [] - }, - "trees": { - "edges": [ - { - "node": { - "name": "data" - } - }, - { - "node": { - "name": "digital twins" - } - }, - { - "node": { - "name": "functions" - } - }, - { - "node": { - "name": "models" - } - }, - { - "node": { - "name": "tools" - } - } - ] - } - } - } - } - } - } - - ``` - - ### Fetch a file from the available files - - === "GraphQL Request" ```graphql-request @@ -250,65 +155,4 @@ send requests to: https://foo.com/lib } ``` -=== "HTTP Request :warning:" - - ```http-request - POST /lib - Host: foo.com: - Content-Type:application/json - User-Agent:Mozilla - Accept:*/* - - { - "query": - query{ - readFile(path:"user2/data/sample.txt"){ - repository{ - blobs{ - nodes{ - name - rawBlob - rawTextBlob - } - } - } - } - } - } - ``` - -=== "HTTP Response :warning:" - - ```http-response - 200 OK - access-control-allow-origin: * - connection: keep-alive - content-length: 76 - content-type: application/json; charset=utf-8 - date: Mon, 15 May 2023 10:13:37 GMT - etag: ................ - keep-alive: timeout=5 - x-powered-by: Express - - { - "data": { - "readFile": { - "repository": { - "blobs": { - "nodes": [ - { - "name": "sample.txt", - "rawBlob": "welcome hello world", - "rawTextBlob": "hello world" - } - ] - } - } - } - } - } - - ``` - - The _path_ refers to the file path to look at: For example, _user1_ looks at files of **user1**; _user1/functions_ looks at contents of _functions/_ directory. diff --git a/mkdocs.yml b/mkdocs.yml index 702e1bd69..d059297ab 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -17,8 +17,16 @@ theme: nav: - Home: index.md - Admin: - - website: admin/client/CLIENT.md - - library microservice: admin/servers/lib/LIB-MS.md + - Host Install: + - Trial: admin/trial.md + - Production: admin/host.md + - Vagrant: + - Base Box: admin/vagrant/base-box.md + - Single Machine: admin/vagrant/single-machine.md + - Two Machines: admin/vagrant/two-machines.md + - Separate Packages: + - website: admin/client/CLIENT.md + - library microservice: admin/servers/lib/LIB-MS.md - User: - Features: user/features.md - Website: user/website/index.md diff --git a/mkdocs_offline.yml b/mkdocs_offline.yml index 7cf5db10f..a4613421f 100644 --- a/mkdocs_offline.yml +++ b/mkdocs_offline.yml @@ -20,8 +20,16 @@ plugins: nav: - Home: index.md - Admin: - - website: admin/client/CLIENT.md - - library microservice: admin/servers/lib/LIB-MS.md + - Host Install: + - Trial: admin/trial.md + - Production: admin/host.md + - Vagrant: + - Base Box: admin/vagrant/base-box.md + - Single Machine: admin/vagrant/single-machine.md + - Two Machines: admin/vagrant/two-machines.md + - Separate Packages: + - website: admin/client/CLIENT.md + - library microservice: admin/servers/lib/LIB-MS.md - User: - Features: user/features.md - Website: user/website/index.md diff --git a/packages/client/README.md b/packages/client/README.md new file mode 100644 index 000000000..93adb9c9f --- /dev/null +++ b/packages/client/README.md @@ -0,0 +1,53 @@ +# Host the DTaaS Client Website + +To host DTaaS client website on your server, follow these steps: + +- The `site` folder contains all the optimized static files that are ready for deployment. +- Locate the file `site/env.js` to replace the `https://foo.com` in `REACT_APP_URL: 'https://foo.com/'` with your webserver URL. +- Copy the entire contents of the `site` folder to the correct directory of your server where you want to deploy the app. You can use FTP, SFTP, or any other file transfer protocol to transfer the files. +- Make sure your server is configured to serve static files. This can vary depending on the server technology you are using, but typically you will need to configure your server to serve files from a specific directory. +- Once the files are on your server, you should be able to access your app by visiting your server's IP address or domain name in a web browser. + +## Complementary Components + +The website requires background services for providing actual functionality. The minimum background service required is atleast one [ML Workspace](https://github.com/ml-tooling/ml-workspace) serving the following routes. + +```js +https:foo.com//lab +https:foo.com//terminals/main' +https:foo.com//tools/vnc/?password=vncpassword +https:foo.com//tools/vscode/ +``` + +The `username` is the user workspace created using ML Workspace docker container. Please follow the instructions in [README](https://github.com/ml-tooling/ml-workspace/blob/main/README.md). You can create as many user workspaces as you want. If you have two users - alice and bob - on your system, then the following the commands in will instantiate the required user workspaces. + +```bash +mkdir -p files/alice files/bob files/common + +printf "\n\n start the user workspaces" +docker run -d \ + -p 8090:8080 \ + --name "ml-workspace-alice" \ + -v "$(pwd)/files/alice:/workspace" \ + -v "$(pwd)/files/common:/workspace/common" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="alice" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 + + + +docker run -d \ + -p 8091:8080 \ + --name "ml-workspace-bob" \ + -v "$(pwd)/files/bob:/workspace" \ + -v "$(pwd)/files/common:/workspace/common" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="bob" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 +``` + +Given that multiple services are running at different routes, a reverse proxy is needed to map the background services to external routes. You can use Apache, NGINX, Traefik or any other software to work as reverse proxy. diff --git a/packages/lib/README.md b/packages/lib/README.md new file mode 100644 index 000000000..95817f549 --- /dev/null +++ b/packages/lib/README.md @@ -0,0 +1,201 @@ +# Overview + +The **lib microservice** is a simplified file manager providing graphQL API. It has three features: + +* provide a listing of directory contents. +* transfer a file to user. +* Source files can either come from local file system or from a gitlab instance. + +## Gitlab setup + +For this microserivce to be functional, a certain directory or gitlab project structure is expected. The microservice expects that the gitlab consisting of one group, DTaaS, and within that group, all of the projects be located, **user1**, **user2**, ... , as well as a **commons** project. Each project corresponds to files of one user. +A sample file structure can be seen in [gitlab dtaas group](https://gitlab.com/dtaas). You can visit the gitlab documentation on [groups](https://docs.gitlab.com/ee/user/group/) for help on the management of gitlab groups. + +You can clone the git repositories from the `dtaas` group to get a sample file system structure for the lib microservice. + +## Configuration setup + +The microservices uses `.env` environment files to receive configuration. + +In order to create this environment, you need to create a `.env` file, wherein you create the following environment variables, +and insert with the correct-information relevant for your setup: + +``` +PORT='4001' +MODE='local' or 'gitlab' +LOCAL_PATH ='/Users//DTaaS/files' +GITLAB_GROUP ='dtaas' +GITLAB_URL='https://gitlab.com/api/graphql' +TOKEN='123-sample-token' +LOG_LEVEL='debug' +TEST_PATH='/Users//DTaaS/servers/lib/test/data/test_assets' +APOLLO_PATH='/lib' or '' +GRAPHQL_PLAYGROUND='false' or 'true' +``` + +The `TOKEN` should be set to your GitLab Group access API token. For more information on how to create and use your access token, [gitlab page](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html). + +Once you've generated a token, copy it and replace the value of `TOKEN` with your token for the gitlab group, can be found. + +## Developer Commands + +```bash +yarn install # Install dependencies for the microservice +yarn build # build the application +yarn start # start the application +``` + +You can press `Ctl+C` to halt the application. If you wish to run the microservice in the background, use + +```bash +nohup yarn start & disown +``` + +## Developer Commands + +```bash +yarn install # Install dependencies for the microservice +yarn syntax # analyzes source code for potential errors, style violations, and other issues, +yarn build # compile ES6 files into ES5 javascript files and copy all JS files into build/ directory +yarn test -a # run all tests +yarn test -e # run end-to-end tests +yarn test -i # run integration tests +yarn test -u # run unit tests +yarn start # start the application +yarn clean # deletes directories "build", "coverage", and "dist" +``` + +## Service Endpoint + +The URL endpoint for this microservice is located at: `localhost:PORT/lib` + +## GraphQL API Calls + +The lib microservice takes two distinct GraphQL queries. + +### Directory Listing + +This query receives directory path and provides list of files in that directory. A sample query and response are given here. + +``` graphql +query { + listDirectory(path: "user1") { + repository { + tree { + blobs { + edges { + node { + name + type + } + } + } + trees { + edges { + node { + name + type + } + } + } + } + } + } +} +``` + +``` graphql +{ + "data": { + "listDirectory": { + "repository": { + "tree": { + "blobs": { + "edges": [] + }, + "trees": { + "edges": [ + { + "node": { + "name": "common", + "type": "tree" + } + }, + { + "node": { + "name": "data", + "type": "tree" + } + }, + { + "node": { + "name": "digital twins", + "type": "tree" + } + }, + { + "node": { + "name": "functions", + "type": "tree" + } + }, + { + "node": { + "name": "models", + "type": "tree" + } + }, + { + "node": { + "name": "tools", + "type": "tree" + } + } + ] + } + } + } + } + } +} +``` + +### Fetch a file + +This query receives directory path and send the file contents to user in response. A sample query and response are given here. + +```graphql +query { + readFile(path: "user2/data/sample.txt") { + repository { + blobs { + nodes { + name + rawBlob + rawTextBlob + } + } + } + } +} +``` + +```graphql +{ + "data": { + "readFile": { + "repository": { + "blobs": { + "nodes": [ + { + "name": "sample.txt", + "rawBlob": "hello world", + "rawTextBlob": "hello world" + } + ] + } + } + } + } +} +``` diff --git a/servers/config/gateway/README.md b/servers/config/gateway/README.md index de5a0a0da..8d26efa1b 100644 --- a/servers/config/gateway/README.md +++ b/servers/config/gateway/README.md @@ -28,10 +28,6 @@ docker run -d \ traefik:v2.5 ``` -## Access - -# Run Lib MS at port 4001 - ## Authentication The dummy username is `foo` and the password is `bar`.