TACC Core Portal

The base Portal code for TACC WMA Workspace Portals

Related Repositories:

• Camino, a Docker container-based deployment scheme
• Core CMS, the base CMS code for TACC WMA CMS Websites
• Core Portal Deployments, private repository that facilitates deployments of Core Portal images via Camino and Jenkins

Local Development Setup

Prequisites for running the portal application

• Docker 20.10.7
• Docker Compose 1.29.2
• Python 3.7.12
• Nodejs 12.x (LTS)

The Core Portal can be run using Docker and Docker Compose. You will need both Docker and Docker Compose pre-installed on the system you wish to run the portal on.

If you are on a Mac or a Windows machine, the recommended method is to install Docker Desktop, which will install both Docker and Docker Compose as well as Docker Machine, which is required to run Docker on Mac/Windows hosts.

Code Configuration

After you clone the repository locally, there are several configuration steps required to prepare the project.

Create settings_secret.py, settings_local.py, and secrets.py

• Create server/portal/settings/settings_secret.py containing what is in secret field in the Core Portal Settings Secret entry secured on UT Stache

• Copy server/portal/settings/settings_local.example.py to server/portal/settings/settings_local.py

  • Note: Setup ngrok and update WH_BASE_URL in settings_local.py to enable webhook notifications locally.

• Copy server/conf/cms/secrets.sample.py to server/conf/cms/secrets.py

Build the image for the portal's django container:

make build

OR

docker-compose -f ./server/conf/docker/docker-compose.yml build

Start the development environment:

make start

OR

docker-compose -f ./server/conf/docker/docker-compose-dev.all.debug.yml up

Install client-side dependencies and bundle code with webpack:

cd client
npm ci
npm run build

• Note: During local development you can also use npm run dev to set a livereload watch on your local system that will update the portal code in real-time. Again, make sure that you are using NodeJS 12.x and not an earlier version. You will also need the port 8080 available locally.

Initialize the application in the core_portal_django container:

docker exec -it core_portal_django /bin/bash
python3 manage.py migrate
python3 manage.py collectstatic --noinput
python3 manage.py createsuperuser  # Unless you will only login with your TACC account
python3 manage.py projects_id --update-using-max-value-found # Update projects id counter
python3 manage.py import-apps # Add set of example apps used in Frontnera portal (optional)

Initialize the CMS in the core_portal_cms container:

docker exec -it core_portal_cms /bin/bash
python3 manage.py migrate
python3 manage.py collectstatic --noinput
python3 manage.py createsuperuser

Finally, create a home page in the CMS.

NOTE: TACC VPN or physical connection to the TACC network is required to log-in to CMS using LDAP, otherwise the password set with python3 manage.py createsuperuser is used

Setting up search index:

Requirements:

• At least one page in CMS (see above).
• At least 15% of free disk space.
• For Mac/Windows
  • At least 4GB of RAM allocated to Docker (see Docker Desktop > Preferences > Resources > Advanced).
• For Linux (Locally)
  • Run sudo sysctl -w vm.max_map_count=2146999999 (The minimum required by ES is 262144 but it doesn't seem to work).
  • Run sudo sysctl -w vm.overcommit_memory=1.
  • Run sudo sysctl -p (In order to persist in /etc/sysctl.conf).

First, rebuild the cms search index:

docker exec -it core_portal_cms /bin/bash
python3 manage.py rebuild_index

Then, use the django shell in the core_portal_django container—

docker exec -it core_portal_django /bin/bash
python3 manage.py shell

—to run the following code to set up the search index:

from portal.libs.elasticsearch.indexes import setup_files_index, setup_projects_index, setup_allocations_index
setup_files_index(force=True)
setup_projects_index(force=True)
setup_allocations_index(force=True)

Setting up notifications locally:

1. Run an ngrok session to route webhooks to core_portal_nginx:

ngrok http 443

2. Then, take the https url generated by ngrok and paste it into the WH_BASE_URL setting in settings_local.py

Setup local access to the portal:

1. Add a record to your local hosts file for 127.0.0.1 cep.dev

   • sudo vim /etc/hosts

   WARNING: This name must match the agave callback URL defined for the client in settings_secret.py for _AGAVE_TENANT_ID.

   NOTE: Do NOT have your VPN connected when you do this. Otherwise your hosts file will be overwritten and you will have to do this step again.

2. Direct your browser to https://cep.dev. This will display the django CMS default page. To login to the portal, point your browser to https://cep.dev/login.

   NOTE: When logging in, make sure that you are going through SSL (https://cep.dev/login). After succesful login, you can use the debug server at https://cep.dev.

   NOTE: Evergreen browsers will no longer allow self-signed certificates. Currently Chrome and Firefox deny access to the local portal for this reason. A cert solution needs to be established in alignment with current TACC policies to resolve this.

Installing local CA

For your browser to open your local development environment, you need to configure your computer to accept the development environment's self-signed certificates. Every file needed is in conf/nginx/certificates.

NOTE: This may require a computer restart to take effect.

OSX

1. Open mac's Keychain Access
2. With Default Keychains > login selected, choose File > Import Items... from the menu.
3. Navigate to ./server/conf/nginx/certificates
4. Select ca.pem
5. Search for CEP and double click on the certificate
6. In the Trust section, find the "When using this certificate" dropdown and select "Always Trust"
7. Close the window to save.

Linux

1. $ cd ./server/conf/nginx/certificates
2. $ sudo mkdir /usr/local/share/ca-certificates/extra
3. $ sudo cp ca.pem /usr/local/share/ca-certificates/extra/cepCA.pem
4. $ sudo update-ca-certificates

Firefox UI

1. Go to preferences
2. Search for Authorities
3. Click on "View Certificates" under "Certificates"
4. On the Certificate Manager go to the "Authorities" tab
5. Click on "Import..."
6. Browse to ./server/conf/nginx/certificates
7. Select ca.pem

Firefox CLI (not tested)

1. sudo apt-get install libnss3-tools (or proper package manager)
2. certutil -A -n "cepCA" -t "TCu,Cu,Tu" -i ca.pem -d ${DBDIR}
3. $DBDIR differs from browser to browser for more info: Chromium: https://chromium.googlesource.com/chromium/src/+/master/docs/linux_cert_management.md Firefox: https://support.mozilla.org/en-US/kb/profiles-where-firefox-stores-user-data?redirectlocale=en-US&redirectslug=Profiles#How_to_find_your_profile

Linting and Formatting Conventions

Client-side code is linted (JavaScript via eslint, CSS via stylelint), and is enforced on commits to the repo. To see a list of linting issues, in the console:

1. Navigate to client/ directory.
2. Run npm run lint, which is the same as linting both languages independently:
   • npm run lint:js
   • npm run lint:css

You may auto-fix your linting errors to conform with configured standards, for specific languages, via:

• npm run lint:js -- --fix
• npm run lint:css -- --fix

Server-side Python code is linted via Flake8, and is also enforced on commits to the repo. To see server side linting errors, run flake8 from the command line. To do so, run the following in the core_portal_django container:

flake8

Testing

Server-side python testing is run through pytest. Start docker container first by docker exec -it core_portal_django bash, Then run pytest -ra from the server folder to run backend tests and display a report at the bottom of the output.

Client-side javascript testing is run through Jest. Run npm run test* from the client folder to ensure tests are running correctly.

* To run tests without console logging, run npm run test -- --silent.

Test Coverage

Coverage is sent to codecov on commits to the repo (see Github Actions for branch to see branch coverage). Ideally we only merge positive code coverage changes to main.

Production Deployment

The Core Portal runs in a Docker container as part of a set of services managed with Docker Compose.

Portal images are built by Jenkins and published to the Docker Hub repo.

To update the portal in production or dev, the corresponding Core Portal Deployments env file should be updated with a tag matching an image previously built and published to the taccwma/core-portal repo.

Deployments are initiated via Jenkins and orchestrated, tracked, and directed by Camino on the target server.

Deployment Steps

1. Build and publish portal image with Jenkins
2. Update deployment settings, particularly the PORTAL_TAG environment variable in Core Portal Deployments with new tag name
3. Deploy new image with Jenkins

Contributing

Development Workflow

We use a modifed version of GitFlow as our development workflow. Our development site (accessible behind the TACC Network) is always up-to-date with main, while the production site is built to a hashed commit tag.

• Feature branches contain major updates, bug fixes, and hot fixes with respective branch prefixes:
  • task/ for features and updates
  • bug/ for bugfixes
  • fix/ for hotfixes

Best Practices

Sign your commits (see this link for help)

Resources

• Learn Markdown
• Tapis Project (Formerly Agave)