Merge pull request #534 from ODM2/develop for release v0.12.0

Update `main` from `develop` for release v0.12.0

doc/deployment_guide.md

@@ -1,5 +1,81 @@
 # Deployment Guide
 
+The tech stack supporting the Data Sharing Portal has been updated and we are now deploying the application to two separate servers (one for the web application and another for the database). 
+
+## Web Application Server
+
+While deploying the web application server, I recommend following the instructions outlined in this guide [Digital Ocean Deployment Guide for Ubuntu 20.04](https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-20-04). However, many of the steps in the Digital Ocean guide have been modified, so additional instructions can be found below.
+
+1. **Clone repositories:** Clone the web application and config repositories. I recommend cloning the repositories to the **'/opt'** directory using the command below.
+    - `cd /opt`
+    - `git clone https://github.com/ODM2/ODM2DataSharingPortal.git`
+    - `git checkout develop` use 'master' branch for production server
+    - `cd /opt`
+    - `git clone https://github.com/LimnoTech/ODM2DataSharingPortalConfig.git`
+    - `cd ./ODM2DataSharingPortalConfig`
+    - `git checkout develop`
+2. **Set up Python Environment:** For this project we will be using mini conda to create and manage our Python environments. I did not find a mini conda build in our package manager, so opted to get one directly from conda [Anaconda.com](https://anaconda.com). The instructions below include a link to the latest version of conda, which is what we used in our deployment. Future users of these instructions should double check available versions and consider if the latest release is correct for their application. It may make more sense to use an older LTS release for example. Also note these deployment instructions are for a server using an ARM CPU architecture. If deploying to a more traditional x86 CPU, look for the release of mini conda that is not on the 'aarch64' Linux platform.
+    - `cd ~`
+    - `wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh`
+    - `chmod +x` Miniconda-latest-Linux-aarch64.sh`
+    - `sudo ./Miniconda3-latest-Linux-aarch64.sh`
+        - when prompted where to install I selected '/opt/miniconda3' 
+    - I also needed to add out conda install to the system path
+        -`export PATH=$PATH:/opt/miniconda3/bin`
+    - `conda init bash` which initializes mini conda
+    - `exec bash` restart bash so that initialization takes effect 
+
+    Next we'll set up a virtual environment from the .yml file.
+    - `conda env create -f /opt/ODM2DataSharingPortal/environment_py38_dj22.yml` Note I modified this yml file so that the environment name was 'ODM2DataSharingPortal'
+    - `conda activate ODM2DataSharingPortal`
+3. **Create a symlink so the Web App will use the settings.json file in the config repo:** Note the instructions below use the development/staging settings. You may need to point to a different file for the production deployment.
+	- `sudo ln -s /opt/ODM2DataSharingPortalConfig/django/staging.settings.json /opt/ODM2DataSharingPortal/src/WebSDL/settings/settings.json`
+4. **Set Up Gunicorn:**
+    - Gunicorn is not in the default mini conda channel, so we'll need to get it from conda forge.
+        - `conda config --add channels conda-forge`
+        - `conda install -c conda-forge gunicorn`
+    - Test if guincorn starts
+        - `conda activate ODM2DataSharingPortal`
+        - `cd  /opt/ODM2DataSharingPortal/src`
+		- `gunicorn wsgi:application --bind 0.0.0.0:8000`
+    - After we know the testing works, copy over the gunicorn service file from the config repo to the server. I tried a symlink here but that did not work (probably because it is a service). I ended up just copying the file from the config repo to system services. Also note my copy command renames the file from envirodiy to gunicorn. The service we just created will automatically gunicorn with the appropriate arugments and create a socket to the application that will become the entry point for nginx. 
+        - `sudo cp /opt/ODM2DataSharingPortalConfig/GUnicorn/envirodiy.service /etc/systemd/system/gunicorn.service`
+    - As mentioned above, the service will automatically call gunicorn, but where gunicorn is on your system will depend on a variey of thing (i.e. how and where python was installed). In order to make the service file flexible and not dependent on the specifics of the installation I reference gunicorn located at '/usr/bin/gunicorn'. However this is very likely not were unicorn was installed. The solution is to create a symlink.
+        - optional `whereis gunicorn` to help find the install location
+        - `sudo ln /path/to/gunicorn /usr/bin/gunicorn`
+    - Finally I needed to modify the permissions of wsgi.py so that gunicorn to spin up the application.
+        - `cd /opt/ODM2DataSharingPortal/src`
+        - `chmod +755 wsgi.py`
+    - Now we just need to start the GUnicorn service we created
+        - `sudo systemctl start gunicorn`` 
+5. **Set up nginx**
+	- Install nginx 
+	    - `sudo apt install nginx`
+    - Create symlink between config repo and nginx
+        - `sudo ln -s /opt/ODM2DataSharingPortalConfig/nginx/staging_data_environdiy /etc/nginx/sites-enabled/ODM2DataSharingPortal`
+    - Test nginx
+        - `sudo nginx -t`
+    - If there were no errors during the test, start nginx which should make the site accessable online.
+        - `sudo systemctl start nginx`
+        
+6. **Set up SSL certificate**
+
+  - The following commands are taken verbatim from https://certbot.eff.org/lets-encrypt/ubuntufocal-nginx.html:
+
+    - Install current version of certbot
+      - `sudo snap install core; sudo snap refresh core`
+      - `sudo snap install --classic certbot`
+      - `sudo ln -s /snap/bin/certbot /usr/bin/certbot`
+    - Install certificate
+      - `sudo certbot --nginx` (enter domain name `staging.monitormywatershed.org` when prompted)
+    - Verify installation and auto-renewal
+      - `sudo certbot certificates`
+      - `sudo certbot renew --dry-run`
+
+---
+
+## Python 2.7 
+
 To deploy an instance of the Data Sharing Portal, follow the [Digital Ocean Ubuntu deployment guide](https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04).
 This deployment guide is an outline with specific Data Sharing Portal configuration steps.
 

environment.yml

@@ -0,0 +1,50 @@
+# Develop Environments for ODM2DataSharingPortal
+name: ODM2DataSharingPortal
+channels:
+    - conda-forge
+    - defaults
+
+dependencies:
+    # For ODM2DataSharingPortal migration to AWS
+    - python =3.8.10  # May 3, 2021: final regular Py 3.8 release. https://www.python.org/downloads/release/python-3810/
+    # - django =2.2.*  # Installs 2.2.14 but lastest is 2.2.24. Use pip to install.
+                       # https://docs.djangoproject.com/en/3.2/releases
+
+    # # Other Requirements
+    # - beautifulsoup4 =4.9.3  # with python 3.8
+    - coverage >=5.5
+    - google-api-python-client >=2.12.0
+    - hs_restclient >=1.3.7  # https://github.com/hydroshare/hs_restclient
+    - markdown >=3.3.4
+    # - oauthlib >=3.1.1  # with google api
+    - pandas >=1.3
+    - psycopg2 >=2.9.1
+    - python-crontab  >=2.5.1
+    # - requests  # with python 3.8
+    # - six  # with python 3.8
+
+
+    # Dev tools
+    - python-language-server
+
+    # package management
+    - conda
+    - conda-build
+    - pip
+
+    # Dependency versions not available on conda-forge
+    - pip:
+        # - codegen >=1.0  # necessary? Last updated in 2012. https://pypi.org/project/codegen/
+        - django ==2.2.24
+        - django-admin-select2 #  >=1.0.1
+        - django-debug-toolbar # >=1.11.1
+        - django-discover-runner # >=1.0
+        - django-reset-migrations # >=0.3.1
+        - django-webtest # >=1.8.0
+        - django-widget-tweaks # >=1.4.1
+        - djangorestframework
+        # - patterns >=0.3  # necessary? Last updated in 2014. https://pypi.org/project/patterns/
+        # - sqlparse  # with django=2.2
+        # - waitress  # with django extensions
+        # - WebOb  # with django extensions
+        # - WebTest  # with django extensions

src/WebSDL/settings/base.py

@@ -61,7 +61,8 @@
     'django.contrib.staticfiles',
     'widget_tweaks',
     'requests',
-    'reset_migrations'
+    'reset_migrations',
+    'timeseries_visualization'
 ]
 
 MIDDLEWARE = [
@@ -70,7 +71,6 @@
     'django.middleware.common.CommonMiddleware',
     'django.middleware.csrf.CsrfViewMiddleware',
     'django.contrib.auth.middleware.AuthenticationMiddleware',
-    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
     'django.contrib.messages.middleware.MessageMiddleware',
     'django.middleware.clickjacking.XFrameOptionsMiddleware',
     'hydroshare_util.middleware.AuthMiddleware',
@@ -104,7 +104,7 @@
     },
 ]
 
-WSGI_APPLICATION = 'WebSDL.wsgi.application'
+#WSGI_APPLICATION = 'WebSDL.wsgi.application'
 
 
 # Database
@@ -123,8 +123,6 @@
         'TEST': database['test'] if 'test' in database else {},
     }
 
-INFLUX_CONNECTION = data['influx_connection']
-
 # Password validation
 # https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators
 
@@ -148,13 +146,9 @@
 # https://docs.djangoproject.com/en/1.9/topics/i18n/
 
 LANGUAGE_CODE = 'en-us'
-
 USE_I18N = True
-
 USE_L10N = True
-
 LOGIN_URL = '/login/'
-
 DATABASE_ROUTERS = ['WebSDL.db_routers.WebSDLRouter']
 
 
@@ -165,32 +159,20 @@
 # SECURE_SSL_REDIRECT = True
 
 RECAPTCHA_KEY = data["recaptcha_secret_key"] if "recaptcha_secret_key" in data else ""
-
 RECAPTCHA_USER_KEY = data["recaptcha_user_key"] if "recaptcha_user_key" in data else ""
-
 RECAPTCHA_VERIFY_URL = "https://www.google.com/recaptcha/api/siteverify"
 
 EMAIL_SENDER = data['password_email_sender'] if 'password_email_sender' in data else '',
-
 NOTIFY_EMAIL = data['notify_email_sender'] if 'notify_email_sender' in data else ''
-
 DEFAULT_FROM_EMAIL = EMAIL_SENDER[0] if isinstance(EMAIL_SENDER, tuple) else EMAIL_SENDER
-
 NOTIFY_EMAIL_SENDER = NOTIFY_EMAIL[0] if isinstance(NOTIFY_EMAIL, tuple) else NOTIFY_EMAIL
-
 EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
-
 EMAIL_SERVER = data['email_host'] if 'email_host' in data else '',
-
 EMAIL_HOST = EMAIL_SERVER[0] if isinstance(EMAIL_SERVER, tuple) else EMAIL_SERVER
-
 EMAIL_HOST_USER = data['email_user'] if 'email_user' in data else ''
-
 EMAIL_HOST_PASSWORD = data['email_password'] if 'email_password' in data else ''
-
 EMAIL_USE_TLS = True
 
-
 DATETIME_FORMAT = "N j, Y g:i a"
 
 HYDROSHARE_UTIL_CONFIG = {
@@ -199,26 +181,16 @@
     'REDIRECT_URI': data['hydroshare_oauth']['redirect_uri']
 }
 
-INFLUX_URL_QUERY = data['influx_query']
-
-INFLUX_UPDATE_URL = data['influx_updater_query']['url']
-
-INFLUX_UPDATE_BODY = data['influx_updater_query']['body']
-
 # This data period is measured in days
 SENSOR_DATA_PERIOD = data['sensor_data_period'] if 'sensor_data_period' in data else '2'
 
-TSA_URL = data['tsa_url'] if 'tsa_url' in data else ''
-
 # crontab job settings
 CRONTAB_USER = data.get('crontab_user', getpass.getuser())
-
 CRONTAB_LOGFILE_PATH = data.get('crontab_log_file', '/var/log/odm2websdl-cron.log')
-
 CRONTAB_EXECUTE_DAILY_AT_HOUR = 5
 
 GOOGLE_API_CONF = data.get('google_api_conf', None)
 
 AUTH_USER_MODEL = 'accounts.User'
 
-DEBUG = True if 'debug_mode' in data and data['debug_mode'] == "True" else False
+DEBUG = True if 'debug_mode' in data and data['debug_mode'] == "True" else False

src/WebSDL/urls.py

@@ -17,12 +17,13 @@
 from django.conf.urls import url, include
 from django.contrib import admin
 from django.contrib.auth import views as auth_views
-from django.core.urlresolvers import reverse_lazy
+from django.urls import reverse_lazy
 
 from accounts.views import UserRegistrationView, UserUpdateView
 
 
-BASE_URL = settings.SITE_URL[1:]
+#BASE_URL = settings.SITE_URL[1:]
+BASE_URL = ''
 
 login_configuration = {
     'redirect_field_name': 'next'
@@ -41,19 +42,20 @@
 }
 
 urlpatterns = [
-    url(r'^' + BASE_URL + 'password-reset/$', auth_views.password_reset, password_reset_configuration, name='password_reset'),
-    url(r'^' + BASE_URL + 'password-reset/done/$', auth_views.password_reset_done, name='password_reset_done'),
-    url(r'^' + BASE_URL + 'password-reset/(?P<uidb64>[0-9A-Za-z]+)-(?P<token>.+)/$', auth_views.password_reset_confirm, password_done_configuration, name='password_reset_confirm'),
-    url(r'^' + BASE_URL + 'password-reset/completed/$', auth_views.password_reset_complete, name='password_reset_complete'),
+    url(r'^' + BASE_URL + 'password-reset/$', auth_views.PasswordChangeView, password_reset_configuration, name='password_reset'),
+    url(r'^' + BASE_URL + 'password-reset/done/$', auth_views.PasswordChangeView, name='password_reset_done'),
+    url(r'^' + BASE_URL + 'password-reset/(?P<uidb64>[0-9A-Za-z]+)-(?P<token>.+)/$', auth_views.PasswordResetConfirmView, password_done_configuration, name='password_reset_confirm'),
+    url(r'^' + BASE_URL + 'password-reset/completed/$', auth_views.PasswordResetCompleteView, name='password_reset_complete'),
     url(r'^' + BASE_URL + 'admin/', admin.site.urls),
-    url(r'^' + BASE_URL + 'login/$', auth_views.login, login_configuration, name='login'),
-    url(r'^' + BASE_URL + 'logout/$', auth_views.logout, logout_configuration, name='logout'),
+    url(r'^' + BASE_URL + 'login/$', auth_views.LoginView.as_view(), login_configuration, name='login'),
+    url(r'^' + BASE_URL + 'logout/$', auth_views.LogoutView.as_view(), logout_configuration, name='logout'),
     url(r'^' + BASE_URL + 'register/$', UserRegistrationView.as_view(), name='user_registration'),
     url(r'^' + BASE_URL + 'account/$', UserUpdateView.as_view(), name='user_account'),
     url(r'^' + BASE_URL + 'api-auth/', include('rest_framework.urls', namespace='rest_framework')),
     url(r'^' + BASE_URL + 'hydroshare/', include('hydroshare.urls', namespace='hydroshare')),
     url(BASE_URL, include('dataloaderinterface.urls')),
-    url(BASE_URL, include('dataloaderservices.urls'))
+    url(BASE_URL, include('dataloaderservices.urls')),
+    url(BASE_URL, include('timeseries_visualization.urls'))
 ]
 
 # if settings.DEBUG: