SpotDiff

English description:

• The SpotDiff project invites citizens to identify illegal industrial factories on the locations that are supposed to be used as farmland by comparing satellite images before and after May 20th, 2016.

繁體中文介紹：

• SpotDiff專案希望能讓鄉民比對 2016.5.20 前後衛星雲圖，去抓出農委會2017年和2020年六千筆落差資料中的疑似工廠位址上的建物是不是新增建物，可以集中火力去檢舉或是可以把台灣疑似工廠的地點掃一遍。

Table of Content

• Coding standards
• Install PostgreSQL (administrator only)
• Setup back-end (administrator only)
• Setup development environment
• Manipulate database
• Test cases
• Deploy back-end using uwsgi (administrator only)
• API calls

Coding standards

When contributing code for this repository, please follow the guidelines below:

Language

• The primary language for this repository is set to English. Please use English when writing comments and docstrings in the code. The code should not contain languages that are other than English.
• The secondary language for this repository is set to Mandarin Chinese. It is OK to use Mandarin Chinese in git issues, pull requests, wiki pages, commit messages, and README files. However, English is encouraged.

Git workflow

• Follow the Git Feature Branch Workflow. The master branch preserves the development history with no broken code. When working on a system feature, create a separate feature branch.
• Always create a pull request before merging the feature branch into the main branch. Doing so helps keep track of the project history and manage git issues.
• NEVER perform git rebasing on public branches, which means that you should not run "git rebase [FEATURE-BRANCH]" while you are on a public branch (e.g., the main branch). Doing so will badly confuse other developers since rebasing rewrites the git history, and other people's works may be based on the public branch. Check this tutorial for details.
• NEVER push credentials to the repository, for example, database passwords or private keys for signing digital signatures (e.g., the user tokens).
• Request a code review when you are not sure if the feature branch can be safely merged into the main branch.

Python package installation

• Make sure you are in the correct conda environment before installing packages. Otherwise, the packages will be installed to the server's general python environment, which can be problematic.
• Make sure the packages are in the install_packages.sh script with version numbers, which makes it easy for others to install packages.
• Use the pip command first. Only use the conda command to install packages when the pip command does not work.

Coding style

• Use the functional programming style (check this Python document for the concept). It means that each function is self-contained and does NOT depend on a state that may change outside the function (e.g., global variables). Avoid using the object-oriented programming style unless necessary. In this way, we can accelerate the development progress while maintaining code reusability.
• Minimize the usage of global variables, unless necessary, such as system configuration variables. For each function, avoid modifying its input parameters. In this way, each function can be independent, which is good for debugging code and assigning coding tasks to a specific collaborator.
• Use a consistent coding style.
  • For Python, follow the PEP 8 style guide, for example, putting two blank lines between functions, using the lower_snake_case naming convention for variable and function names. Please use double quote (not single quote) for strings.
  • For JavaScript, follow the Idiomatic JavaScript style guide, for example, using lowerCamelCase naming convention for variable and function names. Please use double quote (not single quote) for strings.
• Document functions and script files using docstrings.
  • For Python, follow the numpydoc style guide. Here is an example. More detailed numpydoc style can be found on LSST's docstrings guide.
  • For JavaScript, follow the JSDoc style guide
• For naming files, never use white spaces.
  • For Python script files (and shell script files), use the lower_snake_case naming convention. Avoid using uppercase.
  • For JavaScript files, use the lower_snake_case naming convention. Avoid using uppercases.
• Always comment the code, which helps others read the code and reduce our pain in the future when debugging or adding new features.
• Write testing cases to make sure that functions work as expected.

Install PostgreSQL (administrator only)

WARNING: this section is only for system administrators, not developers.

Install and start postgresql database (we will use version 13). This assumes that Ubuntu 18.04 LTS or Ubuntu 20.04 LTS is installed.

# For Ubuntu

# Get the signing key and import it
wget https://www.postgresql.org/media/keys/ACCC4CF8.asc
sudo apt-key add ACCC4CF8.asc

# Add the repository
echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list

# Fetch the metadata from the new repo
sudo apt-get update

# Install PostgreSQL 13
sudo apt-get install -y postgresql-13

# Start the service
sudo systemctl start postgresql

# Check postgresql status
sudo systemctl status postgresql

# Check postgresql log
sudo tail -100 /var/log/postgresql/postgresql-13-main.log

For Mac OS, I recommend installing postgresql by using Homebrew.

# For Mac OS

# Install PostgreSQL 13
brew install postgresql@13

# Start the service
brew services start postgresql

Enter the postgres shell.

# For Ubuntu
sudo -u postgres psql postgres

# For Mac OS
psql postgres

In the psql shell, create a project user, create a database for the user with a password, and check if the user and database exist. Replace the [DATABASE_PASSWORD] with the project's database password. IMPORTANT: do not forget the semicolon and the end of the commands.

# Set the password encryption method
SET password_encryption = 'scram-sha-256';
# Give the project user with a password
CREATE USER spotdiff PASSWORD '[DATABASE_PASSWORD]';

# Create databases for the project user
# For the staging server
CREATE DATABASE spotdiff_staging OWNER spotdiff;
# For the production server
CREATE DATABASE spotdiff_production OWNER spotdiff;
# For the test cases
CREATE DATABASE spotdiff_testing OWNER spotdiff;

# Check the list of user roles
SELECT rolname FROM pg_authid;

# Check the list of encrypted user passwords
SELECT rolpassword FROM pg_authid;

# Check if the user role exists
\du

# Check if the database exists
\l

# Exist the shell
\q

Edit the "pg_hba.conf" file to set the authentication methods to the ones that require encrypted passwords. This step is used to increase the security of the database on the Ubuntu server. You can skip this step if you are using Mac OS for development.

# For Ubuntu
sudo vim /etc/postgresql/13/main/pg_hba.conf
# Scroll to the end and relace all "peer" with "scram-sha-256", except those for the local connections
# Below are examples
local   all             postgres                                peer
local   all             all                                     peer
host    all             all             127.0.0.1/32            scram-sha-256
host    all             all             ::1/128                 scram-sha-256
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            scram-sha-256
host    replication     all             ::1/128                 scram-sha-256

# For Mac OS
vim /usr/local/var/postgres/pg_hba.conf
# Scroll to the end and relace all "trust" with "scram-sha-256", except those for the local connections
# Below are examples
local   all             all                                     trust
host    all             all             127.0.0.1/32            scram-sha-256
host    all             all             ::1/128                 scram-sha-256
local   replication     all                                     trust
host    replication     all             127.0.0.1/32            scram-sha-256
host    replication     all             ::1/128                 scram-sha-256

If you want to delete a user or a database, enter the postgres shell and use the following:

# Delete the staging server database
DROP DATABASE spotdiff_staging;

# Delete the project user
DROP USER spotdiff;

Setup back-end (administrator only)

WARNING: this section is only for system administrators, not developers.

Install conda for all users. This assumes that Ubuntu is installed. A detailed documentation is here. First visit here to obtain the downloading path. The following script install conda for all users:

# For Ubuntu
cd ~
wget https://repo.anaconda.com/miniconda/Miniconda3-py38_4.9.2-Linux-x86_64.sh
sudo sh Miniconda3-py38_4.9.2-Linux-x86_64.sh -b -p /opt/miniconda3
echo '' | sudo tee -a /etc/bash.bashrc
echo '# For miniconda3' | sudo tee -a /etc/bash.bashrc
echo 'export PATH="/opt/miniconda3/bin:$PATH"' | sudo tee -a /etc/bash.bashrc
echo '. /opt/miniconda3/etc/profile.d/conda.sh' | sudo tee -a /etc/bash.bashrc
source /etc/bash.bashrc

For Mac OS, I recommend installing conda by using Homebrew.

# For Mac OS
brew install --cask miniconda
echo 'export PATH="/usr/local/Caskroom/miniconda/base/bin:$PATH"' >> ~/.bash_profile
echo '. /usr/local/Caskroom/miniconda/base/etc/profile.d/conda.sh' >> ~/.bash_profile
source ~/.bash_profile

Clone this repository

git clone [PROJECT_REPOSITORY_URL] spotdiff

Set the permission of the folder (for Ubuntu server setup only, not Mac OS).

# Add a development group for the project
sudo addgroup spotdiff-dev

# Add yourself and collaborators to the group
sudo usermod -a -G spotdiff-dev $USER
sudo usermod -a -G spotdiff-dev [USER_NAME]

# Check the groups of a user
groups [USER_NAME]

# Check the group list
cat /etc/group

# Set permissions
sudo chown -R root spotdiff/
sudo chmod -R 775 spotdiff/
sudo chgrp -R spotdiff-dev spotdiff/

Create three text files to store the database urls in the "back-end/secret/" directory for the staging, production, and testing environments. For the url format, refer to the flask-sqlalchemy documentation. Replace [DATABASE_PASSWORD] with the database password. IMPORTANT: never push the database urls to the repository.

cd spotdiff/back-end/
mkdir secret
cd secret/
echo "postgresql://spotdiff:[DATABASE_PASSWORD]@localhost/spotdiff_staging" > db_url_staging
echo "postgresql://spotdiff:[DATABASE_PASSWORD]@localhost/spotdiff_production" > db_url_production
echo "postgresql://spotdiff:[DATABASE_PASSWORD]@localhost/spotdiff_testing" > db_url_testing

Create a private key for the server to encode the JSON Web Tokens for user login:

cd spotdiff/back-end/www/
python gen_key.py ../secret/private_key confirm

Setup development environment

Create conda environment and install packages. It is important to install pip first inside the newly created conda environment.

conda create -n spotdiff
conda activate spotdiff
conda install python=3.8
conda install pip
which pip # make sure this is the pip inside the conda environment
sh spotdiff/back-end/install_packages.sh

If the environment already exists and you want to remove it before installing packages, use the following:

conda deactivate
conda env remove -n spotdiff

Run the following to upgrade the database to the latest migration.

cd spotdiff/back-end/www/

# Upgrade the database to the latest migration
sh db.sh upgrade

If this is the first time that you set up the database, run the following to initialize the database migration. IMPORTANT: do NOT perform this step if the database migration folder exists on the repository.

# Generate the migration directory
# IMPORTANT: do not perform this step if the database migration folder exists 
sh db.sh init

# Generate the migration script
# IMPORTANT: do not perform this step if the database migration folder exists 
sh db.sh migrate "initial migration"

Run server in the conda environment for development purpose.

sh development.sh

You can test the application using http://localhost:5000/ or the following curl command.

curl localhost:5000

Manipulate database

We use flask-migrate to manage database migrations. The script "db.sh" enhances the workflow by adding the FLASK_APP environment. If you edit the database model and want to perform database migration, run the following:

cd spotdiff/back-end/www/

# Generate the migration script
sh db.sh migrate "[YOUR_MIGRATION_COMMIT_MESSAGE]"

Then, a new migration script will be generated under the "back-end/www/migrations/versions" folder. Make sure that you open the file and check if the code make sense. After that, run the following to upgrade the database to the latest migration:

# Upgrade the database to the latest migration
sh db.sh upgrade

If you want to downgrade the database to a previous state, run the following.

# Downgrade the database to the previous migration
sh db.sh downgrade

Test cases

For the back-end, the test cases are stored in the "back-end/www/tests" folder and written using Flask-Testing. Remember to write test cases for the model operations in the "back-end/www/models/model_operations" folder. Below shows how to run test cases:

cd spotdiff/back-end/www/tests
# Run all tests
python run_all_tests.py
# Run one test
python user_tests.py

Deploy back-end using uwsgi (administrator only)

WARNING: this section is only for system administrators, not developers.

Install uwsgi using conda.

conda activate spotdiff
conda install -c conda-forge uwsgi=2.0.20

Create a folder for server logging.

mkdir spotdiff/back-end/log/

Run the uwsgi staging server and check if it works.

cd spotdiff/back-end/www/
sh deploy_staging.sh

Check if the uwsgi staging server works.

curl localhost:8080

The staging server log is stored in the "back-end/log/uwsgi_staging.log" file. Refer to the "back-end/www/uwsgi_staging.ini" file for details. The documentation is on the uwsgi website. A custom log is stored in the "back-end/log/app.log" file.

# Keep printing the log files when updated
tail -f ../log/uwsgi_staging.log
tail -f ../log/app.log

Create a service on Ubuntu, so that the uwsgi server will start automatically after rebooting the system. Replace [PATH] with the path to the cloned repository. Replace [USERNAME] with your user name on Ubuntu.

sudo vim /etc/systemd/system/spotdiff-staging.service
# Add the following line to this file
[Unit]
Description=uWSGI instance to serve spotdiff
After=network.target

[Service]
User=[USERNAME]
Group=www-data
WorkingDirectory=/[PATH]/spotdiff/back-end/www
Environment="PATH=/home/[USERNAME]/.conda/envs/spotdiff/bin"
ExecStart=/home/[USERNAME]/.conda/envs/spotdiff/bin/uwsgi --ini uwsgi_staging.ini

[Install]
WantedBy=multi-user.target

Register the uwsgi staging server as a service on Ubuntu.

sudo systemctl enable spotdiff-staging
sudo systemctl start spotdiff-staging

# Check the status of the service
sudo systemctl status spotdiff-staging

# Restart the service
sudo systemctl restart spotdiff-staging

# Stop and disable the service
sudo systemctl stop spotdiff-staging
sudo systemctl disable spotdiff-staging

Check if the service work.

curl localhost:8080

The procedure of deploying the production server is the same as deploying the staging server (with differences in replacing the "staging" text with "production"). When the back-end code repository on the staging or production server is updated, run the following to restart the deployed service.

# Restart the uwsgi service
sudo systemctl restart spotdiff-staging
sudo systemctl restart spotdiff-production

# If error happend, check the uwsgi log files
tail -100 spotdiff/back-end/log/uwsgi_staging.log
tail -100 spotdiff/back-end/log/uwsgi_production.log

# Restart the apache service
sudo systemctl restart apache2

API calls

The following code examples assusme that the root url is http://localhost:5000.

Get the unique user token from the system

The server will return a user token in the form of JWT (JSON Web Token).

• Path:
  • /user/
• Available methods:
  • POST
• Required fields:
  • "client_id": the client ID provided by the front-end client
• Returned fields:
  • "user_token": user token provided by the back-end server

// jQuery examples
$.ajax({
  url: "http://localhost:5000/user/",
  type: "POST",
  data: JSON.stringify({client_id: "uid_for_testing"}),
  contentType: "application/json",
  dataType: "json",
  success: function (data) {console.log(data)},
  error: function (xhr) {console.error(xhr)}
});