Wild Me - Houston Backend Server

Houston is a REST API component within the CODEX application suite. It is used to glue the frontend to Ecological Data Module (EDM) and Wildbook-IA.

For a high-level explanation of the application in relation to other CODEX applications read the documentation at CODEX - Houston.

About this implementation

This project showcases my vision on how the RESTful API server should be implemented.

The goals that were achieved in this example:

• RESTful API server should be self-documented using OpenAPI (fka Swagger) specifications, so interactive documentation UI is in place
• Authentication is handled with OAuth2 and using Resource Owner Password Credentials Grant (Password Flow) for first-party clients makes it usable not only for third-party "external" apps
• Permissions are handled (and automaticaly documented)
• PATCH method can be handled accordingly to RFC 6902
• Extensive testing with good code coverage.

The package Flask-RESTX has been patched (see flask_restx_patched folder), so it can handle Marshmallow schemas and Webargs arguments.

Code Style and Development Guidelines

See Contributing to Houston

Project's Code Structure

See Project Structure

Background and Periodic Tasks using Celery

See Background and Periodic Tasks

Installation

Using docker-compose (recommended)

Setup

git clone --recurse-submodules https://github.com/WildMeOrg/houston.git

# Option 1 - Activate Codex App
./scripts/codex/activate.sh
docker-compose up

# Option 2 - Use Codex Config Explicitly
docker-compose -f docker-compose.codex.yml --env-file .env.codex up

Surf to http://localhost:84/. If you are having issues, see the docker-compose debugging docs.

Installing from source

Development Setup Note

Installation of Houston and the other components of Codex from source is intended to facilitate development leveraging the docker-compose environment.

See Development Environment section in Contributing to Houston for details. Full deployment of Codex outside docker-compose orchestration is not supported, and any changes should not be considered finished until they have been tested in the docker-compose environment.

Clone the Project

git clone --recurse-submodules https://github.com/WildMeOrg/houston.git
cd houston/

Setup Codex Environment

It is recommended to use virtualenv or Anaconda/Miniconda to manage Python dependencies. Please, learn details yourself. For quickstart purposes the following will set up a virtualenv for you:

./scripts/codex/venv.sh
source virtualenv/houston3.7/bin/activate

# To add bash-completion
export SCRIPT="$(pwd)/.invoke-completion.sh"
invoke --print-completion-script bash > $SCRIPT
echo "source $SCRIPT" >> virtualenv/houston3.7/bin/activate

Set up and install the package:

invoke dependencies.install

Run Server

NOTE: All dependencies and database migrations will be automatically handled, so go ahead and turn the server ON! (Read more details on this in Tips section)

export HOUSTON_APP_CONTEXT=codex
$ invoke app.run

Deploy Server

In general, you deploy this app as any other Flask/WSGI application. There are a few basic deployment strategies documented in the ./deploy/ folder.

Usage

Open online interactive API documentation: http://127.0.0.1:5000/api/v1/

Autogenerated swagger config is always available from http://127.0.0.1:5000/api/v1/swagger.json

NOTE: Use On/Off switch in documentation to sign in.

Usage Tips

Once you have invoke, you can learn all available commands related to this project from:

$ invoke --list

Learn more about each command with the following syntax:

$ invoke --help <task>

For example:

$ invoke --help codex.run
Usage: inv[oke] [--core-opts] codex.run [--options] [other tasks here ...]

Docstring:
  Run DDOTS RESTful API Server.

Options:
  -d, --[no-]development
  -h STRING, --host=STRING
  -i, --[no-]install-dependencies
  -p, --port
  -u, --[no-]upgrade-db

Use the following command to enter ipython shell (ipython must be installed):

$ invoke app.env.enter

codex.run and app.env.enter tasks automatically prepare all dependencies (using pip install) and migrate database schema to the latest version.

Database schema migration is handled via app.db.* tasks group. The most common migration commands are app.db.upgrade (it is automatically run on codex.run), and app.db.migrate (creates a new migration).

You can use better_exceptions package to enable detailed tracebacks. Just add better_exceptions to the app/requirements.txt and import better_exceptions in the app/__init__.py.

Dependencies

Project Dependencies

This project requires either Python >= 3.7 or Docker.

The tus portions of this application require Redis to facilitate resumable file uploads.

GitLab (community edition) is required for asset and submission storage and management.

Postgres is an optional dependency that can be used for a highly reliable scaled database solution.

Documentation

Build the documentation

To build and view the documentation use the following commands:

cd docs
pip install -r requirements.txt
make html
open _build/html/index.html

Site Settings

There are some site settings that some features are dependent on:

Site setting name	Environment variable
flatfileKey	FLATFILE_KEY	Flatfile API key for bulk upload
recaptchaPublicKey	RECAPTCHA_PUBLIC_KEY	Recaptcha site key (disabled if empty)
recaptchaSecretKey	RECAPTCHA_SECRET_KEY	Recaptcha secret key
googleMapsApiKey	GOOGLE_MAPS_API_KEY	Google maps API key (sighting report form)
email_service	DEFAULT_EMAIL_SERVICE	e.g. "mailchimp"
email_service_username	DEFAULT_EMAIL_SERVICE_USERNAME	mailchimp username
email_service_password	DEFAULT_EMAIL_SERVICE_PASSWORD	mailchimp password

The way these site settings work is:

• look in the database table site_setting for key, return value if exists
• if not in the database, return the environment variable

For example, you can set the environment variables in the .env file or use docker-compose.override.yml to override the environment variables without having to edit any checked in files. Run docker-compose up -d to update any affected containers.

reCAPTCHA

Register at https://www.google.com/recaptcha/admin/create for reCAPTCHA v3.

Add the site (public) key and secret key to docker-compose.override.yml:

services:
  houston:
    environment:
      RECAPTCHA_PUBLIC_KEY: "recaptcha-public-key"
      RECAPTCHA_SECRET_KEY: "recaptcha-secret-key"

docker-compose up -d to update running services.

Settings can also be set via SiteSetting, the keys are recaptchaPublicKey and recaptchaSecretKey.

License

This software is subject to the provisions of Apache License Version 2.0 (APL). See LICENSE for details. Copyright (c) 2020 Wild Me