Skip to content

Latest commit

 

History

History
274 lines (196 loc) · 8.39 KB

README.md

File metadata and controls

274 lines (196 loc) · 8.39 KB
Raw

WES-ELIXIR

Synopsis

Flask microservice implementing the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification.

Description

WES-ELIXIR is an implementation of the GA4GH WES OpenAPI specification based on Flask and Connexion. It allows clients/users to send workflows for execution, list current and previous workflow runs, and get the status and/or detailed information on individual workflow runs. It interprets workflows and breaks them down into individual tasks, for each task emitting a request that is compatible with the GA4GH Task Execution Service (TES) OpenAPI specification. Thus, for end-to-end execution of workflows, a local or remote instance of a TES service such as TESK or funnel is required.

The service is backed by a MongoDB database and optionally uses JWT token-based authorization, e.g. through ELIXIR AAI. While currently only workflows written in the Common Workflow Language are supported (leveraged by CWL-TES, we are planning to abstract workflow interpretation away from API business logic on the one hand and task execution on the other, thus hoping to provide an abstract middleware layer that can be interfaced by any workflow language interpreter in a pluggable manner.

Note that the project is currently still under active development. Nevertheless, a largely FUNCTIONAL PROTOTYPE is available as of October 2018, hosted at the CSC in Helsinki.

WES-ELIXIR is part of ELIXIR, a multinational effort at establishing and implementing FAIR data sharing and promoting reproducible data analyses and responsible data handling in the Life Sciences. Infrastructure and IT support are provided by ELIXIR Finland at the CSC, the TESK service is being developed and maintained by ELIXIR UK at the EBI in Hinxton, and WES-ELIXIR itself is being developed by ELIXIR Switzerland at the Biozentrum in Basel and the Swiss Institute of Bioinformatics.

Installation

Docker

Requirements (Docker)

Ensure you have the following software installed:

  • Docker (18.06.1-ce, build e68fc7a)
  • docker-compose (1.22.0, build f46880fe)
  • Git (1.8.3.1)

Note: These are the versions used for development/testing. Other versions may or may not work.

Instructions (Docker)

Create data directory and required subdiretories

mkdir -p data/db data/output data/tmp

Clone repository

git clone https://github.com/elixir-europe/WES-ELIXIR.git app

Traverse to app directory

cd app

Place a .netrc file for access to a FTP server in app directory. Don't forget to replace <USERNAME> and <PASSWORD> with real values.

cat << EOF > .netrc
machine ftp-private.ebi.ac.uk
login <USERNAME>
password <PASSWORD>
EOF

Optional: edit default and override app config

vi wes_elixir/config/app_config.yaml
vi wes_elixir/config/override/app_config.dev.yaml  # for development service
vi wes_elixir/config/override/app_config.prod.yaml  # for production server

Build container image

docker-compose build

Run docker-compose services in detached/daemonized mode

docker-compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d  # for development service
docker-compose -f docker-compose.yaml -f docker-compose.prod.yaml up -d  # for production service

Visit Swagger UI

firefox http://localhost:7777/ga4gh/wes/v1/ui

Non-dockerized

Requirements (non-dockerized)

Ensure you have the following software installed:

  • curl (7.47.0)
  • Git (2.7.4)
  • MongoDB (4.0.1)
  • Python3 (3.5.2)
  • RabbitMQ (3.5.7)
  • virtualenv (16.0.0)

Note: These are the versions used for development/testing. Other versions may or may not work.

Instructions (non-dockerized)

Ensure RabbitMQ is running (actual command is OS-dependent)

sudo service rabbitmq-server status

Start MongoDB daemon (actual command is OS-dependent)

sudo service mongod start

Place a .netrc file for access to a FTP server in your $HOME directory. Don't forget to replace <USERNAME> and <PASSWORD> with real values.

cat << EOF > "${HOME}/.netrc"
machine ftp-private.ebi.ac.uk
login <USERNAME>
password <PASSWORD>
EOF

Clone repository

git clone https://github.com/elixir-europe/WES-ELIXIR.git app

Traverse to project directory

cd app
project_dir="$PWD"

Create and activate virtual environment

virtualenv -p `which python3` venv
source venv/bin/activate

Install required packages

pip install -r requirements.txt

Install editable packages

cd "${project_dir}/venv/src/cwl-tes"
python setup.py develop
cd "${project_dir}/venv/src/cwltool"
python setup.py develop
cd "${project_dir}/venv/src/py-tes"
python setup.py develop
cd "$project_dir"

Install app

python setup.py develop

Optionally, override default config by setting environment variable and pointing it to a YAML config file. Ensure the file is accessible.

export WES_CONFIG=<path/to/override/config/file.yaml>

Start service

python wes_elixir/app.py

In another terminal, load virtual environment & start Celery worker for executing background tasks

# Traverse to project directory ("app") first
source venv/bin/activate
cd wes_elixir
celery worker -A celery_worker -E --loglevel=info

Visit Swagger UI

firefox http://localhost:8888/ga4gh/wes/v1/ui

Note: If you have edited WES_CONFIG, ensure that host and port match the values specified in the config file.

Q&A

Coming soon...

Contributing

Join us at the 2018 BioHackathon in Paris, organized by ELIXIR Europe (November 12-16)! Check out our project description.

This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please read these guidelines if you want to contribute. And please mind the code of conduct for all interactions with the community.

Versioning

Development of the app is currently still in alpha stage, and current "versions" are for internal use only. We are aiming to have a fully spec-compliant ("feature complete") version of the app available by the end of 2018. The plan is to then adopt a semantic versioning scheme in which we would shadow WES spec versioning for major and minor versions, and release patched versions intermittently.

License

This project is covered by the Apache License 2.0 also shipped with this repository.

Contact

The project is a collaborative effort under the umbrella of ELIXIR Europe.

Please contact the project leader for inquiries, proposals, questions etc. that are not covered by the Q&A and Contributing sections.

References

See also krini-cwl for an older, more rudimentary, yet functional TES-independent WES implementation that is part of the Krini project and leverages Toil.