diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 0472c35408..e64b466c0f 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -2,63 +2,86 @@ Hi there! Many thanks for taking an interest in improving nf-core/tools. -We try to manage the required tasks for nf-core/tools using GitHub issues, you probably came to this page when creating one. Please use the pre-filled templates to save time. - -However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;) - -> If you need help using or developing nf-core/tools then the best place to ask is the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/). +If you need help then the best place to ask is the [`#tools` channel](https://nfcore.slack.com/channels/tools) on the nf-core Slack. +You can get an invite on the [nf-core website](https://nf-co.re/join/slack/). ## Contribution workflow + If you'd like to write some code for nf-core/tools, the standard workflow is as follows: -1. Check that there isn't already an issue about your idea in the - [nf-core/tools issues](https://github.com/nf-core/tools/issues) to avoid - duplicating work. +1. Check that there isn't [already an issue](https://github.com/nf-core/tools/issues) about your idea to avoid duplicating work. * If there isn't one already, please create one so that others know you're working on this 2. Fork the [nf-core/tools repository](https://github.com/nf-core/tools) to your GitHub account 3. Make the necessary changes / additions within your forked repository 4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged. -If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/). +If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/). -## Style guide -Google provides an excellent [style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md), which -is a best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the -[docstring](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) section, which is in particular -important, as nf-core tool's code documentation is generated out of these automatically. +## Installing dev requirements -In order to test the documentation, you have to install Sphinx on the machine, where the documentation should be generated. +If you want to work with developing the nf-core/tools code, you'll need a couple of extra Python packages. +These are listed in `requirements-dev.txt` and can be installed as follows: -Please follow Sphinx's [installation instruction](http://www.sphinx-doc.org/en/master/usage/installation.html). +```bash +pip install --upgrade -r requirements-dev.txt +``` -Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`, where the `Makefile` is located. +Then install your local fork of nf-core/tools: -The HTML will then be generated in `docs/api/_build/html`. +```bash +pip install -e . +``` + +## Code formatting with Black + +All Python code in nf-core/tools must be passed through the [Black Python code formatter](https://black.readthedocs.io/en/stable/). +This ensures a harmonised code formatting style throughout the package, from all contributors. + +You can run Black on the command line (it's included in `requirements-dev.txt`) - eg. to run recursively on the whole repository: +```bash +black . +``` + +Alternatively, Black has [integrations for most common editors](https://black.readthedocs.io/en/stable/editor_integration.html) +to automatically format code when you hit save. +You can also set it up to run when you [make a commit](https://black.readthedocs.io/en/stable/version_control_integration.html). + +There is an automated CI check that runs when you open a pull-request to nf-core/tools that will fail if +any code does not adhere to Black formatting. + +## API Documentation + +We aim to write function docstrings according to the [Google Python style-guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings). These are used to automatically generate package documentation on the nf-core website using Sphinx. +You can find this documentation here: [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/) + +If you would like to test the documentation, you can install Sphinx locally by following Sphinx's [installation instruction](https://www.sphinx-doc.org/en/master/usage/installation.html). +Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`. +The HTML will then be generated in `docs/api/_build/html`. ## Tests + When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests. Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then. There are two types of tests that run: ### Unit Tests + The nf-core tools package has a set of unit tests bundled, which can be found in the `tests/` directory. New features should also come with new tests, to keep the test-coverage high (we use [codecov.io](https://codecov.io/gh/nf-core/tools/) to check this automatically). You can try running the tests locally before pushing code using the following command: ```bash -python -m pytest . +pytest --color=yes tests/ ``` ### Lint Tests -nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to. -To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint ` command. -The nf-core/tools repo itself contains the master template for creating new nf-core pipelines. -Once you have created a new pipeline from this template GitHub Actions is automatically set up to run lint tests on it. +nf-core/tools contains both the main nf-core template for pipelines and the code used to test that pipelines adhere to the nf-core guidelines. +As these two commonly need to be edited together, we test the creation of a pipeline and then linting using a CI check. This ensures that any changes we make to either the linting or the template stay in sync. You can replicate this process locally with the following commands: @@ -66,6 +89,3 @@ You can replicate this process locally with the following commands: nf-core create -n testpipeline -d "This pipeline is for testing" nf-core lint nf-core-testpipeline ``` - -## Getting help -For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/). diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 9ad9e59a74..8f7661bd76 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,12 +1,17 @@ + ## PR checklist + - [ ] This comment contains a description of changes (with reason) + - [ ] `CHANGELOG.md` is updated - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] Documentation in `docs` is updated - - [ ] `CHANGELOG.md` is updated - - [ ] `README.md` is updated - -**Learn more about contributing:** https://github.com/nf-core/tools/tree/master/.github/CONTRIBUTING.md diff --git a/.github/RELEASE_CHECKLIST.md b/.github/RELEASE_CHECKLIST.md index 4325eeb757..00b6f94730 100644 --- a/.github/RELEASE_CHECKLIST.md +++ b/.github/RELEASE_CHECKLIST.md @@ -1,12 +1,12 @@ ## Before release 1. Check issue milestones to see outstanding issues to resolve if possible or transfer to the milestones for the next release e.g. [`v1.9`](https://github.com/nf-core/tools/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.9) -2. Create a PR to `dev` to bump the version in `CHANGELOG.md` and `setup.py`. -3. Make sure all CI tests are passing! -4. Create a PR from `dev` to `master` -5. Make sure all CI tests are passing again (additional tests are run on PRs to `master`) -6. Request review (2 approvals required) -7. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*. +2. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*. +3. Create a PR to `dev` to bump the version in `CHANGELOG.md` and `setup.py`. +4. Make sure all CI tests are passing! +5. Create a PR from `dev` to `master` +6. Make sure all CI tests are passing again (additional tests are run on PRs to `master`) +7. Request review (2 approvals required) 8. Merge the PR into `master` 9. Once CI tests on commit have passed, create a new release copying the `CHANGELOG` for that release into the description section. diff --git a/.github/markdownlint.yml b/.github/markdownlint.yml index c6b3f58f08..a0eac5a096 100644 --- a/.github/markdownlint.yml +++ b/.github/markdownlint.yml @@ -1,7 +1,9 @@ # Markdownlint configuration file -default: true, +default: true line-length: false no-duplicate-header: siblings_only: true -no-bare-urls: false # tools only - the {{ jinja variables }} break URLs and cause this to error -commands-show-output: false # tools only - suppresses error messages for usage of $ in main README +# tools only - the {{ jinja variables }} break URLs and cause this to error +no-bare-urls: false +# tools only - suppresses error messages for usage of $ in main README +commands-show-output: false diff --git a/.github/workflows/branch.yml b/.github/workflows/branch.yml new file mode 100644 index 0000000000..b0dd5cc25a --- /dev/null +++ b/.github/workflows/branch.yml @@ -0,0 +1,35 @@ +name: nf-core branch protection +# This workflow is triggered on PRs to master branch on the repository +# It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev` +on: + pull_request: + branches: [master] + +jobs: + test: + runs-on: ubuntu-latest + steps: + + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches + - name: Check PRs + if: github.repository == 'nf-core/tools' + run: | + { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + + # If the above check failed, post a comment on the PR explaining the failure + - name: Post PR comment + if: failure() + uses: mshick/add-pr-comment@v1 + with: + message: | + Hi @${{ github.event.pull_request.user.login }}, + + It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. + The `master` branch on nf-core repositories should always contain code from the latest release. + Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch. + + You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page. + + Thanks again for your contribution! + repo-token: ${{ secrets.GITHUB_TOKEN }} + allow-repeats: false diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index b029e0c42f..ce6a3672fe 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -25,7 +25,13 @@ jobs: wget -qO- get.nextflow.io | bash sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow - - name: Run nf-core tools + - name: Run nf-core/tools run: | nf-core create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" nf-core lint nf-core-testpipeline + nf-core list + nf-core licences nf-core-testpipeline + nf-core sync nf-core-testpipeline/ + nf-core schema build nf-core-testpipeline/ --no-prompts + nf-core bump-version nf-core-testpipeline/ 1.1 + nf-core modules install nf-core-testpipeline/ fastqc diff --git a/.github/workflows/code-tests.yml b/.github/workflows/pytest.yml similarity index 63% rename from .github/workflows/code-tests.yml rename to .github/workflows/pytest.yml index 7ea31ab9e3..c87b3b55f8 100644 --- a/.github/workflows/code-tests.yml +++ b/.github/workflows/pytest.yml @@ -1,9 +1,16 @@ name: Python tests # This workflow is triggered on pushes and PRs to the repository. -on: [push, pull_request] +# Only run if we changed a Python file +on: + push: + paths: + - '**.py' + pull_request: + paths: + - '**.py' jobs: - PythonLint: + pytest: runs-on: ubuntu-latest strategy: @@ -21,8 +28,8 @@ jobs: - name: Install python dependencies run: | - python -m pip install --upgrade pip pytest pytest-datafiles pytest-cov mock jsonschema - pip install . + python -m pip install --upgrade pip -r requirements-dev.txt + pip install -e . - name: Install Nextflow run: | @@ -31,14 +38,6 @@ jobs: wget -qO- get.nextflow.io | bash sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow - - name: Lint with flake8 - run: | - pip install flake8 - # stop the build if there are Python syntax errors or undefined names - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics - # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - - name: Test with pytest run: python3 -m pytest --color=yes --cov-report=xml --cov-config=.github/.coveragerc --cov=nf_core diff --git a/.github/workflows/python-lint.yml b/.github/workflows/python-lint.yml new file mode 100644 index 0000000000..4c8b5b00f9 --- /dev/null +++ b/.github/workflows/python-lint.yml @@ -0,0 +1,17 @@ +name: Lint Python +on: + push: + paths: + - '**.py' + pull_request: + paths: + - '**.py' + +jobs: + PythonLint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - name: Check code lints with Black + uses: jpetrucciani/black-check@master diff --git a/.github/workflows/sync.yml b/.github/workflows/sync.yml index c395c9c4f0..f8063c53d8 100644 --- a/.github/workflows/sync.yml +++ b/.github/workflows/sync.yml @@ -12,10 +12,10 @@ jobs: - uses: actions/checkout@v2 name: Check out source-code repository - - name: Set up Python 3.7 + - name: Set up Python 3.8 uses: actions/setup-python@v1 with: - python-version: 3.7 + python-version: 3.8 - name: Install python dependencies run: | @@ -29,6 +29,11 @@ jobs: wget -qO- get.nextflow.io | bash sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow + - name: Configure git + run: | + git config user.email "core@nf-co.re" + git config user.name "nf-core-bot" + - name: Run synchronisation if: github.repository == 'nf-core/tools' env: diff --git a/.github/workflows/tools-api-docs.yml b/.github/workflows/tools-api-docs.yml index 2b75e9421a..403e1e3878 100644 --- a/.github/workflows/tools-api-docs.yml +++ b/.github/workflows/tools-api-docs.yml @@ -4,8 +4,8 @@ on: branches: [master, dev] jobs: - build-n-publish: - name: Build and publish nf-core to PyPI + api-docs: + name: Build & push Sphinx API docs runs-on: ubuntu-18.04 steps: diff --git a/.gitignore b/.gitignore index 27c031f40c..bc2d21a3d5 100644 --- a/.gitignore +++ b/.gitignore @@ -41,6 +41,7 @@ pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports +testing* htmlcov/ .tox/ .coverage @@ -107,3 +108,4 @@ ENV/ # backup files *~ +*\? diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ce251bddc..88a2aefb90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,107 @@ # nf-core/tools: Changelog -## v1.9 +## [v1.10 - Copper Camel](https://github.com/nf-core/tools/releases/tag/1.10) - [2020-07-30] + +### Pipeline schema + +This release of nf-core/tools introduces a major change / new feature: pipeline schema. +These are [JSON Schema](https://json-schema.org/) files that describe all of the parameters for a given +pipeline with their ID, a description, a longer help text, an optional default value, a variable _type_ +(eg. `string` or `boolean`) and more. + +The files will be used in a number of places: + +* Automatic validation of supplied parameters when running pipelines + * Pipeline execution can be immediately stopped if a required `param` is missing, + or does not conform to the patterns / allowed values in the schema. +* Generation of pipeline command-line help + * Running `nextflow run --help` will use the schema to generate a help text automatically +* Building online documentation on the [nf-core website](https://nf-co.re) +* Integration with 3rd party graphical user interfaces + +To support these new schema files, nf-core/tools now comes with a new set of commands: `nf-core schema`. + +* Pipeline schema can be generated or updated using `nf-core schema build` - this takes the parameters from + the pipeline config file and prompts the developer for any mismatch between schema and pipeline. + * Once a skeleton Schema file has been built, the command makes use of a new nf-core website tool to provide + a user friendly graphical interface for developers to add content to their schema: [https://nf-co.re/pipeline_schema_builder](https://nf-co.re/pipeline_schema_builder) +* Pipelines will be automatically tested for valid schema that describe all pipeline parameters using the + `nf-core schema lint` command (also included as part of the main `nf-core lint` command). +* Users can validate their set of pipeline inputs using the `nf-core schema validate` command. + +In addition to the new schema commands, the `nf-core launch` command has been completely rewritten from +scratch to make use of the new pipeline schema. This command can use either an interactive command-line +prompt or a rich web interface to help users set parameters for a pipeline run. + +The parameter descriptions and help text are fully used and embedded into the launch interfaces to make +this process as user-friendly as possible. We hope that it's particularly well suited to those new to nf-core. + +Whilst we appreciate that this new feature will add a little work for pipeline developers, we're excited at +the possibilities that it brings. If you have any feedback or suggestions, please let us know either here on +GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com/channels/json-schema). + +### Python code formatting + +We have adopted the use of the [Black Python code formatter](https://black.readthedocs.io/en/stable/). +This ensures a harmonised code formatting style throughout the package, from all contributors. +If you are editing any Python code in nf-core/tools you must now pass the files through Black when +making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) for details. + +### Template + +* Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) +* Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) +* Added new style of pipeline parameters JSON schema to pipeline template +* Add ability to attach MultiQC reports to completion emails when using `mail` +* Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. +* Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) +* Add Slack channel badge in pipeline README +* Allow multiple container tags in `ci.yml` if performing multiple tests in parallel +* Add AWS CI tests and full tests GitHub Actions workflows +* Update AWS CI tests and full tests secrets names +* Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) +* Add information about config files used for workflow execution (`workflow.configFiles`) to summary +* Fix `markdown_to_html.py` to work with Python 2 and 3. +* Change `params.reads` -> `params.input` +* Change `params.readPaths` -> `params.input_paths` +* Added a `.github/.dockstore.yml` config file for automatic workflow registration with [dockstore.org](https://dockstore.org/) + +### Linting + +* Refactored PR branch tests to be a little clearer. +* Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. +* Adapted linting docs to the new PR branch tests. +* Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant +* Added test for template `{{ cookiecutter.var }}` placeholders +* Fix failure when providing version along with build id for Conda packages +* New `--json` and `--markdown` options to print lint results to JSON / markdown files +* Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can +* Added AWS GitHub Actions workflows linting +* Fail if `params.input` isn't defined. +* Beautiful new progress bar to look at whilst linting is running and awesome new formatted output on the command line :heart_eyes: + * All made using the excellent [`rich` python library](https://github.com/willmcgugan/rich) - check it out! +* Tests looking for `TODO` strings should now ignore editor backup files. [#477](https://github.com/nf-core/tools/issues/477) + +### nf-core/tools Continuous Integration + +* Added CI test to check for PRs against `master` in tools repo +* CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong +* Move some of the issue and PR templates into HTML `` so that they don't show in issues / PRs + +### Other + +* Describe alternative installation method via conda with `conda env create` +* nf-core/tools version number now printed underneath header artwork +* Bumped Conda version shipped with nfcore/base to 4.8.2 +* Added log message when creating new pipelines that people should talk to the community about their plans +* Fixed 'on completion' emails sent using the `mail` command not containing body text. +* Improved command-line help text for nf-core/tools +* `nf-core list` now hides archived pipelines unless `--show_archived` flag is set +* Command line tools now checks if there is a new version of nf-core/tools available + * Disable this by setting the environment variable `NFCORE_NO_VERSION_CHECK`, eg. `export NFCORE_NO_VERSION_CHECK=1` +* Better command-line output formatting of nearly all `nf-core` commands using [`rich`](https://github.com/willmcgugan/rich) + +## [v1.9 - Platinum Pigeon](https://github.com/nf-core/tools/releases/tag/1.9) - [2020-02-20] ### Continuous integration @@ -13,6 +114,7 @@ ### Template * Rewrote the documentation markdown > HTML conversion in Python instead of R +* Fixed rendering of images in output documentation [#391](https://github.com/nf-core/tools/issues/391) * Removed the requirement for R in the conda environment * Make `params.multiqc_config` give an _additional_ MultiQC config file instead of replacing the one that ships with the pipeline * Ignore only `tests/` and `testing/` directories in `.gitignore` to avoid ignoring `test.config` configuration file @@ -42,7 +144,7 @@ * Add social preview image * Added a [release checklist](.github/RELEASE_CHECKLIST.md) for the tools repo -## v1.8 +## [v1.8 - Black Sheep](https://github.com/nf-core/tools/releases/tag/1.8) - [2020-01-27] ### Continuous integration @@ -114,7 +216,7 @@ * Entirely switched from Travis-Ci.org to Travis-Ci.com for template and tools * Improved core documentation (`-profile`) -## v1.7 +## [v1.7 - Titanium Kangaroo](https://github.com/nf-core/tools/releases/tag/1.7) - [2019-10-07] ### Tools helper code @@ -177,7 +279,7 @@ * Added a Code of Conduct to nf-core/tools, as only the template had this before * TravisCI tests will now also start for PRs from `patch` branches, [to allow fixing critical issues](https://github.com/nf-core/tools/pull/392) without making a new major release -## v1.6 +## [v1.6 - Brass Walrus](https://github.com/nf-core/tools/releases/tag/1.6) - [2020-04-09] ### Syncing @@ -215,7 +317,7 @@ * As a solution for [#103](https://github.com/nf-core/tools/issues/103)) * Add Bowtie2 and BWA in iGenome config file template -## [v1.5](https://github.com/nf-core/tools/releases/tag/1.5) - 2019-03-13 Iron Shark +## [v1.5 - Iron Shark](https://github.com/nf-core/tools/releases/tag/1.5) - [2019-03-13] ### Template pipeline @@ -260,7 +362,7 @@ * Bump `conda` to 4.6.7 in base nf-core Dockerfile -## [v1.4](https://github.com/nf-core/tools/releases/tag/1.4) - 2018-12-12 Tantalum Butterfly +## [v1.4 - Tantalum Butterfly](https://github.com/nf-core/tools/releases/tag/1.4) - [2018-12-12] ### Template pipeline @@ -285,7 +387,7 @@ * Handle exception if nextflow isn't installed * Linting: Update for Travis: Pull the `dev` tagged docker image for testing -## [v1.3](https://github.com/nf-core/tools/releases/tag/1.3) - 2018-11-21 +## [v1.3 - Citreous Swordfish](https://github.com/nf-core/tools/releases/tag/1.3) - [2018-11-21] * `nf-core create` command line interface updated * Interactive prompts for required arguments if not given @@ -299,7 +401,7 @@ * Ordering alphabetically for profiles now * Added `pip install --upgrade pip` to `.travis.yml` to update pip in the Travis CI environment -## [v1.2](https://github.com/nf-core/tools/releases/tag/1.2) - 2018-10-01 +## [v1.2](https://github.com/nf-core/tools/releases/tag/1.2) - [2018-10-01] * Updated the `nf-core release` command * Now called `nf-core bump-versions` instead @@ -320,7 +422,7 @@ * Updated PyPI deployment to correctly parse the markdown readme (hopefully!) * New GitHub contributing instructions and pull request template -## [v1.1](https://github.com/nf-core/tools/releases/tag/1.1) - 2018-08-14 +## [v1.1](https://github.com/nf-core/tools/releases/tag/1.1) - [2018-08-14] Very large release containing lots of work from the first nf-core hackathon, held in SciLifeLab Stockholm. @@ -340,11 +442,11 @@ Very large release containing lots of work from the first nf-core hackathon, hel * New sync tool to automate pipeline updates * Once initial merges are complete, a nf-core bot account will create PRs for future template updates -## [v1.0.1](https://github.com/nf-core/tools/releases/tag/1.0.1) - 2018-07-18 +## [v1.0.1](https://github.com/nf-core/tools/releases/tag/1.0.1) - [2018-07-18] The version 1.0 of nf-core tools cannot be installed from PyPi. This patch fixes it, by getting rid of the requirements.txt plus declaring the dependent modules in the setup.py directly. -## [v1.0](https://github.com/nf-core/tools/releases/tag/1.0) - 2018-06-12 +## [v1.0](https://github.com/nf-core/tools/releases/tag/1.0) - [2018-06-12] Initial release of the nf-core helper tools package. Currently includes four subcommands: diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 1cda760094..7d8e03ed8f 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -40,7 +40,7 @@ Project maintainers who do not follow or enforce the Code of Conduct in good fai ## Attribution -This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://contributor-covenant.org/version/1/4][version] -[homepage]: http://contributor-covenant.org -[version]: http://contributor-covenant.org/version/1/4/ +[homepage]: https://contributor-covenant.org +[version]: https://contributor-covenant.org/version/1/4/ diff --git a/Dockerfile b/Dockerfile index f8b138e9ed..80da351807 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,9 @@ -FROM continuumio/miniconda3:4.7.12 +FROM continuumio/miniconda3:4.8.2 LABEL authors="phil.ewels@scilifelab.se,alexander.peltzer@qbic.uni-tuebingen.de" \ description="Docker image containing base requirements for the nfcore pipelines" -# Install procps so that Nextflow can poll CPU usage -RUN apt-get update && apt-get install -y procps && apt-get clean -y +# Install procps so that Nextflow can poll CPU usage and +# deep clean the apt cache to reduce image/layer size +RUN apt-get update \ + && apt-get install -y procps \ + && apt-get clean -y && rm -rf /var/lib/apt/lists/* \ No newline at end of file diff --git a/README.md b/README.md index 3c519f17cf..99a8c32573 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,17 @@ # ![nf-core/tools](docs/images/nfcore-tools_logo.png) -[![GitHub Actions CI Status](https://github.com/nf-core/tools/workflows/CI%20tests/badge.svg)](https://github.com/nf-core/tools/actions) +[![Python tests](https://github.com/nf-core/tools/workflows/Python%20tests/badge.svg?branch=master&event=push)](https://github.com/nf-core/tools/actions?query=workflow%3A%22Python+tests%22+branch%3Amaster) [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools) -[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat-square)](http://bioconda.github.io/recipes/nf-core/README.html) +[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) + +[![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/recipes/nf-core/README.html) +[![install with PyPI](https://img.shields.io/badge/install%20with-PyPI-blue.svg)](https://pypi.org/project/nf-core/) +[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23tools-4A154B?logo=slack)](https://nfcore.slack.com/channels/tools) A python package with helper tools for the nf-core community. +> **Read this documentation on the nf-core website: [https://nf-co.re/tools](https://nf-co.re/tools)** + ## Table of contents * [`nf-core` tools installation](#installation) @@ -15,6 +21,7 @@ A python package with helper tools for the nf-core community. * [`nf-core licences` - List software licences in a pipeline](#pipeline-software-licences) * [`nf-core create` - Create a new workflow from the nf-core template](#creating-a-new-workflow) * [`nf-core lint` - Check pipeline code against nf-core guidelines](#linting-a-workflow) +* [`nf-core schema` - Work with pipeline schema files](#working-with-pipeline-schema) * [`nf-core bump-version` - Update nf-core pipeline version number](#bumping-a-pipeline-version-number) * [`nf-core sync` - Synchronise pipeline TEMPLATE branches](#sync-a-pipeline-with-the-template) * [Citation](#citation) @@ -24,42 +31,84 @@ For documentation of the internal Python functions, please refer to the [Tools P ## Installation -You can install `nf-core/tools` using [bioconda](https://bioconda.github.io/recipes/nf-core/README.html): +### Bioconda + +You can install `nf-core/tools` from [bioconda](https://bioconda.github.io/recipes/nf-core/README.html). + +First, install conda and configure the channels to use bioconda +(see the [bioconda documentation](https://bioconda.github.io/user/install.html)). +Then, just run the conda installation command: + +```bash +conda install nf-core +``` + +Alternatively, you can create a new environment with both nf-core/tools and nextflow: ```bash -conda install -c bioconda nf-core +conda create --name nf-core python=3.7 nf-core nextflow +conda activate nf-core ``` -It can also be installed from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows: +### Python Package Index + +`nf-core/tools` can also be installed from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows: ```bash pip install nf-core ``` -Or, if you would like the development version instead, the command is: +### Development version + +If you would like the latest development version of tools, the command is: ```bash pip install --upgrade --force-reinstall git+https://github.com/nf-core/tools.git@dev ``` -Alternatively, if you would like to edit the files locally: -Clone the repository code - you should probably specify your fork instead +If you intend to make edits to the code, first make a fork of the repository and then clone it locally. +Go to the cloned directory and install with pip (also installs development requirements): ```bash -git clone https://github.com/nf-core/tools.git nf-core-tools -cd nf-core-tools +pip install --upgrade -r requirements-dev.txt -e . ``` -Install with pip +### Using a specific Python interpreter + +If you prefer, you can also run tools with a specific Python interpreter. +The command line usage and flags are then exactly the same as if you ran with the `nf-core` command. +Note that the module is `nf_core` with an underscore, not a hyphen like the console command. + +For example: ```bash -pip install -e . +python -m nf_core --help +python3 -m nf_core list +~/my_env/bin/python -m nf_core create --name mypipeline --description "This is a new skeleton pipeline" +``` + +### Using with your own Python scripts + +The tools functionality is written in such a way that you can import it into your own scripts. +For example, if you would like to get a list of all available nf-core pipelines: + +```python +import nf_core.list +wfs = nf_core.list.Workflows() +wfs.get_remote_workflows() +for wf in wfs.remote_workflows: + print(wf.full_name) ``` -Alternatively, install the package with Python +Please see [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/) for the function documentation. + +### Automatic version check + +nf-core/tools automatically checks the web to see if there is a new version of nf-core/tools available. +If you would prefer to skip this check, set the environment variable `NFCORE_NO_VERSION_CHECK`. For example: ```bash -python setup.py develop +export NFCORE_NO_VERSION_CHECK=1 ``` ## Listing pipelines @@ -77,22 +126,18 @@ $ nf-core list | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' - -Name Version Released Last Pulled Have latest release? -------------------------- --------- ------------ -------------- ---------------------- -nf-core/rnaseq 1.3 4 days ago 27 minutes ago Yes -nf-core/hlatyping 1.1.4 3 weeks ago 1 months ago No -nf-core/eager 2.0.6 3 weeks ago - - -nf-core/mhcquant 1.2.6 3 weeks ago - - -nf-core/rnafusion 1.0 1 months ago - - -nf-core/methylseq 1.3 1 months ago 3 months ago No -nf-core/ampliseq 1.0.0 3 months ago - - -nf-core/deepvariant 1.0 4 months ago - - -nf-core/atacseq dev - 1 months ago No -nf-core/bacass dev - - - -nf-core/bcellmagic dev - - - -nf-core/chipseq dev - 1 months ago No -nf-core/clinvap dev - - - + nf-core/tools version 1.10 + +┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Pipeline Name ┃ Stars ┃ Latest Release ┃ Released ┃ Last Pulled ┃ Have latest release? ┃ +┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ +│ rnafusion │ 45 │ 1.2.0 │ 2 weeks ago │ - │ - │ +│ hic │ 17 │ 1.2.1 │ 3 weeks ago │ 4 months ago │ No (v1.1.0) │ +│ chipseq │ 56 │ 1.2.0 │ 4 weeks ago │ 4 weeks ago │ No (dev - bfe7eb3) │ +│ atacseq │ 40 │ 1.2.0 │ 4 weeks ago │ 6 hours ago │ No (master - 79bc7c2) │ +│ viralrecon │ 20 │ 1.1.0 │ 1 months ago │ 1 months ago │ Yes (v1.1.0) │ +│ sarek │ 59 │ 2.6.1 │ 1 months ago │ - │ - │ +[..truncated..] ``` To narrow down the list, supply one or more additional keywords to filter the pipelines based on matches in titles, descriptions and topics: @@ -106,13 +151,16 @@ $ nf-core list rna rna-seq | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 -Name Version Released Last Pulled Have latest release? ------------------ --------- ------------ -------------- ---------------------- -nf-core/rnaseq 1.3 4 days ago 28 minutes ago Yes -nf-core/rnafusion 1.0 1 months ago - - -nf-core/lncpipe dev - - - -nf-core/smrnaseq dev - - - +┏━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Pipeline Name ┃ Stars ┃ Latest Release ┃ Released ┃ Last Pulled ┃ Have latest release? ┃ +┡━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ +│ rnafusion │ 45 │ 1.2.0 │ 2 weeks ago │ - │ - │ +│ rnaseq │ 207 │ 1.4.2 │ 9 months ago │ 5 days ago │ Yes (v1.4.2) │ +│ smrnaseq │ 12 │ 1.0.0 │ 10 months ago │ - │ - │ +│ lncpipe │ 18 │ dev │ - │ - │ - │ +└───────────────┴───────┴────────────────┴───────────────┴─────────────┴──────────────────────┘ ``` You can sort the results by latest release (`-s release`, default), @@ -129,22 +177,24 @@ $ nf-core list -s stars | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' - -Name Stargazers Version Released Last Pulled Have latest release? -------------------------- ------------ --------- ------------ -------------- ---------------------- -nf-core/rnaseq 81 1.3 4 days ago 30 minutes ago Yes -nf-core/methylseq 22 1.3 1 months ago 3 months ago No -nf-core/ampliseq 21 1.0.0 3 months ago - - -nf-core/chipseq 20 dev - 1 months ago No -nf-core/deepvariant 15 1.0 4 months ago - - -nf-core/eager 14 2.0.6 3 weeks ago - - -nf-core/rnafusion 14 1.0 1 months ago - - -nf-core/lncpipe 9 dev - - - -nf-core/exoseq 8 dev - - - -nf-core/mag 8 dev - - - + nf-core/tools version 1.10 + +┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Pipeline Name ┃ Stars ┃ Latest Release ┃ Released ┃ Last Pulled ┃ Have latest release? ┃ +┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ +│ rnaseq │ 207 │ 1.4.2 │ 9 months ago │ 5 days ago │ Yes (v1.4.2) │ +│ sarek │ 59 │ 2.6.1 │ 1 months ago │ - │ - │ +│ chipseq │ 56 │ 1.2.0 │ 4 weeks ago │ 4 weeks ago │ No (dev - bfe7eb3) │ +│ methylseq │ 47 │ 1.5 │ 4 months ago │ - │ - │ +│ rnafusion │ 45 │ 1.2.0 │ 2 weeks ago │ - │ - │ +│ ampliseq │ 41 │ 1.1.2 │ 7 months ago │ - │ - │ +│ atacseq │ 40 │ 1.2.0 │ 4 weeks ago │ 6 hours ago │ No (master - 79bc7c2) │ +[..truncated..] ``` -Finally, to return machine-readable JSON output, use the `--json` flag. +To return results as JSON output for downstream use, use the `--json` flag. + +Archived pipelines are not returned by default. To include them, use the `--show_archived` flag. ## Launch a pipeline @@ -152,13 +202,14 @@ Some nextflow pipelines have a considerable number of command line flags that ca To help with this, the `nf-core launch` command uses an interactive command-line wizard tool to prompt you for values for running nextflow and the pipeline parameters. -If the pipeline in question has a `parameters.settings.json` file following the [nf-core parameter JSON schema](https://nf-co.re/parameter-schema), parameters will be grouped and have associated description text and variable typing. +The tool uses the `nextflow_schema.json` file from a pipeline to give parameter descriptions, defaults and grouping. +If no file for the pipeline is found, one will be automatically generated at runtime. -Nextflow `params` variables are saved in to a JSON file called `nfx-params.json` and used by nextflow with the `-params-file` flag. +Nextflow `params` variables are saved in to a JSON file called `nf-params.json` and used by nextflow with the `-params-file` flag. This makes it easier to reuse these in the future. -It is not essential to run the pipeline - the wizard will ask you if you want to launch the command at the end. -If not, you finish with the `params` JSON file and a nextflow command that you can copy and paste. +The `nf-core launch` command is an interactive command line tool and prompts you to overwrite the default values for each parameter. +Entering `?` for any parameter will give a full description from the documentation of what that value does. ```console $ nf-core launch rnaseq @@ -169,48 +220,60 @@ $ nf-core launch rnaseq | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 -INFO: Launching nf-core/rnaseq -Main nextflow options - -Config profile to use - -profile [standard]: docker - -Unique name for this nextflow run - -name [None]: test_run - -Work directory for intermediate files - -w [./work]: - -Resume a previous workflow run - -resume [y/N]: -Release / revision to use - -r [None]: 1.3 +INFO: [✓] Pipeline schema looks valid +INFO: This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles -Parameter group: Main options -Do you want to change the group's defaults? [y/N]: y +? Nextflow command-line flags (Use arrow keys) + ❯ Continue >> + --------------- + -name + -revision + -profile + -work-dir + -resume +``` -Input files -Specify the location of your input FastQ files. - --reads ['data/*{1,2}.fastq.gz']: '/path/to/reads_*{R1,R2}.fq.gz' +Once complete, the wizard will ask you if you want to launch the Nextflow run. +If not, you can copy and paste the Nextflow command with the `nf-params.json` file of your inputs. -[..truncated..] +```console +? Nextflow command-line flags Continue >> +? Input/output options input -Nextflow command: - nextflow run nf-core/rnaseq -profile "docker" -name "test_run" -r "1.3" -params-file "/Users/ewels/testing/nfx-params.json" +Input FastQ files. (? for help) +? input data/*{1,2}.fq.gz +? Input/output options Continue >> +? Reference genome options Continue >> +INFO: [✓] Input parameters look valid -Do you want to run this command now? [y/N]: y +INFO: Nextflow command: + nextflow run nf-core-testpipeline/ -params-file "nf-params.json" -INFO: Launching workflow! -N E X T F L O W ~ version 19.01.0 -Launching `nf-core/rnaseq` [evil_engelbart] - revision: 37f260d360 [master] -[..truncated..] +Do you want to run this command now? [y/N]: n ``` +### Launch tool options + +* `-c`, `--command-only` + * If you prefer not to save your inputs in a JSON file and use `-params-file`, this option will specify all entered params directly in the nextflow command. +* `-p`, `--params-in PATH` + * To use values entered in a previous pipeline run, you can supply the `nf-params.json` file previously generated. + * This will overwrite the pipeline schema defaults before the wizard is launched. +* `-o`, `--params-out PATH` + * Path to save parameters JSON file to. (Default: `nf-params.json`) +* `-a`, `--save-all` + * Without this option the pipeline will ignore any values that match the pipeline schema defaults. + * This option saves _all_ parameters found to the JSON file. +* `-h`, `--show-hidden` + * A pipeline JSON schema can define some parameters as 'hidden' if they are rarely used or for internal pipeline use only. + * This option forces the wizard to show all parameters, including those labelled as 'hidden'. + ## Downloading pipelines for offline use Sometimes you may need to run an nf-core pipeline on a server or HPC system that has no internet connection. In this case you will need to fetch the pipeline files first, then manually transfer them to your system. @@ -231,31 +294,17 @@ $ nf-core download methylseq -r 1.4 --singularity | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' - -INFO: Saving methylseq - Pipeline release: 1.4 - Pull singularity containers: Yes - Output file: nf-core-methylseq-1.4.tar.gz - -INFO: Downloading workflow files from GitHub - -INFO: Downloading centralised configs from GitHub - -INFO: Downloading 1 singularity container - -INFO: Building singularity image from dockerhub: docker://nfcore/methylseq:1.4 -INFO: Converting OCI blobs to SIF format -INFO: Starting build... -Getting image source signatures -.... -INFO: Creating SIF file... -INFO: Build complete: /my-pipelines/nf-core-methylseq-1.4/singularity-images/nf-core-methylseq-1.4.simg - -INFO: Compressing download.. - -INFO: Command to extract files: tar -xzf nf-core-methylseq-1.4.tar.gz - -INFO: MD5 checksum for nf-core-methylseq-1.4.tar.gz: f5c2b035619967bb227230bc3ec986c5 + nf-core/tools version 1.10 + + INFO Saving methylseq + Pipeline release: 1.4 + Pull singularity containers: No + Output file: nf-core-methylseq-1.4.tar.gz + INFO Downloading workflow files from GitHub + INFO Downloading centralised configs from GitHub + INFO Compressing download.. + INFO Command to extract files: tar -xzf nf-core-methylseq-1.4.tar.gz + INFO MD5 checksum for nf-core-methylseq-1.4.tar.gz: 4d173b1cb97903dbb73f2fd24a2d2ac1 ``` The tool automatically compresses all of the resulting file in to a `.tar.gz` archive. @@ -291,7 +340,7 @@ nf-core-methylseq-1.4 ├── LICENSE ├── main.nf ├── nextflow.config - ├── parameters.settings.json + ├── nextflow_schema.json └── README.md 10 directories, 15 files @@ -309,7 +358,7 @@ nextflow run /path/to/nf-core-methylseq-1.4/workflow/ \ -profile singularity \ -with-singularity /path/to/nf-core-methylseq-1.4/singularity-images/nf-core-methylseq-1.4.simg \ # .. other normal pipeline parameters from here on.. - --reads '*_R{1,2}.fastq.gz' --genome GRCh38 + --input '*_R{1,2}.fastq.gz' --genome GRCh38 ``` ## Pipeline software licences @@ -325,29 +374,40 @@ $ nf-core licences rnaseq | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' - -INFO: Warning: This tool only prints licence information for the software tools packaged using conda. - The pipeline may use other software and dependencies not described here. - -Package Name Version Licence ---------------------- --------- -------------------- -stringtie 1.3.3 Artistic License 2.0 -preseq 2.0.3 GPL -trim-galore 0.4.5 GPL -bioconductor-edger 3.20.7 GPL >=2 -fastqc 0.11.7 GPL >=3 -openjdk 8.0.144 GPLv2 -r-gplots 3.0.1 GPLv2 -r-markdown 0.8 GPLv2 -rseqc 2.6.4 GPLv2 -bioconductor-dupradar 1.8.0 GPLv3 -hisat2 2.1.0 GPLv3 -multiqc 1.5 GPLv3 -r-data.table 1.10.4 GPLv3 -star 2.5.4a GPLv3 -subread 1.6.1 GPLv3 -picard 2.18.2 MIT -samtools 1.8 MIT + nf-core/tools version 1.10 + + INFO Fetching licence information for 25 tools + INFO Warning: This tool only prints licence information for the software tools packaged using conda. + INFO The pipeline may use other software and dependencies not described here. +┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Package Name ┃ Version ┃ Licence ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ +│ stringtie │ 2.0 │ Artistic License 2.0 │ +│ bioconductor-summarizedexperiment │ 1.14.0 │ Artistic-2.0 │ +│ preseq │ 2.0.3 │ GPL │ +│ trim-galore │ 0.6.4 │ GPL │ +│ bioconductor-edger │ 3.26.5 │ GPL >=2 │ +│ fastqc │ 0.11.8 │ GPL >=3 │ +│ bioconductor-tximeta │ 1.2.2 │ GPLv2 │ +│ qualimap │ 2.2.2c │ GPLv2 │ +│ r-gplots │ 3.0.1.1 │ GPLv2 │ +│ r-markdown │ 1.1 │ GPLv2 │ +│ rseqc │ 3.0.1 │ GPLv2 │ +│ bioconductor-dupradar │ 1.14.0 │ GPLv3 │ +│ deeptools │ 3.3.1 │ GPLv3 │ +│ hisat2 │ 2.1.0 │ GPLv3 │ +│ multiqc │ 1.7 │ GPLv3 │ +│ salmon │ 0.14.2 │ GPLv3 │ +│ star │ 2.6.1d │ GPLv3 │ +│ subread │ 1.6.4 │ GPLv3 │ +│ r-base │ 3.6.1 │ GPLv3.0 │ +│ sortmerna │ 2.1b │ LGPL │ +│ gffread │ 0.11.4 │ MIT │ +│ picard │ 2.21.1 │ MIT │ +│ samtools │ 1.9 │ MIT │ +│ r-data.table │ 1.12.4 │ MPL-2.0 │ +│ matplotlib │ 3.0.3 │ PSF-based │ +└───────────────────────────────────┴─────────┴──────────────────────┘ ``` ## Creating a new workflow @@ -367,28 +427,27 @@ $ nf-core create | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 + Workflow Name: nextbigthing Description: This pipeline analyses data from the next big 'omics technique Author: Big Steve - -INFO: Creating new nf-core pipeline: nf-core/nextbigthing - -INFO: Initialising pipeline git repository - -INFO: Done. Remember to add a remote and push to GitHub: - cd /path/to/nf-core-nextbigthing - git remote add origin git@github.com:USERNAME/REPO_NAME.git - git push -``` - -Once you have run the command, create a new empty repository on GitHub under your username (not the `nf-core` organisation, yet). -On your computer, add this repository as a git remote and push to it: - -```console -git remote add origin https://github.com/ewels/nf-core-nextbigthing.git -git push --set-upstream origin master + INFO Creating new nf-core pipeline: nf-core/nextbigthing + INFO Initialising pipeline git repository + INFO Done. Remember to add a remote and push to GitHub: + cd /Users/philewels/GitHub/nf-core/tools/test-create/nf-core-nextbigthing + git remote add origin git@github.com:USERNAME/REPO_NAME.git + git push --all origin + INFO This will also push your newly created dev branch and the TEMPLATE branch for syncing. + INFO !!!!!! IMPORTANT !!!!!! + + If you are interested in adding your pipeline to the nf-core community, + PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE! + + Please read: https://nf-co.re/developers/adding_pipelines#join-the-community ``` +Once you have run the command, create a new empty repository on GitHub under your username (not the `nf-core` organisation, yet) and push the commits from your computer using the example commands in the above log. You can then continue to edit, commit and push normally as you build your pipeline. Please see the [nf-core documentation](https://nf-co.re/developers/adding_pipelines) for a full walkthrough of how to create a new nf-core workflow. @@ -411,20 +470,126 @@ $ nf-core lint . |\ | |__ __ / ` / \ |__) |__ } { | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10.dev0 + + + INFO Testing pipeline: nf-core-testpipeline/ +╭──────────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│ [!] 3 Test Warnings │ +├──────────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ https://nf-co.re/errors#5: GitHub Actions AWS full test should test full datasets: nf-core-testpipeline… │ +│ https://nf-co.re/errors#8: Conda package is not latest available: bioconda::fastqc=0.11.8, 0.11.9 avail… │ +│ https://nf-co.re/errors#8: Conda package is not latest available: bioconda::multiqc=1.7, 1.9 available │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭───────────────────────╮ +│ LINT RESULTS SUMMARY │ +├───────────────────────┤ +│ [✔] 117 Tests Passed │ +│ [!] 3 Test Warnings │ +│ [✗] 0 Test Failed │ +╰───────────────────────╯ +``` + +You can find extensive documentation about each of the lint tests in the [lint errors documentation](https://nf-co.re/errors). + +## Working with pipeline schema + +nf-core pipelines have a `nextflow_schema.json` file in their root which describes the different parameters used by the workflow. +These files allow automated validation of inputs when running the pipeline, are used to generate command line help and can be used to build interfaces to launch pipelines. +Pipeline schema files are built according to the [JSONSchema specification](https://json-schema.org/) (Draft 7). + +To help developers working with pipeline schema, nf-core tools has three `schema` sub-commands: + +* `nf-core schema validate` +* `nf-core schema build` +* `nf-core schema lint` + +### nf-core schema validate + +Nextflow can take input parameters in a JSON or YAML file when running a pipeline using the `-params-file` option. +This command validates such a file against the pipeline schema. + +Usage is `nextflow schema validate --params `, eg: + +```console +$ nf-core schema validate my_pipeline --params my_inputs.json + + ,--./,-. + ___ __ __ __ ___ /,-._.--~\ + |\ | |__ __ / ` / \ |__) |__ } { + | \| | \__, \__/ | \ |___ \`-._,-`-, + `._,._,' + + nf-core/tools version 1.10 + + INFO [✓] Pipeline schema looks valid (found 26 params) + ERROR [✗] Input parameters are invalid: 'input' is a required property +``` + +The `pipeline` option can be a directory containing a pipeline, a path to a schema file or the name of an nf-core pipeline (which will be downloaded using `nextflow pull`). + +### nf-core schema build + +Manually building JSONSchema documents is not trivial and can be very error prone. +Instead, the `nf-core schema build` command collects your pipeline parameters and gives interactive prompts about any missing or unexpected params. +If no existing schema is found it will create one for you. -Running pipeline tests [####################################] 100% None +Once built, the tool can send the schema to the nf-core website so that you can use a graphical interface to organise and fill in the schema. +The tool checks the status of your schema on the website and once complete, saves your changes locally. + +Usage is `nextflow schema build `, eg: + +```console +$ nf-core schema build nf-core-testpipeline -INFO: =========== - LINTING RESULTS -================= - 72 tests passed 2 tests had warnings 0 tests failed + ,--./,-. + ___ __ __ __ ___ /,-._.--~\ + |\ | |__ __ / ` / \ |__) |__ } { + | \| | \__, \__/ | \ |___ \`-._,-`-, + `._,._,' -WARNING: Test Warnings: - http://nf-co.re/errors#8: Conda package is not latest available: picard=2.18.2, 2.18.6 available - http://nf-co.re/errors#8: Conda package is not latest available: bwameth=0.2.0, 0.2.1 available + nf-core/tools version 1.10 + + INFO [✓] Pipeline schema looks valid (found 25 params) schema.py:82 +❓ Unrecognised 'params.old_param' found in schema but not pipeline! Remove it? [y/n]: y +❓ Unrecognised 'params.we_removed_this_too' found in schema but not pipeline! Remove it? [y/n]: y +✨ Found 'params.input' in pipeline but not in schema. Add to pipeline schema? [y/n]: y +✨ Found 'params.outdir' in pipeline but not in schema. Add to pipeline schema? [y/n]: y + INFO Writing schema with 25 params: 'nf-core-testpipeline/nextflow_schema.json' schema.py:121 +🚀 Launch web builder for customisation and editing? [y/n]: y + INFO: Opening URL: https://nf-co.re/pipeline_schema_builder?id=1234567890_abc123def456 + INFO: Waiting for form to be completed in the browser. Remember to click Finished when you're done. + INFO: Found saved status from nf-core JSON Schema builder + INFO: Writing JSON schema with 25 params: nf-core-testpipeline/nextflow_schema.json ``` -You can find extensive documentation about each of the lint tests in the [lint errors documentation](https://nf-co.re/errors). +There are three flags that you can use with this command: + +* `--no-prompts`: Make changes without prompting for confirmation each time. Does not launch web tool. +* `--web-only`: Skips comparison of the schema against the pipeline parameters and only launches the web tool. +* `--url `: Supply a custom URL for the online tool. Useful when testing locally. + +### nf-core schema lint + +The pipeline schema is linted as part of the main `nf-core lint` command, +however sometimes it can be useful to quickly check the syntax of the JSONSchema without running a full lint run. + +Usage is `nextflow schema lint `, eg: + +```console +$ nf-core schema lint nextflow_schema.json + + ,--./,-. + ___ __ __ __ ___ /,-._.--~\ + |\ | |__ __ / ` / \ |__) |__ } { + | \| | \__, \__/ | \ |___ \`-._,-`-, + `._,._,' + + nf-core/tools version 1.10 + + ERROR [✗] Pipeline schema does not follow nf-core specs: + Definition subschema 'input_output_options' not included in schema 'allOf' +``` ## Bumping a pipeline version number @@ -444,44 +609,56 @@ $ nf-core bump-version . 1.0 | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 + +INFO Running nf-core lint tests +INFO Testing pipeline: nf-core-testpipeline/ +INFO Changing version number: + Current version number is '1.4' + New version number will be '1.5' +INFO Updating version in nextflow.config + - version = '1.4' + + version = '1.5' +INFO Updating version in nextflow.config + - process.container = 'nfcore/testpipeline:1.4' + + process.container = 'nfcore/testpipeline:1.5' +INFO Updating version in .github/workflows/ci.yml + - run: docker build --no-cache . -t nfcore/testpipeline:1.4 + + run: docker build --no-cache . -t nfcore/testpipeline:1.5 +INFO Updating version in .github/workflows/ci.yml + - docker tag nfcore/testpipeline:dev nfcore/testpipeline:1.4 + + docker tag nfcore/testpipeline:dev nfcore/testpipeline:1.5 +INFO Updating version in environment.yml + - name: nf-core-testpipeline-1.4 + + name: nf-core-testpipeline-1.5 +INFO Updating version in Dockerfile + - ENV PATH /opt/conda/envs/nf-core-testpipeline-1.4/bin:$PATH + - RUN conda env export --name nf-core-testpipeline-1.4 > nf-core-testpipeline-1.4.yml + + ENV PATH /opt/conda/envs/nf-core-testpipeline-1.5/bin:$PATH + + RUN conda env export --name nf-core-testpipeline-1.5 > nf-core-testpipeline-1.5.yml +``` -INFO: Running nf-core lint tests -Running pipeline tests [####################################] 100% None - -INFO: =========== - LINTING RESULTS -================= - 118 tests passed 0 tests had warnings 0 tests failed - -INFO: Changing version number: - Current version number is '1.0dev' - New version number will be '1.0' - -INFO: Updating version in nextflow.config - - version = '1.0dev' - + version = '1.0' +To change the required version of Nextflow instead of the pipeline version number, use the flag `--nextflow`. -INFO: Updating version in nextflow.config - - process.container = 'nfcore/mypipeline:dev' - + process.container = 'nfcore/mypipeline:1.0' +To export the lint results to a JSON file, use `--json [filename]`. For markdown, use `--markdown [filename]`. -INFO: Updating version in .github/workflows/ci.yml - - docker tag nfcore/mypipeline:dev nfcore/mypipeline:dev - + docker tag nfcore/mypipeline:dev nfcore/mypipeline:1.0 +As linting tests can give a pass state for CI but with warnings that need some effort to track down, the linting +code attempts to post a comment to the GitHub pull-request with a summary of results if possible. +It does this when the environment variables `GITHUB_COMMENTS_URL` and `GITHUB_TOKEN` are set and if there are +any failing or warning tests. If a pull-request is updated with new commits, the original comment will be +updated with the latest results instead of posting lots of new comments for each `git push`. -INFO: Updating version in environment.yml - - name: nf-core-mypipeline-1.0dev - + name: nf-core-mypipeline-1.0 +A typical GitHub Actions step with the required environment variables may look like this (will only work on pull-request events): -INFO: Updating version in Dockerfile - - ENV PATH /opt/conda/envs/nf-core-mypipeline-1.0dev/bin:$PATH - - RUN conda env export --name nf-core-mypipeline-1.0dev > nf-core-mypipeline-1.0dev.yml - + ENV PATH /opt/conda/envs/nf-core-mypipeline-1.0/bin:$PATH - + RUN conda env export --name nf-core-mypipeline-1.0 > nf-core-mypipeline-1.0.yml +```yaml +- name: Run nf-core lint + env: + GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }} + run: nf-core lint $GITHUB_WORKSPACE ``` -To change the required version of Nextflow instead of the pipeline version number, use the flag `--nextflow`. - ## Sync a pipeline with the template Over time, the main nf-core pipeline template is updated. To keep all nf-core pipelines up to date, @@ -502,27 +679,23 @@ $ nf-core sync my_pipeline/ | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 -INFO: Pipeline directory: /path/to/my_pipeline - -INFO: Fetching workflow config variables - -INFO: Deleting all files in TEMPLATE branch - -INFO: Making a new template pipeline using pipeline variables - -INFO: Committed changes to TEMPLATE branch - -INFO: Now try to merge the updates in to your pipeline: - cd /path/to/my_pipeline - git merge TEMPLATE +INFO Pipeline directory: /path/to/my_pipeline +INFO Fetching workflow config variables +INFO Deleting all files in TEMPLATE branch +INFO Making a new template pipeline using pipeline variables +INFO Committed changes to TEMPLATE branch +INFO Now try to merge the updates in to your pipeline: + cd /path/to/my_pipeline + git merge TEMPLATE ``` -If your pipeline repository does not already have a `TEMPLATE` branch, you can instruct -the command to try to create one by giving the `--make-template-branch` flag. -If it has to, the sync tool will then create an orphan branch - see the -[nf-core website sync documentation](https://nf-co.re/developers/sync) for details on -how to handle this. +The sync command tries to check out the `TEMPLATE` branch from the `origin` remote +or an existing local branch called `TEMPLATE`. It will fail if it cannot do either +of these things. The `nf-core create` command should make this template automatically +when you first start your pipeline. Please see the +[nf-core website sync documentation](https://nf-co.re/developers/sync) if you have difficulties. By default, the tool will collect workflow variables from the current branch in your pipeline directory. You can supply the `--from-branch` flag to specific a different branch. @@ -550,12 +723,11 @@ $ nf-core sync --all | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 -INFO: Syncing nf-core/ampliseq - +INFO Syncing nf-core/ampliseq [...] - -INFO: Successfully synchronised [n] pipelines +INFO Successfully synchronised [n] pipelines ``` ## Citation @@ -566,5 +738,5 @@ If you use `nf-core tools` in your work, please cite the `nf-core` publication a > > Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. > -> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). +> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). > ReadCube: [Full Access Link](https://rdcu.be/b1GjZ) diff --git a/codecov.yml b/codecov.yml new file mode 100644 index 0000000000..1ecf8960c0 --- /dev/null +++ b/codecov.yml @@ -0,0 +1,6 @@ +coverage: + status: + project: + default: + threshold: 5% + patch: off diff --git a/docs/api/_src/conf.py b/docs/api/_src/conf.py index 5ad72e335a..e35007f3fb 100644 --- a/docs/api/_src/conf.py +++ b/docs/api/_src/conf.py @@ -4,7 +4,7 @@ # # This file does only contain a selection of the most common options. For a # full list see the documentation: -# http://www.sphinx-doc.org/en/master/config +# https://www.sphinx-doc.org/en/master/config # -- Path setup -------------------------------------------------------------- @@ -14,18 +14,19 @@ # import os import sys -sys.path.insert(0, os.path.abspath('../../../nf_core')) + +sys.path.insert(0, os.path.abspath("../../../nf_core")) # -- Project information ----------------------------------------------------- -project = 'nf-core tools API' -copyright = '2019, Phil Ewels, Sven Fillinger' -author = 'Phil Ewels, Sven Fillinger' +project = "nf-core tools API" +copyright = "2019, Phil Ewels, Sven Fillinger" +author = "Phil Ewels, Sven Fillinger" # The short X.Y version -version = '1.4' +version = "1.4" # The full version, including alpha/beta/rc tags -release = '1.4' +release = "1.4" # -- General configuration --------------------------------------------------- @@ -37,22 +38,19 @@ # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.napoleon' -] +extensions = ["sphinx.ext.autodoc", "sphinx.ext.napoleon"] # Add any paths that contain templates here, relative to this directory. -templates_path = ['./_templates'] +templates_path = ["./_templates"] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] -source_suffix = '.rst' +source_suffix = ".rst" # The master toctree document. -master_doc = 'index' +master_doc = "index" # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. @@ -64,7 +62,7 @@ # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] +exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] # The name of the Pygments (syntax highlighting) style to use. pygments_style = None @@ -75,7 +73,7 @@ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = 'nature' +html_theme = "nature" # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the @@ -86,7 +84,7 @@ # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['./_static'] +html_static_path = ["./_static"] # Custom sidebar templates, must be a dictionary that maps document names # to template names. @@ -102,7 +100,7 @@ # -- Options for HTMLHelp output --------------------------------------------- # Output file base name for HTML help builder. -htmlhelp_basename = 'nf-coredoc' +htmlhelp_basename = "nf-coredoc" # -- Options for LaTeX output ------------------------------------------------ @@ -111,15 +109,12 @@ # The paper size ('letterpaper' or 'a4paper'). # # 'papersize': 'letterpaper', - # The font size ('10pt', '11pt' or '12pt'). # # 'pointsize': '10pt', - # Additional stuff for the LaTeX preamble. # # 'preamble': '', - # Latex figure (float) alignment # # 'figure_align': 'htbp', @@ -129,8 +124,7 @@ # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, 'nf-core.tex', 'nf-core tools API documentation', - 'Phil Ewels, Sven Fillinger', 'manual'), + (master_doc, "nf-core.tex", "nf-core tools API documentation", "Phil Ewels, Sven Fillinger", "manual"), ] @@ -138,10 +132,7 @@ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). -man_pages = [ - (master_doc, 'nf-core', 'nf-core tools API documentation', - [author], 1) -] +man_pages = [(master_doc, "nf-core", "nf-core tools API documentation", [author], 1)] # -- Options for Texinfo output ---------------------------------------------- @@ -150,9 +141,15 @@ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'nf-core', 'nf-core tools API documentation', - author, 'nf-core', 'One line description of project.', - 'Miscellaneous'), + ( + master_doc, + "nf-core", + "nf-core tools API documentation", + author, + "nf-core", + "One line description of project.", + "Miscellaneous", + ), ] @@ -171,7 +168,7 @@ # epub_uid = '' # A list of files that should not be packed into the epub file. -epub_exclude_files = ['search.html'] +epub_exclude_files = ["search.html"] # -- Extension configuration ------------------------------------------------- diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 20bcab9828..d4aa8b8390 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -30,9 +30,15 @@ The following files are suggested but not a hard requirement. If they are missin * `conf/base.config` * A `conf` directory with at least one config called `base.config` -Additionally, the following files must not be present: +The following files will cause a failure if the _are_ present (to fix, delete them): * `Singularity` + * As we are relying on [Docker Hub](https://https://hub.docker.com/) instead of Singularity + and all containers are automatically pulled from there, repositories should not + have a `Singularity` file present. +* `parameters.settings.json` + * The syntax for pipeline schema has changed - old `parameters.settings.json` should be + deleted and new `nextflow_schema.json` files created instead. ## Error #2 - Docker file check failed ## {#2} @@ -86,6 +92,10 @@ The following variables fail the test if missing: * The nextflow timeline, trace, report and DAG should be enabled by default (set to `true`) * `process.cpus`, `process.memory`, `process.time` * Default CPUs, memory and time limits for tasks +* `params.input` + * Input parameter to specify input data, specify this to avoid a warning + * Typical usage: + * `params.input`: Input data that is not NGS sequencing data The following variables throw warnings if missing: @@ -97,18 +107,9 @@ The following variables throw warnings if missing: * The DAG file path should end with `.svg` * If Graphviz is not installed, Nextflow will generate a `.dot` file instead * `process.container` - * Dockerhub handle for a single default container for use by all processes. + * Docker Hub handle for a single default container for use by all processes. * Must specify a tag that matches the pipeline version number if set. * If the pipeline version number contains the string `dev`, the DockerHub tag must be `:dev` -* `params.reads` or `params.input` or `params.design` - * Input parameter to specify input data - one or more of these can be used to avoid a warning - * Typical usage: - * `params.reads`: FastQ files (or pairs) - * `params.input`: Input data that is not NGS sequencing data - * `params.design`: A CSV/TSV design file specifying input files and metadata for the run -* `params.single_end` - * Specify to work with single-end sequence data instead of paired-end by default - * Nextflow implementation: `.fromFilePairs( params.reads, size: params.single_end ? 1 : 2 )` The following variables are depreciated and fail the test if they are still present: @@ -118,8 +119,8 @@ The following variables are depreciated and fail the test if they are still pres * The old method for specifying the minimum Nextflow version. Replaced by `manifest.nextflowVersion` * `params.container` * The old method for specifying the dockerhub container address. Replaced by `process.container` -* `singleEnd` and `igenomesIgnore` - * Changed to `single_end` and `igenomes_ignore` +* `igenomesIgnore` + * Changed to `igenomes_ignore` * The `snake_case` convention should now be used when defining pipeline parameters Process-level configuration syntax is checked and fails if uses the old Nextflow syntax, for example: @@ -129,29 +130,33 @@ Process-level configuration syntax is checked and fails if uses the old Nextflow nf-core pipelines must have CI testing with GitHub Actions. -### GitHub Actions +### GitHub Actions CI -There are 3 main GitHub Actions CI test files: `ci.yml`, `linting.yml` and `branch.yml` and they can all be found in the `.github/workflows/` directory. +There are 4 main GitHub Actions CI test files: `ci.yml`, `linting.yml`, `branch.yml` and `awstests.yml`, and they can all be found in the `.github/workflows/` directory. +You can always add steps to the workflows to suit your needs, but to ensure that the `nf-core lint` tests pass, keep the steps indicated here. This test will fail if the following requirements are not met in these files: 1. `ci.yml`: Contains all the commands required to test the pipeline - * Must be turned on for `push` and `pull_request`: + * Must be triggered on the following events: ```yaml - on: [push, pull_request] + on: + push: + branches: + - dev + pull_request: + release: + types: [published] ``` - * The minimum Nextflow version specified in the pipeline's `nextflow.config` has to match that defined by `nxf_ver` in this file: + * The minimum Nextflow version specified in the pipeline's `nextflow.config` has to match that defined by `nxf_ver` in the test matrix: ```yaml - jobs: - test: - runs-on: ubuntu-18.04 - strategy: - matrix: - # Nextflow versions: check pipeline minimum and current latest - nxf_ver: ['19.10.0', ''] + strategy: + matrix: + # Nextflow versions: check pipeline minimum and current latest + nxf_ver: ['19.10.0', ''] ``` * The `Docker` container for the pipeline must be tagged appropriately for: @@ -159,14 +164,15 @@ This test will fail if the following requirements are not met in these files: * Released pipelines: `docker tag nfcore/:dev nfcore/:` ```yaml - jobs: - test: - runs-on: ubuntu-18.04 - steps: - - name: Pull image - run: | - docker pull nfcore/:dev - docker tag nfcore/:dev nfcore/:1.0.0 + - name: Build new docker image + if: env.GIT_DIFF + run: docker build --no-cache . -t nfcore/:1.0.0 + + - name: Pull docker image + if: ${{ !env.GIT_DIFF }} + run: | + docker pull nfcore/:dev + docker tag nfcore/:dev nfcore/:1.0.0 ``` 2. `linting.yml`: Specifies the commands to lint the pipeline repository using `nf-core lint` and `markdownlint` @@ -174,7 +180,7 @@ This test will fail if the following requirements are not met in these files: * Must have the command `nf-core lint ${GITHUB_WORKSPACE}`. * Must have the command `markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml`. -3. `branch.yml`: Ensures that pull requests to the protected `master` branch are coming from the correct branch +3. `branch.yml`: Ensures that pull requests to the protected `master` branch are coming from the correct branch when a PR is opened against the _nf-core_ repository. * Must be turned on for `pull_request` to `master`. ```yaml @@ -184,16 +190,49 @@ This test will fail if the following requirements are not met in these files: - master ``` - * Checks that PRs to the protected `master` branch can only come from an nf-core `dev` branch or a fork `patch` branch: + * Checks that PRs to the protected nf-core repo `master` branch can only come from an nf-core `dev` branch or a fork `patch` branch: + + ```yaml + steps: + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches + - name: Check PRs + if: github.repository == 'nf-core/' + run: | + { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + ``` + + * For branch protection in repositories outside of _nf-core_, you can add an additional step to this workflow. Keep the _nf-core_ branch protection step, to ensure that the `nf-core lint` tests pass. Here's an example: ```yaml steps: # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch - name: Check PRs + if: github.repository == 'nf-core/' + run: | + { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + - name: Check PRs in another repository + if: github.repository == '/' run: | - { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name}} == / ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] ``` +4. `awstest.yml`: Triggers tests on AWS batch. As running tests on AWS incurs costs, they should be only triggered on `push` to `master` and `release`. + * Must be turned on for `push` to `master` and `release`. + * Must not be turned on for `pull_request` or other events. + +### GitHub Actions AWS full tests + +Additionally, we provide the possibility of testing the pipeline on full size datasets on AWS. +This should ensure that the pipeline runs as expected on AWS and provide a resource estimation. +The GitHub Actions workflow is: `awsfulltest.yml`, and it can be found in the `.github/workflows/` directory. +This workflow incurrs higher AWS costs, therefore it should only be triggered on `release`. +For tests on full data prior to release, [https://tower.nf](Nextflow Tower's launch feature) can be employed. + +`awsfulltest.yml`: Triggers full sized tests run on AWS batch after releasing. + +* Must be only turned on for `release`. +* Should run the profile `test_full`. If it runs the profile `test` a warning is given. + ## Error #6 - Repository `README.md` tests ## {#6} The `README.md` files for a project are very important and must meet some requirements: @@ -212,7 +251,7 @@ The `README.md` files for a project are very important and must meet some requir * Required badge code: ```markdown - [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/) + [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) ``` ## Error #7 - Pipeline and container version numbers ## {#7} @@ -265,7 +304,7 @@ LABEL authors="your@email.com" \ description="Docker image containing all requirements for the nf-core mypipeline pipeline" COPY environment.yml / -RUN conda env create -f /environment.yml && conda clean -a +RUN conda env create --quiet -f /environment.yml && conda clean -a RUN conda env export --name nf-core-mypipeline-1.0 > nf-core-mypipeline-1.0.yml ENV PATH /opt/conda/envs/nf-core-mypipeline-1.0/bin:$PATH ``` @@ -290,10 +329,64 @@ The nf-core workflow template contains a number of comment lines with the follow This lint test runs through all files in the pipeline and searches for these lines. -## Error #11 - Singularity file found ##{#11} +## Error #11 - Pipeline name ## {#11} -As we are relying on [Docker Hub](https://hub.docker.com/) instead of Singularity and all containers are automatically pulled from there, repositories should not have a `Singularity` file present. +_..removed.._ ## Error #12 - Pipeline name ## {#12} -In order to ensure consistent naming, pipeline names should contain only lower case, alphabetical characters. Otherwise a warning is displayed. +In order to ensure consistent naming, pipeline names should contain only lower case, alphanumeric characters. Otherwise a warning is displayed. + +## Error #13 - Pipeline name ## {#13} + +The `nf-core create` pipeline template uses [cookiecutter](https://github.com/cookiecutter/cookiecutter) behind the scenes. +This check fails if any cookiecutter template variables such as `{{ cookiecutter.pipeline_name }}` are fouund in your pipeline code. +Finding a placeholder like this means that something was probably copied and pasted from the template without being properly rendered for your pipeline. + +## Error #14 - Pipeline schema syntax ## {#14} + +Pipelines should have a `nextflow_schema.json` file that describes the different pipeline parameters (eg. `params.something`, `--something`). + +* Schema should be valid JSON files +* Schema should adhere to [JSONSchema](https://json-schema.org/), Draft 7. +* Parameters can be described in two places: + * As `properties` in the top-level schema object + * As `properties` within subschemas listed in a top-level `definitions` objects +* The schema must describe at least one parameter +* There must be no duplicate parameter IDs across the schema and definition subschema +* All subschema in `definitions` must be referenced in the top-level `allOf` key +* The top-level `allOf` key must not describe any non-existent definitions +* Core top-level schema attributes should exist and be set as follows: + * `$schema`: `https://json-schema.org/draft-07/schema` + * `$id`: URL to the raw schema file, eg. `https://raw.githubusercontent.com/YOURPIPELINE/master/nextflow_schema.json` + * `title`: `YOURPIPELINE pipeline parameters` + * `description`: The piepline config `manifest.description` + +For example, an _extremely_ minimal schema could look like this: + +```json +{ + "$schema": "https://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/YOURPIPELINE/master/nextflow_schema.json", + "title": "YOURPIPELINE pipeline parameters", + "description": "This pipeline is for testing", + "properties": { + "first_param": { "type": "string" } + }, + "definitions": { + "my_first_group": { + "properties": { + "second_param": { "type": "string" } + } + } + }, + "allOf": [{"$ref": "#/definitions/my_first_group"}] +} +``` + +## Error #15 - Schema config check ## {#15} + +The `nextflow_schema.json` pipeline schema should describe every flat parameter returned from the `nextflow config` command (params that are objects or more complex structures are ignored). +Missing parameters result in a lint failure. + +If any parameters are found in the schema that were not returned from `nextflow config` a warning is given. diff --git a/nf_core/__main__.py b/nf_core/__main__.py new file mode 100755 index 0000000000..ecbccbf6ee --- /dev/null +++ b/nf_core/__main__.py @@ -0,0 +1,578 @@ +#!/usr/bin/env python +""" nf-core: Helper tools for use with nf-core Nextflow pipelines. """ + +from rich import print +import click +import logging +import os +import re +import rich.console +import rich.logging +import rich.traceback +import sys + +import nf_core +import nf_core.bump_version +import nf_core.create +import nf_core.download +import nf_core.launch +import nf_core.licences +import nf_core.lint +import nf_core.list +import nf_core.modules +import nf_core.schema +import nf_core.sync +import nf_core.utils + +log = logging.getLogger(__name__) + + +def run_nf_core(): + # Set up the rich traceback + rich.traceback.install(width=200, word_wrap=True) + + # Print nf-core header to STDERR + stderr = rich.console.Console(file=sys.stderr) + stderr.print("\n[green]{},--.[grey39]/[green],-.".format(" " * 42)) + stderr.print("[blue] ___ __ __ __ ___ [green]/,-._.--~\\") + stderr.print("[blue] |\ | |__ __ / ` / \ |__) |__ [yellow] } {") + stderr.print("[blue] | \| | \__, \__/ | \ |___ [green]\`-._,-`-,") + stderr.print("[green] `._,._,'\n") + stderr.print("[grey39] nf-core/tools version {}".format(nf_core.__version__), highlight=False) + try: + is_outdated, current_vers, remote_vers = nf_core.utils.check_if_outdated() + if is_outdated: + stderr.print( + "[bold bright_yellow] There is a new version of nf-core/tools available! ({})".format(remote_vers), + highlight=False, + ) + except Exception as e: + log.debug("Could not check latest version: {}".format(e)) + stderr.print("\n\n") + + # Lanch the click cli + nf_core_cli() + + +# Customise the order of subcommands for --help +# https://stackoverflow.com/a/47984810/713980 +class CustomHelpOrder(click.Group): + def __init__(self, *args, **kwargs): + self.help_priorities = {} + super(CustomHelpOrder, self).__init__(*args, **kwargs) + + def get_help(self, ctx): + self.list_commands = self.list_commands_for_help + return super(CustomHelpOrder, self).get_help(ctx) + + def list_commands_for_help(self, ctx): + """reorder the list of commands when listing the help""" + commands = super(CustomHelpOrder, self).list_commands(ctx) + return (c[1] for c in sorted((self.help_priorities.get(command, 1000), command) for command in commands)) + + def command(self, *args, **kwargs): + """Behaves the same as `click.Group.command()` except capture + a priority for listing command names in help. + """ + help_priority = kwargs.pop("help_priority", 1000) + help_priorities = self.help_priorities + + def decorator(f): + cmd = super(CustomHelpOrder, self).command(*args, **kwargs)(f) + help_priorities[cmd.name] = help_priority + return cmd + + return decorator + + def group(self, *args, **kwargs): + """Behaves the same as `click.Group.group()` except capture + a priority for listing command names in help. + """ + help_priority = kwargs.pop("help_priority", 1000) + help_priorities = self.help_priorities + + def decorator(f): + cmd = super(CustomHelpOrder, self).command(*args, **kwargs)(f) + help_priorities[cmd.name] = help_priority + return cmd + + return decorator + + +@click.group(cls=CustomHelpOrder) +@click.version_option(nf_core.__version__) +@click.option("-v", "--verbose", is_flag=True, default=False, help="Verbose output (print debug statements).") +def nf_core_cli(verbose): + stderr = rich.console.Console(file=sys.stderr) + logging.basicConfig( + level=logging.DEBUG if verbose else logging.INFO, + format="%(message)s", + datefmt=" ", + handlers=[rich.logging.RichHandler(console=stderr, markup=True)], + ) + + +# nf-core list +@nf_core_cli.command(help_priority=1) +@click.argument("keywords", required=False, nargs=-1, metavar="") +@click.option( + "-s", + "--sort", + type=click.Choice(["release", "pulled", "name", "stars"]), + default="release", + help="How to sort listed pipelines", +) +@click.option("--json", is_flag=True, default=False, help="Print full output as JSON") +@click.option("--show-archived", is_flag=True, default=False, help="Print archived workflows") +def list(keywords, sort, json, show_archived): + """ + List available nf-core pipelines with local info. + + Checks the web for a list of nf-core pipelines with their latest releases. + Shows which nf-core pipelines you have pulled locally and whether they are up to date. + """ + print(nf_core.list.list_workflows(keywords, sort, json, show_archived)) + + +# nf-core launch +@nf_core_cli.command(help_priority=2) +@click.argument("pipeline", required=False, metavar="") +@click.option("-r", "--revision", help="Release/branch/SHA of the project to run (if remote)") +@click.option("-i", "--id", help="ID for web-gui launch parameter set") +@click.option( + "-c", "--command-only", is_flag=True, default=False, help="Create Nextflow command with params (no params file)" +) +@click.option( + "-o", + "--params-out", + type=click.Path(), + default=os.path.join(os.getcwd(), "nf-params.json"), + help="Path to save run parameters file", +) +@click.option( + "-p", "--params-in", type=click.Path(exists=True), help="Set of input run params to use from a previous run" +) +@click.option( + "-a", "--save-all", is_flag=True, default=False, help="Save all parameters, even if unchanged from default" +) +@click.option( + "-h", "--show-hidden", is_flag=True, default=False, help="Show hidden params which don't normally need changing" +) +@click.option( + "--url", type=str, default="https://nf-co.re/launch", help="Customise the builder URL (for development work)" +) +def launch(pipeline, id, revision, command_only, params_in, params_out, save_all, show_hidden, url): + """ + Launch a pipeline using a web GUI or command line prompts. + + Uses the pipeline schema file to collect inputs for all available pipeline + parameters. Parameter names, descriptions and help text are shown. + The pipeline schema is used to validate all inputs as they are entered. + + When finished, saves a file with the selected parameters which can be + passed to Nextflow using the -params-file option. + + Run using a remote pipeline name (such as GitHub `user/repo` or a URL), + a local pipeline directory or an ID from the nf-core web launch tool. + """ + launcher = nf_core.launch.Launch( + pipeline, revision, command_only, params_in, params_out, save_all, show_hidden, url, id + ) + if launcher.launch_pipeline() == False: + sys.exit(1) + + +# nf-core download +@nf_core_cli.command(help_priority=3) +@click.argument("pipeline", required=True, metavar="") +@click.option("-r", "--release", type=str, help="Pipeline release") +@click.option("-s", "--singularity", is_flag=True, default=False, help="Download singularity containers") +@click.option("-o", "--outdir", type=str, help="Output directory") +@click.option( + "-c", + "--compress", + type=click.Choice(["tar.gz", "tar.bz2", "zip", "none"]), + default="tar.gz", + help="Compression type", +) +def download(pipeline, release, singularity, outdir, compress): + """ + Download a pipeline, configs and singularity container. + + Collects all workflow files and shared configs from nf-core/configs. + Configures the downloaded workflow to use the relative path to the configs. + """ + dl = nf_core.download.DownloadWorkflow(pipeline, release, singularity, outdir, compress) + dl.download_workflow() + + +# nf-core licences +@nf_core_cli.command(help_priority=4) +@click.argument("pipeline", required=True, metavar="") +@click.option("--json", is_flag=True, default=False, help="Print output in JSON") +def licences(pipeline, json): + """ + List software licences for a given workflow. + + Checks the pipeline environment.yml file which lists all conda software packages. + Each of these is queried against the anaconda.org API to find the licence. + Package name, version and licence is printed to the command line. + """ + lic = nf_core.licences.WorkflowLicences(pipeline) + lic.as_json = json + try: + print(lic.run_licences()) + except LookupError as e: + log.error(e) + sys.exit(1) + + +# nf-core create +def validate_wf_name_prompt(ctx, opts, value): + """ Force the workflow name to meet the nf-core requirements """ + if not re.match(r"^[a-z]+$", value): + click.echo("Invalid workflow name: must be lowercase without punctuation.") + value = click.prompt(opts.prompt) + return validate_wf_name_prompt(ctx, opts, value) + return value + + +@nf_core_cli.command(help_priority=5) +@click.option( + "-n", + "--name", + prompt="Workflow Name", + required=True, + callback=validate_wf_name_prompt, + type=str, + help="The name of your new pipeline", +) +@click.option("-d", "--description", prompt=True, required=True, type=str, help="A short description of your pipeline") +@click.option("-a", "--author", prompt=True, required=True, type=str, help="Name of the main author(s)") +@click.option("--new-version", type=str, default="1.0dev", help="The initial version number to use") +@click.option("--no-git", is_flag=True, default=False, help="Do not initialise pipeline as new git repository") +@click.option("-f", "--force", is_flag=True, default=False, help="Overwrite output directory if it already exists") +@click.option("-o", "--outdir", type=str, help="Output directory for new pipeline (default: pipeline name)") +def create(name, description, author, new_version, no_git, force, outdir): + """ + Create a new pipeline using the nf-core template. + + Uses the nf-core template to make a skeleton Nextflow pipeline with all required + files, boilerplate code and best-practices. + """ + create_obj = nf_core.create.PipelineCreate(name, description, author, new_version, no_git, force, outdir) + create_obj.init_pipeline() + + +@nf_core_cli.command(help_priority=6) +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") +@click.option( + "--release", + is_flag=True, + default=os.path.basename(os.path.dirname(os.environ.get("GITHUB_REF", "").strip(" '\""))) == "master" + and os.environ.get("GITHUB_REPOSITORY", "").startswith("nf-core/") + and not os.environ.get("GITHUB_REPOSITORY", "") == "nf-core/tools", + help="Execute additional checks for release-ready workflows.", +) +@click.option("--markdown", type=str, metavar="", help="File to write linting results to (Markdown)") +@click.option("--json", type=str, metavar="", help="File to write linting results to (JSON)") +def lint(pipeline_dir, release, markdown, json): + """ + Check pipeline code against nf-core guidelines. + + Runs a large number of automated tests to ensure that the supplied pipeline + meets the nf-core guidelines. Documentation of all lint tests can be found + on the nf-core website: https://nf-co.re/errors + """ + + # Run the lint tests! + lint_obj = nf_core.lint.run_linting(pipeline_dir, release, markdown, json) + if len(lint_obj.failed) > 0: + sys.exit(1) + + +## nf-core module subcommands +@nf_core_cli.group(cls=CustomHelpOrder, help_priority=7) +@click.option( + "-r", + "--repository", + type=str, + default="nf-core/modules", + help="GitHub repository hosting software wrapper modules.", +) +@click.option("-b", "--branch", type=str, default="master", help="Modules GitHub repo git branch to use.") +@click.pass_context +def modules(ctx, repository, branch): + """ + Work with the nf-core/modules software wrappers. + + Tools to manage DSL 2 nf-core/modules software wrapper imports. + """ + # ensure that ctx.obj exists and is a dict (in case `cli()` is called + # by means other than the `if` block below) + ctx.ensure_object(dict) + + # Make repository object to pass to subcommands + ctx.obj["modules_repo_obj"] = nf_core.modules.ModulesRepo(repository, branch) + + +@modules.command(help_priority=1) +@click.pass_context +def list(ctx): + """ + List available software modules. + + Lists all currently available software wrappers in the nf-core/modules repository. + """ + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] + print(mods.list_modules()) + + +@modules.command(help_priority=2) +@click.pass_context +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") +@click.argument("tool", type=str, required=True, metavar="") +def install(ctx, pipeline_dir, tool): + """ + Add a DSL2 software wrapper module to a pipeline. + + Given a software name, finds the relevant files in nf-core/modules + and copies to the pipeline along with associated metadata. + """ + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] + mods.pipeline_dir = pipeline_dir + mods.install(tool) + + +@modules.command(help_priority=3) +@click.pass_context +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") +@click.argument("tool", type=str, metavar="") +@click.option("-f", "--force", is_flag=True, default=False, help="Force overwrite of files") +def update(ctx, tool, pipeline_dir, force): + """ + Update one or all software wrapper modules. + + Compares a currently installed module against what is available in nf-core/modules. + Fetchs files and updates all relevant files for that software wrapper. + + If no module name is specified, loops through all currently installed modules. + If no version is specified, looks for the latest available version on nf-core/modules. + """ + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] + mods.pipeline_dir = pipeline_dir + mods.update(tool, force=force) + + +@modules.command(help_priority=4) +@click.pass_context +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") +@click.argument("tool", type=str, required=True, metavar="") +def remove(ctx, pipeline_dir, tool): + """ + Remove a software wrapper from a pipeline. + """ + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] + mods.pipeline_dir = pipeline_dir + mods.remove(tool) + + +@modules.command(help_priority=5) +@click.pass_context +def check(ctx): + """ + Check that imported module code has not been modified. + + Compares a software module against the copy on nf-core/modules. + If any local modifications are found, the command logs an error + and exits with a non-zero exit code. + + Use by the lint tests and automated CI to check that centralised + software wrapper code is only modified in the central repository. + """ + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] + mods.check_modules() + + +## nf-core schema subcommands +@nf_core_cli.group(cls=CustomHelpOrder, help_priority=8) +def schema(): + """ + Suite of tools for developers to manage pipeline schema. + + All nf-core pipelines should have a nextflow_schema.json file in their + root directory that describes the different pipeline parameters. + """ + pass + + +@schema.command(help_priority=1) +@click.argument("pipeline", required=True, metavar="") +@click.argument("params", type=click.Path(exists=True), required=True, metavar="") +def validate(pipeline, params): + """ + Validate a set of parameters against a pipeline schema. + + Nextflow can be run using the -params-file flag, which loads + script parameters from a JSON file. + + This command takes such a file and validates it against the pipeline + schema, checking whether all schema rules are satisfied. + """ + schema_obj = nf_core.schema.PipelineSchema() + try: + schema_obj.get_schema_path(pipeline) + # Load and check schema + schema_obj.load_lint_schema() + except AssertionError as e: + log.error(e) + sys.exit(1) + schema_obj.load_input_params(params) + try: + schema_obj.validate_params() + except AssertionError as e: + sys.exit(1) + + +@schema.command(help_priority=2) +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") +@click.option("--no-prompts", is_flag=True, help="Do not confirm changes, just update parameters and exit") +@click.option("--web-only", is_flag=True, help="Skip building using Nextflow config, just launch the web tool") +@click.option( + "--url", + type=str, + default="https://nf-co.re/pipeline_schema_builder", + help="Customise the builder URL (for development work)", +) +def build(pipeline_dir, no_prompts, web_only, url): + """ + Interactively build a pipeline schema from Nextflow params. + + Automatically detects parameters from the pipeline config and main.nf and + compares these to the pipeline schema. Prompts to add or remove parameters + if the two do not match one another. + + Once all parameters are accounted for, can launch a web GUI tool on the + https://nf-co.re website where you can annotate and organise parameters. + Listens for this to be completed and saves the updated schema. + """ + schema_obj = nf_core.schema.PipelineSchema() + if schema_obj.build_schema(pipeline_dir, no_prompts, web_only, url) is False: + sys.exit(1) + + +@schema.command(help_priority=3) +@click.argument("schema_path", type=click.Path(exists=True), required=True, metavar="") +def lint(schema_path): + """ + Check that a given pipeline schema is valid. + + Checks whether the pipeline schema validates as JSON Schema Draft 7 + and adheres to the additional nf-core schema requirements. + + This function runs as part of the nf-core lint command, this is a convenience + command that does just the schema linting nice and quickly. + """ + schema_obj = nf_core.schema.PipelineSchema() + try: + schema_obj.get_schema_path(schema_path) + schema_obj.load_lint_schema() + # Validate title and description - just warnings as schema should still work fine + try: + schema_obj.validate_schema_title_description() + except AssertionError as e: + log.warn(e) + except AssertionError as e: + sys.exit(1) + + +@nf_core_cli.command("bump-version", help_priority=9) +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") +@click.argument("new_version", required=True, metavar="") +@click.option( + "-n", "--nextflow", is_flag=True, default=False, help="Bump required nextflow version instead of pipeline version" +) +def bump_version(pipeline_dir, new_version, nextflow): + """ + Update nf-core pipeline version number. + + The pipeline version number is mentioned in a lot of different places + in nf-core pipelines. This tool updates the version for you automatically, + so that you don't accidentally miss any. + + Should be used for each pipeline release, and again for the next + development version after release. + + As well as the pipeline version, you can also change the required version of Nextflow. + """ + + # First, lint the pipeline to check everything is in order + log.info("Running nf-core lint tests") + + # Run the lint tests + try: + lint_obj = nf_core.lint.PipelineLint(pipeline_dir) + lint_obj.lint_pipeline() + except AssertionError as e: + log.error("Please fix lint errors before bumping versions") + return + if len(lint_obj.failed) > 0: + log.error("Please fix lint errors before bumping versions") + return + + # Bump the pipeline version number + if not nextflow: + nf_core.bump_version.bump_pipeline_version(lint_obj, new_version) + else: + nf_core.bump_version.bump_nextflow_version(lint_obj, new_version) + + +@nf_core_cli.command("sync", help_priority=10) +@click.argument("pipeline_dir", type=click.Path(exists=True), nargs=-1, metavar="") +@click.option("-b", "--from-branch", type=str, help="The git branch to use to fetch workflow vars.") +@click.option("-p", "--pull-request", is_flag=True, default=False, help="Make a GitHub pull-request with the changes.") +@click.option("-u", "--username", type=str, help="GitHub username for the PR.") +@click.option("-r", "--repository", type=str, help="GitHub repository name for the PR.") +@click.option("-a", "--auth-token", type=str, help="GitHub API personal access token.") +@click.option("--all", is_flag=True, default=False, help="Sync template for all nf-core pipelines.") +def sync(pipeline_dir, from_branch, pull_request, username, repository, auth_token, all): + """ + Sync a pipeline TEMPLATE branch with the nf-core template. + + To keep nf-core pipelines up to date with improvements in the main + template, we use a method of synchronisation that uses a special + git branch called TEMPLATE. + + This command updates the TEMPLATE branch with the latest version of + the nf-core template, so that these updates can be synchronised with + the pipeline. It is run automatically for all pipelines when ever a + new release of nf-core/tools (and the included template) is made. + """ + + # Pull and sync all nf-core pipelines + if all: + nf_core.sync.sync_all_pipelines(username, auth_token) + else: + # Manually check for the required parameter + if not pipeline_dir or len(pipeline_dir) != 1: + log.error("Either use --all or specify one ") + sys.exit(1) + else: + pipeline_dir = pipeline_dir[0] + + # Sync the given pipeline dir + sync_obj = nf_core.sync.PipelineSync(pipeline_dir, from_branch, pull_request) + try: + sync_obj.sync() + except (nf_core.sync.SyncException, nf_core.sync.PullRequestException) as e: + log.error(e) + sys.exit(1) + + +if __name__ == "__main__": + run_nf_core() diff --git a/nf_core/bump_version.py b/nf_core/bump_version.py index fccd75a7ff..60434f9f55 100644 --- a/nf_core/bump_version.py +++ b/nf_core/bump_version.py @@ -3,12 +3,13 @@ a nf-core pipeline. """ +import click import logging import os import re import sys -import click +log = logging.getLogger(__name__) def bump_pipeline_version(lint_obj, new_version): @@ -20,43 +21,68 @@ def bump_pipeline_version(lint_obj, new_version): new_version (str): The new version tag for the pipeline. Semantic versioning only. """ # Collect the old and new version numbers - current_version = lint_obj.config.get('manifest.version', '').strip(' \'"') - if new_version.startswith('v'): - logging.warning("Stripping leading 'v' from new version number") + current_version = lint_obj.config.get("manifest.version", "").strip(" '\"") + if new_version.startswith("v"): + log.warning("Stripping leading 'v' from new version number") new_version = new_version[1:] if not current_version: - logging.error("Could not find config variable manifest.version") + log.error("Could not find config variable manifest.version") sys.exit(1) - logging.info("Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format(current_version, new_version)) + log.info( + "Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format( + current_version, new_version + ) + ) # Update nextflow.config - nfconfig_pattern = r"version\s*=\s*[\'\"]?{}[\'\"]?".format(current_version.replace('.',r'\.')) + nfconfig_pattern = r"version\s*=\s*[\'\"]?{}[\'\"]?".format(current_version.replace(".", r"\.")) nfconfig_newstr = "version = '{}'".format(new_version) update_file_version("nextflow.config", lint_obj, nfconfig_pattern, nfconfig_newstr) # Update container tag - docker_tag = 'dev' - if new_version.replace('.', '').isdigit(): + docker_tag = "dev" + if new_version.replace(".", "").isdigit(): docker_tag = new_version else: - logging.info("New version contains letters. Setting docker tag to 'dev'") - nfconfig_pattern = r"container\s*=\s*[\'\"]nfcore/{}:(?:{}|dev)[\'\"]".format(lint_obj.pipeline_name.lower(), current_version.replace('.',r'\.')) + log.info("New version contains letters. Setting docker tag to 'dev'") + nfconfig_pattern = r"container\s*=\s*[\'\"]nfcore/{}:(?:{}|dev)[\'\"]".format( + lint_obj.pipeline_name.lower(), current_version.replace(".", r"\.") + ) nfconfig_newstr = "container = 'nfcore/{}:{}'".format(lint_obj.pipeline_name.lower(), docker_tag) update_file_version("nextflow.config", lint_obj, nfconfig_pattern, nfconfig_newstr) - # Update GitHub Actions CI image tag - nfconfig_pattern = r"docker tag nfcore/{name}:dev nfcore/{name}:(?:{tag}|dev)".format(name=lint_obj.pipeline_name.lower(), tag=current_version.replace('.',r'\.')) - nfconfig_newstr = "docker tag nfcore/{name}:dev nfcore/{name}:{tag}".format(name=lint_obj.pipeline_name.lower(), tag=docker_tag) - update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr) + # Update GitHub Actions CI image tag (build) + nfconfig_pattern = r"docker build --no-cache . -t nfcore/{name}:(?:{tag}|dev)".format( + name=lint_obj.pipeline_name.lower(), tag=current_version.replace(".", r"\.") + ) + nfconfig_newstr = "docker build --no-cache . -t nfcore/{name}:{tag}".format( + name=lint_obj.pipeline_name.lower(), tag=docker_tag + ) + update_file_version( + os.path.join(".github", "workflows", "ci.yml"), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True + ) - if 'environment.yml' in lint_obj.files: + # Update GitHub Actions CI image tag (pull) + nfconfig_pattern = r"docker tag nfcore/{name}:dev nfcore/{name}:(?:{tag}|dev)".format( + name=lint_obj.pipeline_name.lower(), tag=current_version.replace(".", r"\.") + ) + nfconfig_newstr = "docker tag nfcore/{name}:dev nfcore/{name}:{tag}".format( + name=lint_obj.pipeline_name.lower(), tag=docker_tag + ) + update_file_version( + os.path.join(".github", "workflows", "ci.yml"), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True + ) + + if "environment.yml" in lint_obj.files: # Update conda environment.yml - nfconfig_pattern = r"name: nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), current_version.replace('.',r'\.')) + nfconfig_pattern = r"name: nf-core-{}-{}".format( + lint_obj.pipeline_name.lower(), current_version.replace(".", r"\.") + ) nfconfig_newstr = "name: nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), new_version) update_file_version("environment.yml", lint_obj, nfconfig_pattern, nfconfig_newstr) # Update Dockerfile ENV PATH and RUN conda env create - nfconfig_pattern = r"nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), current_version.replace('.',r'\.')) + nfconfig_pattern = r"nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), current_version.replace(".", r"\.")) nfconfig_newstr = "nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), new_version) update_file_version("Dockerfile", lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True) @@ -70,26 +96,32 @@ def bump_nextflow_version(lint_obj, new_version): new_version (str): The new version tag for the required Nextflow version. """ # Collect the old and new version numbers - current_version = lint_obj.config.get('manifest.nextflowVersion', '').strip(' \'"') - current_version = re.sub(r'[^0-9\.]', '', current_version) - new_version = re.sub(r'[^0-9\.]', '', new_version) + current_version = lint_obj.config.get("manifest.nextflowVersion", "").strip(" '\"") + current_version = re.sub(r"[^0-9\.]", "", current_version) + new_version = re.sub(r"[^0-9\.]", "", new_version) if not current_version: - logging.error("Could not find config variable manifest.nextflowVersion") + log.error("Could not find config variable manifest.nextflowVersion") sys.exit(1) - logging.info("Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format(current_version, new_version)) + log.info( + "Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format( + current_version, new_version + ) + ) # Update nextflow.config - nfconfig_pattern = r"nextflowVersion\s*=\s*[\'\"]?>={}[\'\"]?".format(current_version.replace('.',r'\.')) + nfconfig_pattern = r"nextflowVersion\s*=\s*[\'\"]?>={}[\'\"]?".format(current_version.replace(".", r"\.")) nfconfig_newstr = "nextflowVersion = '>={}'".format(new_version) update_file_version("nextflow.config", lint_obj, nfconfig_pattern, nfconfig_newstr) # Update GitHub Actions CI - nfconfig_pattern = r"nxf_ver: \[[\'\"]?{}[\'\"]?, ''\]".format(current_version.replace('.',r'\.')) + nfconfig_pattern = r"nxf_ver: \[[\'\"]?{}[\'\"]?, ''\]".format(current_version.replace(".", r"\.")) nfconfig_newstr = "nxf_ver: ['{}', '']".format(new_version) - update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr, True) + update_file_version( + os.path.join(".github", "workflows", "ci.yml"), lint_obj, nfconfig_pattern, nfconfig_newstr, True + ) # Update README badge - nfconfig_pattern = r"nextflow-%E2%89%A5{}-brightgreen.svg".format(current_version.replace('.',r'\.')) + nfconfig_pattern = r"nextflow-%E2%89%A5{}-brightgreen.svg".format(current_version.replace(".", r"\.")) nfconfig_newstr = "nextflow-%E2%89%A5{}-brightgreen.svg".format(new_version) update_file_version("README.md", lint_obj, nfconfig_pattern, nfconfig_newstr, True) @@ -110,12 +142,12 @@ def update_file_version(filename, lint_obj, pattern, newstr, allow_multiple=Fals """ # Load the file fn = os.path.join(lint_obj.path, filename) - content = '' - with open(fn, 'r') as fh: + content = "" + with open(fn, "r") as fh: content = fh.read() # Check that we have exactly one match - matches_pattern = re.findall("^.*{}.*$".format(pattern),content,re.MULTILINE) + matches_pattern = re.findall("^.*{}.*$".format(pattern), content, re.MULTILINE) if len(matches_pattern) == 0: raise SyntaxError("Could not find version number in {}: '{}'".format(filename, pattern)) if len(matches_pattern) > 1 and not allow_multiple: @@ -123,12 +155,13 @@ def update_file_version(filename, lint_obj, pattern, newstr, allow_multiple=Fals # Replace the match new_content = re.sub(pattern, newstr, content) - matches_newstr = re.findall("^.*{}.*$".format(newstr),new_content,re.MULTILINE) + matches_newstr = re.findall("^.*{}.*$".format(newstr), new_content, re.MULTILINE) - logging.info("Updating version in {}\n".format(filename) + - click.style(" - {}\n".format("\n - ".join(matches_pattern).strip()), fg='red') + - click.style(" + {}\n".format("\n + ".join(matches_newstr).strip()), fg='green') + log.info( + "Updating version in {}\n".format(filename) + + "[red] - {}\n".format("\n - ".join(matches_pattern).strip()) + + "[green] + {}\n".format("\n + ".join(matches_newstr).strip()) ) - with open(fn, 'w') as fh: + with open(fn, "w") as fh: fh.write(new_content) diff --git a/nf_core/create.py b/nf_core/create.py index e44a7105f3..2fdeeecff2 100644 --- a/nf_core/create.py +++ b/nf_core/create.py @@ -2,6 +2,7 @@ """Creates a nf-core pipeline matching the current organization's specification based on a template. """ +import click import cookiecutter.main, cookiecutter.exceptions import git import logging @@ -10,9 +11,12 @@ import shutil import sys import tempfile +import textwrap import nf_core +log = logging.getLogger(__name__) + class PipelineCreate(object): """Creates a nf-core pipeline a la carte from the nf-core best-practise template. @@ -27,11 +31,12 @@ class PipelineCreate(object): May the force be with you. outdir (str): Path to the local output directory. """ - def __init__(self, name, description, author, new_version='1.0dev', no_git=False, force=False, outdir=None): - self.short_name = name.lower().replace(r'/\s+/', '-').replace('nf-core/', '').replace('/', '-') - self.name = 'nf-core/{}'.format(self.short_name) - self.name_noslash = self.name.replace('/', '-') - self.name_docker = self.name.replace('nf-core', 'nfcore') + + def __init__(self, name, description, author, new_version="1.0dev", no_git=False, force=False, outdir=None): + self.short_name = name.lower().replace(r"/\s+/", "-").replace("nf-core/", "").replace("/", "-") + self.name = "nf-core/{}".format(self.short_name) + self.name_noslash = self.name.replace("/", "-") + self.name_docker = self.name.replace("nf-core", "nfcore") self.description = description self.author = author self.new_version = new_version @@ -54,40 +59,47 @@ def init_pipeline(self): if not self.no_git: self.git_init_pipeline() + log.info( + "[green bold]!!!!!! IMPORTANT !!!!!!\n\n" + + "[green not bold]If you are interested in adding your pipeline to the nf-core community,\n" + + "PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE!\n\n" + + "[default]Please read: [link=https://nf-co.re/developers/adding_pipelines#join-the-community]https://nf-co.re/developers/adding_pipelines#join-the-community[/link]" + ) + def run_cookiecutter(self): """Runs cookiecutter to create a new nf-core pipeline. """ - logging.info("Creating new nf-core pipeline: {}".format(self.name)) + log.info("Creating new nf-core pipeline: {}".format(self.name)) # Check if the output directory exists if os.path.exists(self.outdir): if self.force: - logging.warning("Output directory '{}' exists - continuing as --force specified".format(self.outdir)) + log.warning("Output directory '{}' exists - continuing as --force specified".format(self.outdir)) else: - logging.error("Output directory '{}' exists!".format(self.outdir)) - logging.info("Use -f / --force to overwrite existing files") + log.error("Output directory '{}' exists!".format(self.outdir)) + log.info("Use -f / --force to overwrite existing files") sys.exit(1) else: os.makedirs(self.outdir) # Build the template in a temporary directory self.tmpdir = tempfile.mkdtemp() - template = os.path.join(os.path.dirname(os.path.realpath(nf_core.__file__)), 'pipeline-template/') + template = os.path.join(os.path.dirname(os.path.realpath(nf_core.__file__)), "pipeline-template/") cookiecutter.main.cookiecutter( template, - extra_context = { - 'name': self.name, - 'description': self.description, - 'author': self.author, - 'name_noslash': self.name_noslash, - 'name_docker': self.name_docker, - 'short_name': self.short_name, - 'version': self.new_version, - 'nf_core_version': nf_core.__version__ + extra_context={ + "name": self.name, + "description": self.description, + "author": self.author, + "name_noslash": self.name_noslash, + "name_docker": self.name_docker, + "short_name": self.short_name, + "version": self.new_version, + "nf_core_version": nf_core.__version__, }, - no_input = True, - overwrite_if_exists = self.force, - output_dir = self.tmpdir + no_input=True, + overwrite_if_exists=self.force, + output_dir=self.tmpdir, ) # Make a logo and save it @@ -105,32 +117,37 @@ def make_pipeline_logo(self): """ logo_url = "https://nf-co.re/logo/{}".format(self.short_name) - logging.debug("Fetching logo from {}".format(logo_url)) + log.debug("Fetching logo from {}".format(logo_url)) email_logo_path = "{}/{}/assets/{}_logo.png".format(self.tmpdir, self.name_noslash, self.name_noslash) - logging.debug("Writing logo to {}".format(email_logo_path)) + log.debug("Writing logo to {}".format(email_logo_path)) r = requests.get("{}?w=400".format(logo_url)) - with open(email_logo_path, 'wb') as fh: + with open(email_logo_path, "wb") as fh: fh.write(r.content) readme_logo_path = "{}/{}/docs/images/{}_logo.png".format(self.tmpdir, self.name_noslash, self.name_noslash) - logging.debug("Writing logo to {}".format(readme_logo_path)) + log.debug("Writing logo to {}".format(readme_logo_path)) if not os.path.exists(os.path.dirname(readme_logo_path)): os.makedirs(os.path.dirname(readme_logo_path)) r = requests.get("{}?w=600".format(logo_url)) - with open(readme_logo_path, 'wb') as fh: + with open(readme_logo_path, "wb") as fh: fh.write(r.content) def git_init_pipeline(self): """Initialises the new pipeline as a Git repository and submits first commit. """ - logging.info("Initialising pipeline git repository") + log.info("Initialising pipeline git repository") repo = git.Repo.init(self.outdir) repo.git.add(A=True) repo.index.commit("initial template build from nf-core/tools, version {}".format(nf_core.__version__)) - #Add TEMPLATE branch to git repository - repo.git.branch('TEMPLATE') - repo.git.branch('dev') - logging.info("Done. Remember to add a remote and push to GitHub:\n cd {}\n git remote add origin git@github.com:USERNAME/REPO_NAME.git\n git push --all origin".format(self.outdir)) - logging.info("This will also push your newly created dev branch and the TEMPLATE branch for syncing.") + # Add TEMPLATE branch to git repository + repo.git.branch("TEMPLATE") + repo.git.branch("dev") + log.info( + "Done. Remember to add a remote and push to GitHub:\n" + + "[white on grey23] cd {} \n".format(self.outdir) + + " git remote add origin git@github.com:USERNAME/REPO_NAME.git \n" + + " git push --all origin " + ) + log.info("This will also push your newly created dev branch and the TEMPLATE branch for syncing.") diff --git a/nf_core/download.py b/nf_core/download.py index e59bc75d6d..a6b2cfc2d3 100644 --- a/nf_core/download.py +++ b/nf_core/download.py @@ -18,6 +18,8 @@ import nf_core.list import nf_core.utils +log = logging.getLogger(__name__) + class DownloadWorkflow(object): """Downloads a nf-core workflow from GitHub to the local file system. @@ -30,14 +32,15 @@ class DownloadWorkflow(object): singularity (bool): Flag, if the Singularity container should be downloaded as well. Defaults to False. outdir (str): Path to the local download directory. Defaults to None. """ - def __init__(self, pipeline, release=None, singularity=False, outdir=None, compress_type='tar.gz'): + + def __init__(self, pipeline, release=None, singularity=False, outdir=None, compress_type="tar.gz"): self.pipeline = pipeline self.release = release self.singularity = singularity self.outdir = outdir self.output_filename = None self.compress_type = compress_type - if self.compress_type == 'none': + if self.compress_type == "none": self.compress_type = None self.wf_name = None @@ -58,59 +61,62 @@ def download_workflow(self): # Set an output filename now that we have the outdir if self.compress_type is not None: - self.output_filename = '{}.{}'.format(self.outdir, self.compress_type) + self.output_filename = "{}.{}".format(self.outdir, self.compress_type) output_logmsg = "Output file: {}".format(self.output_filename) # Check that the outdir doesn't already exist if os.path.exists(self.outdir): - logging.error("Output directory '{}' already exists".format(self.outdir)) + log.error("Output directory '{}' already exists".format(self.outdir)) sys.exit(1) # Check that compressed output file doesn't already exist if self.output_filename and os.path.exists(self.output_filename): - logging.error("Output file '{}' already exists".format(self.output_filename)) + log.error("Output file '{}' already exists".format(self.output_filename)) sys.exit(1) - logging.info( - "Saving {}".format(self.pipeline) + - "\n Pipeline release: {}".format(self.release) + - "\n Pull singularity containers: {}".format('Yes' if self.singularity else 'No') + - "\n {}".format(output_logmsg) + log.info( + "Saving {}".format(self.pipeline) + + "\n Pipeline release: {}".format(self.release) + + "\n Pull singularity containers: {}".format("Yes" if self.singularity else "No") + + "\n {}".format(output_logmsg) ) # Download the pipeline files - logging.info("Downloading workflow files from GitHub") + log.info("Downloading workflow files from GitHub") self.download_wf_files() # Download the centralised configs - logging.info("Downloading centralised configs from GitHub") + log.info("Downloading centralised configs from GitHub") self.download_configs() self.wf_use_local_configs() # Download the singularity images if self.singularity: - logging.debug("Fetching container names for workflow") + log.debug("Fetching container names for workflow") self.find_container_images() if len(self.containers) == 0: - logging.info("No container names found in workflow") + log.info("No container names found in workflow") else: - os.mkdir(os.path.join(self.outdir, 'singularity-images')) - logging.info("Downloading {} singularity container{}".format(len(self.containers), 's' if len(self.containers) > 1 else '')) + os.mkdir(os.path.join(self.outdir, "singularity-images")) + log.info( + "Downloading {} singularity container{}".format( + len(self.containers), "s" if len(self.containers) > 1 else "" + ) + ) for container in self.containers: try: - # Download from Dockerhub in all cases + # Download from Docker Hub in all cases self.pull_singularity_image(container) except RuntimeWarning as r: # Raise exception if this is not possible - logging.error("Not able to pull image. Service might be down or internet connection is dead.") + log.error("Not able to pull image. Service might be down or internet connection is dead.") raise r # Compress into an archive if self.compress_type is not None: - logging.info("Compressing download..") + log.info("Compressing download..") self.compress_download() - def fetch_workflow_details(self, wfs): """Fetches details of a nf-core workflow to download. @@ -132,62 +138,68 @@ def fetch_workflow_details(self, wfs): # Find latest release hash if self.release is None and len(wf.releases) > 0: # Sort list of releases so that most recent is first - wf.releases = sorted(wf.releases, key=lambda k: k.get('published_at_timestamp', 0), reverse=True) - self.release = wf.releases[0]['tag_name'] - self.wf_sha = wf.releases[0]['tag_sha'] - logging.debug("No release specified. Using latest release: {}".format(self.release)) + wf.releases = sorted(wf.releases, key=lambda k: k.get("published_at_timestamp", 0), reverse=True) + self.release = wf.releases[0]["tag_name"] + self.wf_sha = wf.releases[0]["tag_sha"] + log.debug("No release specified. Using latest release: {}".format(self.release)) # Find specified release hash elif self.release is not None: for r in wf.releases: - if r['tag_name'] == self.release.lstrip('v'): - self.wf_sha = r['tag_sha'] + if r["tag_name"] == self.release.lstrip("v"): + self.wf_sha = r["tag_sha"] break else: - logging.error("Not able to find release '{}' for {}".format(self.release, wf.full_name)) - logging.info("Available {} releases: {}".format(wf.full_name, ', '.join([r['tag_name'] for r in wf.releases]))) + log.error("Not able to find release '{}' for {}".format(self.release, wf.full_name)) + log.info( + "Available {} releases: {}".format( + wf.full_name, ", ".join([r["tag_name"] for r in wf.releases]) + ) + ) raise LookupError("Not able to find release '{}' for {}".format(self.release, wf.full_name)) # Must be a dev-only pipeline elif not self.release: - self.release = 'dev' - self.wf_sha = 'master' # Cheating a little, but GitHub download link works - logging.warning("Pipeline is in development - downloading current code on master branch.\n" + - "This is likely to change soon should not be considered fully reproducible.") + self.release = "dev" + self.wf_sha = "master" # Cheating a little, but GitHub download link works + log.warning( + "Pipeline is in development - downloading current code on master branch.\n" + + "This is likely to change soon should not be considered fully reproducible." + ) # Set outdir name if not defined if not self.outdir: - self.outdir = 'nf-core-{}'.format(wf.name) + self.outdir = "nf-core-{}".format(wf.name) if self.release is not None: - self.outdir += '-{}'.format(self.release) + self.outdir += "-{}".format(self.release) # Set the download URL and return - self.wf_download_url = 'https://github.com/{}/archive/{}.zip'.format(wf.full_name, self.wf_sha) + self.wf_download_url = "https://github.com/{}/archive/{}.zip".format(wf.full_name, self.wf_sha) return # If we got this far, must not be a nf-core pipeline - if self.pipeline.count('/') == 1: + if self.pipeline.count("/") == 1: # Looks like a GitHub address - try working with this repo - logging.warning("Pipeline name doesn't match any nf-core workflows") - logging.info("Pipeline name looks like a GitHub address - attempting to download anyway") + log.warning("Pipeline name doesn't match any nf-core workflows") + log.info("Pipeline name looks like a GitHub address - attempting to download anyway") self.wf_name = self.pipeline if not self.release: - self.release = 'master' + self.release = "master" self.wf_sha = self.release if not self.outdir: - self.outdir = self.pipeline.replace('/', '-').lower() + self.outdir = self.pipeline.replace("/", "-").lower() if self.release is not None: - self.outdir += '-{}'.format(self.release) + self.outdir += "-{}".format(self.release) # Set the download URL and return - self.wf_download_url = 'https://github.com/{}/archive/{}.zip'.format(self.pipeline, self.release) + self.wf_download_url = "https://github.com/{}/archive/{}.zip".format(self.pipeline, self.release) else: - logging.error("Not able to find pipeline '{}'".format(self.pipeline)) - logging.info("Available pipelines: {}".format(', '.join([w.name for w in wfs.remote_workflows]))) + log.error("Not able to find pipeline '{}'".format(self.pipeline)) + log.info("Available pipelines: {}".format(", ".join([w.name for w in wfs.remote_workflows]))) raise LookupError("Not able to find pipeline '{}'".format(self.pipeline)) def download_wf_files(self): """Downloads workflow files from GitHub to the :attr:`self.outdir`. """ - logging.debug("Downloading {}".format(self.wf_download_url)) + log.debug("Downloading {}".format(self.wf_download_url)) # Download GitHub zip file into memory and extract url = requests.get(self.wf_download_url) @@ -195,11 +207,11 @@ def download_wf_files(self): zipfile.extractall(self.outdir) # Rename the internal directory name to be more friendly - gh_name = '{}-{}'.format(self.wf_name, self.wf_sha).split('/')[-1] - os.rename(os.path.join(self.outdir, gh_name), os.path.join(self.outdir, 'workflow')) + gh_name = "{}-{}".format(self.wf_name, self.wf_sha).split("/")[-1] + os.rename(os.path.join(self.outdir, gh_name), os.path.join(self.outdir, "workflow")) # Make downloaded files executable - for dirpath, subdirs, filelist in os.walk(os.path.join(self.outdir, 'workflow')): + for dirpath, subdirs, filelist in os.walk(os.path.join(self.outdir, "workflow")): for fname in filelist: os.chmod(os.path.join(dirpath, fname), 0o775) @@ -208,7 +220,7 @@ def download_configs(self): """ configs_zip_url = "https://github.com/nf-core/configs/archive/master.zip" configs_local_dir = "configs-master" - logging.debug("Downloading {}".format(configs_zip_url)) + log.debug("Downloading {}".format(configs_zip_url)) # Download GitHub zip file into memory and extract url = requests.get(configs_zip_url) @@ -216,45 +228,43 @@ def download_configs(self): zipfile.extractall(self.outdir) # Rename the internal directory name to be more friendly - os.rename(os.path.join(self.outdir, configs_local_dir), os.path.join(self.outdir, 'configs')) + os.rename(os.path.join(self.outdir, configs_local_dir), os.path.join(self.outdir, "configs")) # Make downloaded files executable - for dirpath, subdirs, filelist in os.walk(os.path.join(self.outdir, 'configs')): + for dirpath, subdirs, filelist in os.walk(os.path.join(self.outdir, "configs")): for fname in filelist: os.chmod(os.path.join(dirpath, fname), 0o775) def wf_use_local_configs(self): """Edit the downloaded nextflow.config file to use the local config files """ - nfconfig_fn = os.path.join(self.outdir, 'workflow', 'nextflow.config') - find_str = 'https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}' - repl_str = '../configs/' - logging.debug("Editing params.custom_config_base in {}".format(nfconfig_fn)) + nfconfig_fn = os.path.join(self.outdir, "workflow", "nextflow.config") + find_str = "https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}" + repl_str = "../configs/" + log.debug("Editing params.custom_config_base in {}".format(nfconfig_fn)) # Load the nextflow.config file into memory - with open(nfconfig_fn, 'r') as nfconfig_fh: - nfconfig = nfconfig_fh.read() + with open(nfconfig_fn, "r") as nfconfig_fh: + nfconfig = nfconfig_fh.read() # Replace the target string nfconfig = nfconfig.replace(find_str, repl_str) # Write the file out again - with open(nfconfig_fn, 'w') as nfconfig_fh: - nfconfig_fh.write(nfconfig) - + with open(nfconfig_fn, "w") as nfconfig_fh: + nfconfig_fh.write(nfconfig) def find_container_images(self): """ Find container image names for workflow """ # Use linting code to parse the pipeline nextflow config - self.config = nf_core.utils.fetch_wf_config(os.path.join(self.outdir, 'workflow')) + self.config = nf_core.utils.fetch_wf_config(os.path.join(self.outdir, "workflow")) # Find any config variables that look like a container - for k,v in self.config.items(): - if k.startswith('process.') and k.endswith('.container'): + for k, v in self.config.items(): + if k.startswith("process.") and k.endswith(".container"): self.containers.append(v.strip('"').strip("'")) - def pull_singularity_image(self, container): """Uses a local installation of singularity to pull an image from Docker Hub. @@ -265,12 +275,12 @@ def pull_singularity_image(self, container): Raises: Various exceptions possible from `subprocess` execution of Singularity. """ - out_name = '{}.simg'.format(container.replace('nfcore', 'nf-core').replace('/','-').replace(':', '-')) - out_path = os.path.abspath(os.path.join(self.outdir, 'singularity-images', out_name)) - address = 'docker://{}'.format(container.replace('docker://', '')) + out_name = "{}.simg".format(container.replace("nfcore", "nf-core").replace("/", "-").replace(":", "-")) + out_path = os.path.abspath(os.path.join(self.outdir, "singularity-images", out_name)) + address = "docker://{}".format(container.replace("docker://", "")) singularity_command = ["singularity", "pull", "--name", out_path, address] - logging.info("Building singularity image from dockerhub: {}".format(address)) - logging.debug("Singularity command: {}".format(' '.join(singularity_command))) + log.info("Building singularity image from Docker Hub: {}".format(address)) + log.debug("Singularity command: {}".format(" ".join(singularity_command))) # Try to use singularity to pull image try: @@ -278,7 +288,7 @@ def pull_singularity_image(self, container): except OSError as e: if e.errno == errno.ENOENT: # Singularity is not installed - logging.error('Singularity is not installed!') + log.error("Singularity is not installed!") else: # Something else went wrong with singularity command raise e @@ -286,36 +296,35 @@ def pull_singularity_image(self, container): def compress_download(self): """Take the downloaded files and make a compressed .tar.gz archive. """ - logging.debug('Creating archive: {}'.format(self.output_filename)) + log.debug("Creating archive: {}".format(self.output_filename)) # .tar.gz and .tar.bz2 files - if self.compress_type == 'tar.gz' or self.compress_type == 'tar.bz2': - ctype = self.compress_type.split('.')[1] + if self.compress_type == "tar.gz" or self.compress_type == "tar.bz2": + ctype = self.compress_type.split(".")[1] with tarfile.open(self.output_filename, "w:{}".format(ctype)) as tar: tar.add(self.outdir, arcname=os.path.basename(self.outdir)) - tar_flags = 'xzf' if ctype == 'gz' else 'xjf' - logging.info('Command to extract files: tar -{} {}'.format(tar_flags, self.output_filename)) + tar_flags = "xzf" if ctype == "gz" else "xjf" + log.info("Command to extract files: tar -{} {}".format(tar_flags, self.output_filename)) # .zip files - if self.compress_type == 'zip': - with ZipFile(self.output_filename, 'w') as zipObj: - # Iterate over all the files in directory - for folderName, subfolders, filenames in os.walk(self.outdir): - for filename in filenames: - #create complete filepath of file in directory - filePath = os.path.join(folderName, filename) - # Add file to zip - zipObj.write(filePath) - logging.info('Command to extract files: unzip {}'.format(self.output_filename)) + if self.compress_type == "zip": + with ZipFile(self.output_filename, "w") as zipObj: + # Iterate over all the files in directory + for folderName, subfolders, filenames in os.walk(self.outdir): + for filename in filenames: + # create complete filepath of file in directory + filePath = os.path.join(folderName, filename) + # Add file to zip + zipObj.write(filePath) + log.info("Command to extract files: unzip {}".format(self.output_filename)) # Delete original files - logging.debug('Deleting uncompressed files: {}'.format(self.outdir)) + log.debug("Deleting uncompressed files: {}".format(self.outdir)) shutil.rmtree(self.outdir) # Caclualte md5sum for output file self.validate_md5(self.output_filename) - def validate_md5(self, fname, expected=None): """Calculates the md5sum for a file on the disk and validate with expected. @@ -326,7 +335,7 @@ def validate_md5(self, fname, expected=None): Raises: IOError, if the md5sum does not match the remote sum. """ - logging.debug("Validating image hash: {}".format(fname)) + log.debug("Validating image hash: {}".format(fname)) # Calculate the md5 for the file on disk hash_md5 = hashlib.md5() @@ -336,9 +345,9 @@ def validate_md5(self, fname, expected=None): file_hash = hash_md5.hexdigest() if expected is None: - logging.info("MD5 checksum for {}: {}".format(fname, file_hash)) + log.info("MD5 checksum for {}: {}".format(fname, file_hash)) else: if file_hash == expected: - logging.debug('md5 sum of image matches expected: {}'.format(expected)) + log.debug("md5 sum of image matches expected: {}".format(expected)) else: - raise IOError ("{} md5 does not match remote: {} - {}".format(fname, expected, file_hash)) + raise IOError("{} md5 does not match remote: {} - {}".format(fname, expected, file_hash)) diff --git a/nf_core/launch.py b/nf_core/launch.py index ff297c7b02..488d19c224 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -2,416 +2,703 @@ """ Launch a pipeline, interactively collecting params """ from __future__ import print_function -from collections import OrderedDict +from rich.console import Console +from rich.markdown import Markdown +from rich.prompt import Confirm -import click -import errno -import jsonschema +import copy +import json import logging import os +import PyInquirer import re import subprocess +import textwrap +import webbrowser -import nf_core.utils, nf_core.list -import nf_core.workflow.parameters, nf_core.workflow.validation, nf_core.workflow.workflow +import nf_core.schema, nf_core.utils -def launch_pipeline(workflow, params_local_uri, direct): +log = logging.getLogger(__name__) - # Create a pipeline launch object - launcher = Launch(workflow) +# +# NOTE: When PyInquirer 1.0.3 is released we can capture keyboard interruptions +# in a nicer way # with the raise_keyboard_interrupt=True argument in the PyInquirer.prompt() calls +# It also allows list selections to have a default set. +# +# Until then we have workarounds: +# * Default list item is moved to the top of the list +# * We manually raise a KeyboardInterrupt if we get None back from a question +# - # Get nextflow to fetch the workflow if we don't already have it - if not launcher.wf_ispath: - launcher.get_local_wf() - # Get the pipeline default parameters - launcher.parse_parameter_settings(params_local_uri) +class Launch(object): + """ Class to hold config option to launch a pipeline """ - # Find extra params from `nextflow config` command and main.nf - launcher.collect_pipeline_param_defaults() + def __init__( + self, + pipeline=None, + revision=None, + command_only=False, + params_in=None, + params_out=None, + save_all=False, + show_hidden=False, + url=None, + web_id=None, + ): + """Initialise the Launcher class - # Group the parameters - launcher.group_parameters() + Args: + schema: An nf_core.schema.PipelineSchema() object + """ - # Kick off the interactive wizard to collect user inputs - launcher.prompt_core_nxf_flags() - if not direct: - launcher.prompt_param_flags() + self.pipeline = pipeline + self.pipeline_revision = revision + self.schema_obj = None + self.use_params_file = False if command_only else True + self.params_in = params_in + self.params_out = params_out if params_out else os.path.join(os.getcwd(), "nf-params.json") + self.save_all = save_all + self.show_hidden = show_hidden + self.web_schema_launch_url = url if url else "https://nf-co.re/launch" + self.web_schema_launch_web_url = None + self.web_schema_launch_api_url = None + self.web_id = web_id + if self.web_id: + self.web_schema_launch_web_url = "{}?id={}".format(self.web_schema_launch_url, web_id) + self.web_schema_launch_api_url = "{}?id={}&api=true".format(self.web_schema_launch_url, web_id) + self.nextflow_cmd = "nextflow run {}".format(self.pipeline) + + # Prepend property names with a single hyphen in case we have parameters with the same ID + self.nxf_flag_schema = { + "coreNextflow": { + "title": "Nextflow command-line flags", + "type": "object", + "description": "General Nextflow flags to control how the pipeline runs.", + "help_text": "These are not specific to the pipeline and will not be saved in any parameter file. They are just used when building the `nextflow run` launch command.", + "properties": { + "-name": { + "type": "string", + "description": "Unique name for this nextflow run", + "help_text": "If not specified, Nextflow will automatically generate a random mnemonic.", + "pattern": "^[a-zA-Z0-9-_]+$", + }, + "-profile": {"type": "string", "description": "Configuration profile"}, + "-work-dir": { + "type": "string", + "description": "Work directory for intermediate files", + "default": os.getenv("NXF_WORK") if os.getenv("NXF_WORK") else "./work", + }, + "-resume": { + "type": "boolean", + "description": "Resume previous run, if found", + "help_text": "Execute the script using the cached results, useful to continue executions that was stopped by an error", + "default": False, + }, + }, + } + } + self.nxf_flags = {} + self.params_user = {} + self.cli_launch = True - # Build and launch the `nextflow run` command - launcher.build_command() - launcher.launch_workflow() + def launch_pipeline(self): -class Launch(object): - """ Class to hold config option to launch a pipeline """ + # Check that we have everything we need + if self.pipeline is None and self.web_id is None: + log.error( + "Either a pipeline name or web cache ID is required. Please see nf-core launch --help for more information." + ) + return False + + # Check if the output file exists already + if os.path.exists(self.params_out): + log.warning("Parameter output file already exists! {}".format(os.path.relpath(self.params_out))) + if Confirm.ask("[yellow]Do you want to overwrite this file?"): + os.remove(self.params_out) + log.info("Deleted {}\n".format(self.params_out)) + else: + log.info("Exiting. Use --params-out to specify a custom filename.") + return False + + log.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") + + # Check if we have a web ID + if self.web_id is not None: + self.schema_obj = nf_core.schema.PipelineSchema() + try: + if not self.get_web_launch_response(): + log.info( + "Waiting for form to be completed in the browser. Remember to click Finished when you're done." + ) + log.info("URL: {}".format(self.web_schema_launch_web_url)) + nf_core.utils.wait_cli_function(self.get_web_launch_response) + except AssertionError as e: + log.error(e.args[0]) + return False + + # Load local params if supplied + self.set_schema_inputs() + # Load schema defaults + self.schema_obj.get_schema_defaults() + + # No --id supplied, fetch parameter inputs + else: + # Build the schema and starting inputs + if self.get_pipeline_schema() is False: + return False + self.set_schema_inputs() + self.merge_nxf_flag_schema() + + # Collect user inputs via web or cli + if self.prompt_web_gui(): + try: + self.launch_web_gui() + except AssertionError as e: + log.error(e.args[0]) + return False + else: + # Kick off the interactive wizard to collect user inputs + self.prompt_schema() + + # Validate the parameters that we now have + if not self.schema_obj.validate_params(): + return False + + # Strip out the defaults + if not self.save_all: + self.strip_default_params() + + # Build and launch the `nextflow run` command + self.build_command() + self.launch_workflow() + + def get_pipeline_schema(self): + """ Load and validate the schema from the supplied pipeline """ + + # Set up the schema + self.schema_obj = nf_core.schema.PipelineSchema() - def __init__(self, workflow): - """ Initialise the class with empty placeholder vars """ - - # Check if the workflow name is actually a path - self.wf_ispath = os.path.exists(workflow) - - # Prepend nf-core/ if it seems sensible - if 'nf-core' not in workflow and workflow.count('/') == 0 and not self.wf_ispath: - workflow = "nf-core/{}".format(workflow) - logging.debug("Prepending nf-core/ to workflow") - logging.info("Launching {}".format(workflow)) - - # Get list of local workflows to see if we have a cached version - self.local_wf = None - if not self.wf_ispath: - wfs = nf_core.list.Workflows() - wfs.get_local_nf_workflows() - for wf in wfs.local_workflows: - if workflow == wf.full_name: - self.local_wf = wf - - self.workflow = workflow - self.nxf_flag_defaults = { - '-name': None, - '-r': None, - '-profile': 'standard', - '-w': os.getenv('NXF_WORK') if os.getenv('NXF_WORK') else './work', - '-resume': False + # Check if this is a local directory + if os.path.exists(self.pipeline): + # Set the nextflow launch command to use full paths + self.nextflow_cmd = "nextflow run {}".format(os.path.abspath(self.pipeline)) + else: + # Assume nf-core if no org given + if self.pipeline.count("/") == 0: + self.nextflow_cmd = "nextflow run nf-core/{}".format(self.pipeline) + # Add revision flag to commands if set + if self.pipeline_revision: + self.nextflow_cmd += " -r {}".format(self.pipeline_revision) + + # Get schema from name, load it and lint it + try: + self.schema_obj.get_schema_path(self.pipeline, revision=self.pipeline_revision) + self.schema_obj.load_lint_schema() + except AssertionError: + # No schema found + # Check that this was actually a pipeline + if self.schema_obj.pipeline_dir is None or not os.path.exists(self.schema_obj.pipeline_dir): + log.error("Could not find pipeline: {} ({})".format(self.pipeline, self.schema_obj.pipeline_dir)) + return False + if not os.path.exists(os.path.join(self.schema_obj.pipeline_dir, "nextflow.config")) and not os.path.exists( + os.path.join(self.schema_obj.pipeline_dir, "main.nf") + ): + log.error("Could not find a main.nf or nextfow.config file, are you sure this is a pipeline?") + return False + + # Build a schema for this pipeline + log.info("No pipeline schema found - creating one from the config") + try: + self.schema_obj.get_wf_params() + self.schema_obj.make_skeleton_schema() + self.schema_obj.remove_schema_notfound_configs() + self.schema_obj.add_schema_found_configs() + self.schema_obj.get_schema_defaults() + except AssertionError as e: + log.error("Could not build pipeline schema: {}".format(e)) + return False + + def set_schema_inputs(self): + """ + Take the loaded schema and set the defaults as the input parameters + If a nf_params.json file is supplied, apply these over the top + """ + # Set the inputs to the schema defaults unless already set by --id + if len(self.schema_obj.input_params) == 0: + self.schema_obj.input_params = copy.deepcopy(self.schema_obj.schema_defaults) + + # If we have a params_file, load and validate it against the schema + if self.params_in: + log.info("Loading {}".format(self.params_in)) + self.schema_obj.load_input_params(self.params_in) + self.schema_obj.validate_params() + + def merge_nxf_flag_schema(self): + """ Take the Nextflow flag schema and merge it with the pipeline schema """ + # Add the coreNextflow subschema to the schema definitions + if "definitions" not in self.schema_obj.schema: + self.schema_obj.schema["definitions"] = {} + self.schema_obj.schema["definitions"].update(self.nxf_flag_schema) + # Add the new defintion to the allOf key so that it's included in validation + # Put it at the start of the list so that it comes first + if "allOf" not in self.schema_obj.schema: + self.schema_obj.schema["allOf"] = [] + self.schema_obj.schema["allOf"].insert(0, {"$ref": "#/definitions/coreNextflow"}) + + def prompt_web_gui(self): + """ Ask whether to use the web-based or cli wizard to collect params """ + log.info( + "[magenta]Would you like to enter pipeline parameters using a web-based interface or a command-line wizard?" + ) + question = { + "type": "list", + "name": "use_web_gui", + "message": "Choose launch method", + "choices": ["Web based", "Command line"], } - self.nxf_flag_help = { - '-name': 'Unique name for this nextflow run', - '-r': 'Release / revision to use', - '-profile': 'Config profile to use', - '-w': 'Work directory for intermediate files', - '-resume': 'Resume a previous workflow run' + answer = PyInquirer.prompt([question]) + # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + if answer == {}: + raise KeyboardInterrupt + return answer["use_web_gui"] == "Web based" + + def launch_web_gui(self): + """ Send schema to nf-core website and launch input GUI """ + + content = { + "post_content": "json_schema_launcher", + "api": "true", + "version": nf_core.__version__, + "status": "waiting_for_user", + "schema": json.dumps(self.schema_obj.schema), + "nxf_flags": json.dumps(self.nxf_flags), + "input_params": json.dumps(self.schema_obj.input_params), + "cli_launch": True, + "nextflow_cmd": self.nextflow_cmd, + "pipeline": self.pipeline, + "revision": self.pipeline_revision, } - self.nxf_flags = {} - self.parameters = [] - self.parameter_keys = [] - self.grouped_parameters = OrderedDict() - self.params_user = {} - self.nextflow_cmd = "nextflow run {}".format(self.workflow) - self.use_params_file = True + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_url, content) + try: + assert "api_url" in web_response + assert "web_url" in web_response + assert web_response["status"] == "recieved" + except AssertionError: + log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + raise AssertionError( + "Web launch response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format( + self.web_schema_launch_url + ) + ) + else: + self.web_schema_launch_web_url = web_response["web_url"] + self.web_schema_launch_api_url = web_response["api_url"] + + # Launch the web GUI + log.info("Opening URL: {}".format(self.web_schema_launch_web_url)) + webbrowser.open(self.web_schema_launch_web_url) + log.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") + nf_core.utils.wait_cli_function(self.get_web_launch_response) - def get_local_wf(self): + def get_web_launch_response(self): """ - Check if this workflow has a local copy and use nextflow to pull it if not + Given a URL for a web-gui launch response, recursively query it until results are ready. """ - if not self.local_wf: - logging.info("Downloading workflow: {}".format(self.workflow)) + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_api_url) + if web_response["status"] == "error": + raise AssertionError("Got error from launch API ({})".format(web_response.get("message"))) + elif web_response["status"] == "waiting_for_user": + return False + elif web_response["status"] == "launch_params_complete": + log.info("Found completed parameters from nf-core launch GUI") try: - with open(os.devnull, 'w') as devnull: - subprocess.check_output(['nextflow', 'pull', self.workflow], stderr=devnull) - except OSError as e: - if e.errno == errno.ENOENT: - raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") - except subprocess.CalledProcessError as e: - raise AssertionError("`nextflow pull` returned non-zero error code: %s,\n %s", e.returncode, e.output) - else: - self.local_wf = nf_core.list.LocalWorkflow(self.workflow) - self.local_wf.get_local_nf_workflow_details() + # Set everything that we can with the cache results + # NB: If using web builder, may have only run with --id and nothing else + if len(web_response["nxf_flags"]) > 0: + self.nxf_flags = web_response["nxf_flags"] + if len(web_response["input_params"]) > 0: + self.schema_obj.input_params = web_response["input_params"] + self.schema_obj.schema = web_response["schema"] + self.cli_launch = web_response["cli_launch"] + self.nextflow_cmd = web_response["nextflow_cmd"] + self.pipeline = web_response["pipeline"] + self.pipeline_revision = web_response["revision"] + # Sanitise form inputs, set proper variable types etc + self.sanitise_web_response() + except KeyError as e: + raise AssertionError("Missing return key from web API: {}".format(e)) + except Exception as e: + log.debug(web_response) + raise AssertionError( + "Unknown exception ({}) - see verbose log for details. {}".format(type(e).__name__, e) + ) + return True + else: + log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + raise AssertionError( + "Web launch GUI returned unexpected status ({}): {}\n See verbose log for full response".format( + web_response["status"], self.web_schema_launch_api_url + ) + ) - def parse_parameter_settings(self, params_local_uri = None): + def sanitise_web_response(self): """ - Load full parameter info from the pipeline parameters.settings.json file + The web builder returns everything as strings. + Use the functions defined in the cli wizard to convert to the correct types. """ - try: - params_json_str = None - # Params file supplied to launch command - if params_local_uri: - with open(params_local_uri, 'r') as fp: - params_json_str = fp.read() - # Get workflow file from local cached copy + # Collect pyinquirer objects for each defined input_param + pyinquirer_objects = {} + for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items(): + pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) + + for d_key, definition in self.schema_obj.schema.get("definitions", {}).items(): + for param_id, param_obj in definition.get("properties", {}).items(): + pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) + + # Go through input params and sanitise + for params in [self.nxf_flags, self.schema_obj.input_params]: + for param_id in list(params.keys()): + # Remove if an empty string + if str(params[param_id]).strip() == "": + del params[param_id] + continue + # Run filter function on value + filter_func = pyinquirer_objects.get(param_id, {}).get("filter") + if filter_func is not None: + params[param_id] = filter_func(params[param_id]) + + def prompt_schema(self): + """ Go through the pipeline schema and prompt user to change defaults """ + answers = {} + # Start with the subschema in the definitions - use order of allOf + for allOf in self.schema_obj.schema.get("allOf", []): + d_key = allOf["$ref"][14:] + answers.update(self.prompt_group(d_key, self.schema_obj.schema["definitions"][d_key])) + + # Top level schema params + for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items(): + if not param_obj.get("hidden", False) or self.show_hidden: + is_required = param_id in self.schema_obj.schema.get("required", []) + answers.update(self.prompt_param(param_id, param_obj, is_required, answers)) + + # Split answers into core nextflow options and params + for key, answer in answers.items(): + if key in self.nxf_flag_schema["coreNextflow"]["properties"]: + self.nxf_flags[key] = answer else: - if self.wf_ispath: - local_params_path = os.path.join(self.workflow, 'parameters.settings.json') - else: - local_params_path = os.path.join(self.local_wf.local_path, 'parameters.settings.json') - if os.path.exists(local_params_path): - with open(local_params_path, 'r') as fp: - params_json_str = fp.read() - if not params_json_str: - raise LookupError('parameters.settings.json file not found') - try: - self.parameters = nf_core.workflow.parameters.Parameters.create_from_json(params_json_str) - for p in self.parameters: - self.parameter_keys.append(p.name) - logging.debug("Found param from parameters.settings.json: param.{}".format(p.name)) - except ValueError as e: - logging.error("Could not parse pipeline parameters.settings.json JSON:\n {}\n".format(e)) - except jsonschema.exceptions.ValidationError as e: - logging.error("Validation error with pipeline parameters.settings.json:\n Message: {}\n Instance: {}\n".format(e.message, e.instance)) - except LookupError as e: - print("WARNING: Could not parse parameter settings file for `{pipeline}`:\n {exception}".format( - pipeline=self.workflow, exception=e)) - - def collect_pipeline_param_defaults(self): - """ Collect the default params and values from the workflow """ - logging.debug("Collecting pipeline parameter defaults\n") - config = nf_core.utils.fetch_wf_config(self.workflow, self.local_wf) - for key, value in config.items(): - keys = key.split('.') - if keys[0] == 'params' and len(keys) == 2 and keys[1] not in self.parameter_keys: - - # Try to guess the variable type from the default value - p_type = 'string' - p_default = str(value) - # All digits - int - if value.isdigit(): - p_type = 'integer' - p_default = int(value) - else: - # Not just digis - try converting to a float - try: - p_default = float(value) - p_type = 'decimal' - except ValueError: - pass - # Strings 'true' and 'false' - booleans - if value == 'true' or value == 'false': - p_type = 'boolean' - p_default = True if value == 'true' else False - - # Build the Parameter object - parameter = (nf_core.workflow.parameters.Parameter.builder() - .name(keys[1]) - .label(keys[1]) - .usage(None) - .param_type(p_type) - .choices(None) - .default(p_default) - .pattern(".*") - .render("textfield") - .arity(None) - .group("Other pipeline parameters") - .build()) - self.parameters.append(parameter) - self.parameter_keys.append(keys[1]) - logging.debug("Discovered param from `nextflow config`: param.{}".format(keys[1])) - - # Not all parameters can be found with `nextflow config` - try searching main.nf and config files - searchfiles = [] - pattern = re.compile(r'params\.([\w\d]+)') - wf_base = self.workflow if self.wf_ispath else self.local_wf.local_path - if os.path.exists(os.path.join(wf_base, 'main.nf')): - searchfiles.append(os.path.join(wf_base, 'main.nf')) - if os.path.exists(os.path.join(wf_base, 'nextflow.config')): - searchfiles.append(os.path.join(wf_base, 'nextflow.config')) - if os.path.isdir(os.path.join(wf_base, 'conf')): - for fn in os.listdir(os.path.join(wf_base, 'conf')): - searchfiles.append(os.path.join(wf_base, 'conf', fn)) - for sf in searchfiles: - with open(sf, 'r') as fh: - for l in fh: - match = re.search(pattern, l) - if match: - param = match.group(1) - if param not in self.parameter_keys: - # Build the Parameter object - parameter = (nf_core.workflow.parameters.Parameter.builder() - .name(param) - .label(param) - .usage(None) - .param_type("string") - .choices(None) - .default("") - .pattern(".*") - .render("textfield") - .arity(None) - .group("Other pipeline parameters") - .build()) - self.parameters.append(parameter) - self.parameter_keys.append(param) - logging.debug("Discovered param from {}: param.{}".format(os.path.relpath(sf, wf_base), param)) - - def prompt_core_nxf_flags(self): - """ Ask the user if they want to override any default values """ - # Main nextflow flags - click.secho("Main nextflow options", bold=True, underline=True) - for flag, f_default in self.nxf_flag_defaults.items(): - - # Click prompts don't like None, so we have to use an empty string instead - f_default_print = f_default - if f_default is None: - f_default = '' - f_default_print = 'None' - - # Overwrite the default prompt for boolean - if isinstance(f_default, bool): - f_default_print = 'Y/n' if f_default else 'y/N' - - # Prompt for a response - f_user = click.prompt( - "\n{}\n {} {}".format( - self.nxf_flag_help[flag], - click.style(flag, fg='blue'), - click.style('[{}]'.format(str(f_default_print)), fg='green') - ), - default = f_default, - show_default = False - ) + self.params_user[key] = answer + + # Update schema with user params + self.schema_obj.input_params.update(self.params_user) + + def prompt_param(self, param_id, param_obj, is_required, answers): + """Prompt for a single parameter""" + + # Print the question + question = self.single_param_to_pyinquirer(param_id, param_obj, answers) + answer = PyInquirer.prompt([question]) + # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + if answer == {}: + raise KeyboardInterrupt + + # If required and got an empty reponse, ask again + while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required: + log.error("'–-{}' is required".format(param_id)) + answer = PyInquirer.prompt([question]) + # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + if answer == {}: + raise KeyboardInterrupt + + # Don't return empty answers + if answer[param_id] == "": + return {} + return answer + + def prompt_group(self, group_id, group_obj): + """ + Prompt for edits to a group of parameters (subschema in 'definitions') - # Only save if we've changed the default - if f_user != f_default: - # Convert string bools to real bools - try: - f_user = f_user.strip('"').strip("'") - if f_user.lower() == 'true': f_user = True - if f_user.lower() == 'false': f_user = False - except AttributeError: - pass - self.nxf_flags[flag] = f_user + Args: + group_id: Paramater ID (string) + group_obj: JSON Schema keys (dict) - def group_parameters(self): - """Groups parameters by their 'group' property. + Returns: + Dict of param_id:val answers + """ + question = { + "type": "list", + "name": group_id, + "message": group_obj.get("title", group_id), + "choices": ["Continue >>", PyInquirer.Separator()], + } + + for param_id, param in group_obj["properties"].items(): + if not param.get("hidden", False) or self.show_hidden: + question["choices"].append(param_id) + + # Skip if all questions hidden + if len(question["choices"]) == 2: + return {} + + while_break = False + answers = {} + while not while_break: + self.print_param_header(group_id, group_obj) + answer = PyInquirer.prompt([question]) + # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + if answer == {}: + raise KeyboardInterrupt + if answer[group_id] == "Continue >>": + while_break = True + # Check if there are any required parameters that don't have answers + for p_required in group_obj.get("required", []): + req_default = self.schema_obj.input_params.get(p_required, "") + req_answer = answers.get(p_required, "") + if req_default == "" and req_answer == "": + log.error("'{}' is required.".format(p_required)) + while_break = False + else: + param_id = answer[group_id] + is_required = param_id in group_obj.get("required", []) + answers.update(self.prompt_param(param_id, group_obj["properties"][param_id], is_required, answers)) + + return answers + + def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_help=True): + """Convert a JSONSchema param to a PyInquirer question Args: - parameters (list): Collection of parameter objects. + param_id: Parameter ID (string) + param_obj: JSON Schema keys (dict) + answers: Optional preexisting answers (dict) + print_help: If description and help_text should be printed (bool) Returns: - dict: Parameter objects grouped by the `group` property. + Single PyInquirer dict, to be appended to questions list """ - for param in self.parameters: - if param.group not in self.grouped_parameters.keys(): - self.grouped_parameters[param.group] = [] - self.grouped_parameters[param.group].append(param) - - def prompt_param_flags(self): - """ Prompts the user for parameter input values and validates them. """ - for group_label, params in self.grouped_parameters.items(): - click.echo("\n\n{}{}".format( - click.style('Parameter group: ', bold=True, underline=True), - click.style(group_label, bold=True, underline=True, fg='red') - )) - use_defaults = click.confirm( - "Do you want to change the group's defaults? "+click.style('[y/N]', fg='green'), - default=False, show_default=False) - if not use_defaults: - continue - for parameter in params: - # Skip this option if the render mode is none - value_is_valid = parameter.render == 'none' - first_attempt = True - while not value_is_valid: - # Start building the string to show to the user - label and usage - plines = [''] - if parameter.label: - plines.append(click.style(parameter.label, bold=True)) - if parameter.usage: - plines.append(click.style(parameter.usage)) - - # Add the choices / range if applicable - if parameter.choices: - rc = 'Choices' if parameter.type == 'string' else 'Range' - choices_string = ", ".join([click.style(x, fg='yellow') for x in parameter.choices if x != '']) - plines.append('{}: {}'.format(rc, choices_string)) - - # Reset the choice display if boolean - if parameter.type == "boolean": - pdef_val = 'Y/n' if parameter.default_value else 'y/N' - else: - pdef_val = parameter.default_value + if answers is None: + answers = {} + + question = {"type": "input", "name": param_id, "message": param_id} + + # Print the name, description & help text + if print_help: + nice_param_id = "--{}".format(param_id) if not param_id.startswith("-") else param_id + self.print_param_header(nice_param_id, param_obj) + + if param_obj.get("type") == "boolean": + question["type"] = "list" + question["choices"] = ["True", "False"] + question["default"] = "False" + + # Start with the default from the param object + if "default" in param_obj: + # Boolean default is cast back to a string later - this just normalises all inputs + if param_obj["type"] == "boolean" and type(param_obj["default"]) is str: + question["default"] = param_obj["default"].lower() == "true" + else: + question["default"] = param_obj["default"] - # Final line to print - command and default - if pdef_val == '': - flag_default = '' - else: - flag_default = click.style(' [{}]'.format(pdef_val), fg='green') - flag_prompt = click.style(' --{}'.format(parameter.name), fg='blue') + flag_default + # Overwrite default with parsed schema, includes --params-in etc + if self.schema_obj is not None and param_id in self.schema_obj.input_params: + if param_obj["type"] == "boolean" and type(self.schema_obj.input_params[param_id]) is str: + question["default"] = "true" == self.schema_obj.input_params[param_id].lower() + else: + question["default"] = self.schema_obj.input_params[param_id] + # Overwrite default if already had an answer + if param_id in answers: + question["default"] = answers[param_id] - # Only show this final prompt if we're trying again - if first_attempt: - plines.append(flag_prompt) - else: - plines = [flag_prompt] - first_attempt = False - - # Use click.confirm if a boolean for default input handling - if parameter.type == "boolean": - parameter.value = click.confirm("\n".join(plines), - default=parameter.default_value, show_default=False) - # Use click.prompt if anything else - else: - parameter.value = click.prompt("\n".join(plines), - default=parameter.default_value, show_default=False) - - # Set input parameter types - try: - if parameter.type == "integer": - parameter.value = int(parameter.value) - elif parameter.type == "decimal": - parameter.value = float(parameter.value) - elif parameter.type == "string": - parameter.value = str(parameter.value) - except ValueError as e: - logging.error("Could not set variable type: {}".format(e)) - - # Validate the input - try: - parameter.validate() - except Exception as e: - click.secho("\nERROR: {}".format(e), fg='red') - click.secho("Please try again:") - continue - else: - value_is_valid = True + # Coerce default to a string + if "default" in question: + question["default"] = str(question["default"]) + + if param_obj.get("type") == "boolean": + # Filter returned value + def filter_boolean(val): + if isinstance(val, bool): + return val + return val.lower() == "true" + + question["filter"] = filter_boolean + + if param_obj.get("type") == "number": + # Validate number type + def validate_number(val): + try: + if val.strip() == "": + return True + float(val) + except ValueError: + return "Must be a number" + else: + return True + + question["validate"] = validate_number + + # Filter returned value + def filter_number(val): + if val.strip() == "": + return "" + return float(val) + + question["filter"] = filter_number + + if param_obj.get("type") == "integer": + # Validate integer type + def validate_integer(val): + try: + if val.strip() == "": + return True + assert int(val) == float(val) + except (AssertionError, ValueError): + return "Must be an integer" + else: + return True + + question["validate"] = validate_integer + + # Filter returned value + def filter_integer(val): + if val.strip() == "": + return "" + return int(val) + + question["filter"] = filter_integer + + if param_obj.get("type") == "range": + # Validate range type + def validate_range(val): + try: + if val.strip() == "": + return True + fval = float(val) + if "minimum" in param_obj and fval < float(param_obj["minimum"]): + return "Must be greater than or equal to {}".format(param_obj["minimum"]) + if "maximum" in param_obj and fval > float(param_obj["maximum"]): + return "Must be less than or equal to {}".format(param_obj["maximum"]) + return True + except ValueError: + return "Must be a number" + + question["validate"] = validate_range + + # Filter returned value + def filter_range(val): + if val.strip() == "": + return "" + return float(val) + + question["filter"] = filter_range + + if "enum" in param_obj: + # Use a selection list instead of free text input + question["type"] = "list" + question["choices"] = param_obj["enum"] + + # Validate enum from schema + def validate_enum(val): + if val == "": + return True + if val in param_obj["enum"]: + return True + return "Must be one of: {}".format(", ".join(param_obj["enum"])) + + question["validate"] = validate_enum + + # Validate pattern from schema + if "pattern" in param_obj: + + def validate_pattern(val): + if val == "": + return True + if re.search(param_obj["pattern"], val) is not None: + return True + return "Must match pattern: {}".format(param_obj["pattern"]) + + question["validate"] = validate_pattern + + # WORKAROUND - PyInquirer <1.0.3 cannot have a default position in a list + # For now, move the default option to the top. + # TODO: Delete this code when PyInquirer >=1.0.3 is released. + if question["type"] == "list" and "default" in question: + try: + question["choices"].remove(question["default"]) + question["choices"].insert(0, question["default"]) + except ValueError: + log.warning( + "Default value `{}` not found in list of choices: {}".format( + question["default"], ", ".join(question["choices"]) + ) + ) + ### End of workaround code + + return question + + def print_param_header(self, param_id, param_obj): + if "description" not in param_obj and "help_text" not in param_obj: + return + console = Console() + console.print("\n") + console.print(param_obj.get("title", param_id), style="bold") + if "description" in param_obj: + md = Markdown(param_obj["description"]) + console.print(md) + if "help_text" in param_obj: + help_md = Markdown(param_obj["help_text"].strip()) + console.print(help_md, style="dim") + console.print("\n") + + def strip_default_params(self): + """ Strip parameters if they have not changed from the default """ + + # Schema defaults + for param_id, val in self.schema_obj.schema_defaults.items(): + if self.schema_obj.input_params.get(param_id) == val: + del self.schema_obj.input_params[param_id] + + # Nextflow flag defaults + for param_id, val in self.nxf_flag_schema["coreNextflow"]["properties"].items(): + if param_id in self.nxf_flags and self.nxf_flags[param_id] == val.get("default"): + del self.nxf_flags[param_id] def build_command(self): """ Build the nextflow run command based on what we know """ + + # Core nextflow options for flag, val in self.nxf_flags.items(): # Boolean flags like -resume - if isinstance(val, bool): - if val: - self.nextflow_cmd = "{} {}".format(self.nextflow_cmd, flag) - else: - logging.warning("TODO: Can't set false boolean flags currently.") + if isinstance(val, bool) and val: + self.nextflow_cmd += " {}".format(flag) # String values - else: - self.nextflow_cmd = '{} {} "{}"'.format(self.nextflow_cmd, flag, val.replace('"', '\\"')) + elif not isinstance(val, bool): + self.nextflow_cmd += ' {} "{}"'.format(flag, val.replace('"', '\\"')) - # Write the user selection to a file and run nextflow with that - if self.use_params_file: - path = self.create_nfx_params_file() - if path is not None: - self.nextflow_cmd = '{} {} "{}"'.format(self.nextflow_cmd, "-params-file", path) - self.write_params_as_full_json() + # Pipeline parameters + if len(self.schema_obj.input_params) > 0: - # Call nextflow with a list of command line flags - else: - for param, val in self.params_user.items(): - # Boolean flags like --saveTrimmed - if isinstance(val, bool): - if val: - self.nextflow_cmd = "{} --{}".format(self.nextflow_cmd, param) + # Write the user selection to a file and run nextflow with that + if self.use_params_file: + with open(self.params_out, "w") as fp: + json.dump(self.schema_obj.input_params, fp, indent=4) + self.nextflow_cmd += ' {} "{}"'.format("-params-file", os.path.relpath(self.params_out)) + + # Call nextflow with a list of command line flags + else: + for param, val in self.schema_obj.input_params.items(): + # Boolean flags like --saveTrimmed + if isinstance(val, bool) and val: + self.nextflow_cmd += " --{}".format(param) + # everything else else: - logging.error("Can't set false boolean flags.") - # everything else - else: - self.nextflow_cmd = '{} --{} "{}"'.format(self.nextflow_cmd, param, val.replace('"', '\\"')) - - def create_nfx_params_file(self): - working_dir = os.getcwd() - output_file = os.path.join(working_dir, "nfx-params.json") - json_string = nf_core.workflow.parameters.Parameters.in_nextflow_json(self.parameters, indent=4) - if json_string == '{}': - return None - with open(output_file, "w") as fp: - fp.write(json_string) - return output_file - - def write_params_as_full_json(self, outdir = os.getcwd()): - output_file = os.path.join(outdir, "full-params.json") - json_string = nf_core.workflow.parameters.Parameters.in_full_json(self.parameters, indent=4) - with open(output_file, "w") as fp: - fp.write(json_string) - return output_file + self.nextflow_cmd += ' --{} "{}"'.format(param, str(val).replace('"', '\\"')) def launch_workflow(self): """ Launch nextflow if required """ - click.secho("\n\nNextflow command:", bold=True, underline=True) - click.secho(" {}\n\n".format(self.nextflow_cmd), fg='magenta') + log.info("[bold underline]Nextflow command:[/]\n[magenta]{}\n\n".format(self.nextflow_cmd)) - if click.confirm( - 'Do you want to run this command now? '+click.style('[y/N]', fg='green'), - default=False, - show_default=False - ): - logging.info("Launching workflow!") + if Confirm.ask("Do you want to run this command now? "): + log.info("Launching workflow! :rocket:") subprocess.call(self.nextflow_cmd, shell=True) diff --git a/nf_core/licences.py b/nf_core/licences.py index 6e82465d3c..1637367427 100644 --- a/nf_core/licences.py +++ b/nf_core/licences.py @@ -5,14 +5,19 @@ import logging import json +import os import re import requests import sys import tabulate import yaml +import rich.console +import rich.table import nf_core.lint +log = logging.getLogger(__name__) + class WorkflowLicences(object): """A nf-core workflow licenses collection. @@ -26,49 +31,74 @@ class WorkflowLicences(object): pipeline (str): An existing nf-core pipeline name, like `nf-core/hlatyping` or short `hlatyping`. """ + def __init__(self, pipeline): self.pipeline = pipeline - if self.pipeline.startswith('nf-core/'): + self.conda_config = None + if self.pipeline.startswith("nf-core/"): self.pipeline = self.pipeline[8:] + self.conda_packages = {} self.conda_package_licences = {} + self.as_json = False + + def run_licences(self): + """ + Run the nf-core licences action + """ + self.get_environment_file() + self.fetch_conda_licences() + return self.print_licences() + + def get_environment_file(self): + """Get the conda environment file for the pipeline + """ + if os.path.exists(self.pipeline): + env_filename = os.path.join(self.pipeline, "environment.yml") + if not os.path.exists(self.pipeline): + raise LookupError("Pipeline {} exists, but no environment.yml file found".format(self.pipeline)) + with open(env_filename, "r") as fh: + self.conda_config = yaml.safe_load(fh) + else: + env_url = "https://raw.githubusercontent.com/nf-core/{}/master/environment.yml".format(self.pipeline) + log.debug("Fetching environment.yml file: {}".format(env_url)) + response = requests.get(env_url) + # Check that the pipeline exists + if response.status_code == 404: + raise LookupError("Couldn't find pipeline nf-core/{}".format(self.pipeline)) + self.conda_config = yaml.safe_load(response.text) def fetch_conda_licences(self): """Fetch package licences from Anaconda and PyPi. """ - env_url = 'https://raw.githubusercontent.com/nf-core/{}/master/environment.yml'.format(self.pipeline) - response = requests.get(env_url) - - # Check that the pipeline exists - if response.status_code == 404: - logging.error("Couldn't find pipeline nf-core/{}".format(self.pipeline)) - raise LookupError("Couldn't find pipeline nf-core/{}".format(self.pipeline)) lint_obj = nf_core.lint.PipelineLint(self.pipeline) - lint_obj.conda_config = yaml.safe_load(response.text) + lint_obj.conda_config = self.conda_config # Check conda dependency list - for dep in lint_obj.conda_config.get('dependencies', []): + deps = lint_obj.conda_config.get("dependencies", []) + log.info("Fetching licence information for {} tools".format(len(deps))) + for dep in deps: try: if isinstance(dep, str): lint_obj.check_anaconda_package(dep) elif isinstance(dep, dict): lint_obj.check_pip_package(dep) except ValueError: - logging.error("Couldn't get licence information for {}".format(dep)) + log.error("Couldn't get licence information for {}".format(dep)) for dep, data in lint_obj.conda_package_info.items(): try: - depname, depver = dep.split('=', 1) + depname, depver = dep.split("=", 1) licences = set() # Licence for each version - for f in data['files']: - if not depver or depver == f.get('version'): + for f in data["files"]: + if not depver or depver == f.get("version"): try: - licences.add(f['attrs']['license']) + licences.add(f["attrs"]["license"]) except KeyError: pass # Main licence field - if len(list(licences)) == 0 and isinstance(data['license'], str): - licences.add(data['license']) + if len(list(licences)) == 0 and isinstance(data["license"], str): + licences.add(data["license"]) self.conda_package_licences[dep] = self.clean_licence_names(list(licences)) except KeyError: pass @@ -85,39 +115,40 @@ def clean_licence_names(self, licences): """ clean_licences = [] for l in licences: - l = re.sub(r'GNU General Public License v\d \(([^\)]+)\)', r'\1', l) - l = re.sub(r'GNU GENERAL PUBLIC LICENSE', 'GPL', l, flags=re.IGNORECASE) - l = l.replace('GPL-', 'GPLv') - l = re.sub(r'GPL(\d)', r'GPLv\1', l) - l = re.sub(r'GPL \(([^\)]+)\)', r'GPL \1', l) - l = re.sub(r'GPL\s*v', 'GPLv', l) - l = re.sub(r'\s*(>=?)\s*(\d)', r' \1\2', l) + l = re.sub(r"GNU General Public License v\d \(([^\)]+)\)", r"\1", l) + l = re.sub(r"GNU GENERAL PUBLIC LICENSE", "GPL", l, flags=re.IGNORECASE) + l = l.replace("GPL-", "GPLv") + l = re.sub(r"GPL(\d)", r"GPLv\1", l) + l = re.sub(r"GPL \(([^\)]+)\)", r"GPL \1", l) + l = re.sub(r"GPL\s*v", "GPLv", l) + l = re.sub(r"\s*(>=?)\s*(\d)", r" \1\2", l) clean_licences.append(l) return clean_licences - def print_licences(self, as_json=False): + def print_licences(self): """Prints the fetched license information. Args: as_json (boolean): Prints the information in JSON. Defaults to False. """ - logging.info("""Warning: This tool only prints licence information for the software tools packaged using conda. - The pipeline may use other software and dependencies not described here. """) + log.info("Warning: This tool only prints licence information for the software tools packaged using conda.") + log.info("The pipeline may use other software and dependencies not described here. ") - if as_json: - print(json.dumps(self.conda_package_licences, indent=4)) + if self.as_json: + return json.dumps(self.conda_package_licences, indent=4) else: + table = rich.table.Table("Package Name", "Version", "Licence") licence_list = [] for dep, licences in self.conda_package_licences.items(): - depname, depver = dep.split('=', 1) + depname, depver = dep.split("=", 1) try: - depname = depname.split('::')[1] + depname = depname.split("::")[1] except IndexError: pass - licence_list.append([depname, depver, ', '.join(licences)]) + licence_list.append([depname, depver, ", ".join(licences)]) # Sort by licence, then package name licence_list = sorted(sorted(licence_list), key=lambda x: x[2]) - # Print summary table - print("", file=sys.stderr) - print(tabulate.tabulate(licence_list, headers=['Package Name', 'Version', 'Licence'])) - print("", file=sys.stderr) + # Add table rows + for lic in licence_list: + table.add_row(*lic) + return table diff --git a/nf_core/lint.py b/nf_core/lint.py index 167c3335df..9a77c991a2 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -5,17 +5,31 @@ the nf-core community guidelines. """ -import logging +from rich.console import Console +from rich.markdown import Markdown +from rich.table import Table +import datetime +import fnmatch +import git import io +import json +import logging import os import re -import shlex +import requests +import rich +import rich.progress +import subprocess +import textwrap import click import requests import yaml import nf_core.utils +import nf_core.schema + +log = logging.getLogger(__name__) # Set up local caching for requests to speed up remote queries nf_core.utils.setup_requests_cachedir() @@ -25,7 +39,7 @@ logging.getLogger("urllib3").setLevel(logging.WARNING) -def run_linting(pipeline_dir, release_mode=False): +def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): """Runs all nf-core linting checks on a given Nextflow pipeline project in either `release` mode or `normal` mode (default). Returns an object of type :class:`PipelineLint` after finished. @@ -46,19 +60,31 @@ def run_linting(pipeline_dir, release_mode=False): try: lint_obj.lint_pipeline(release_mode) except AssertionError as e: - logging.critical("Critical error: {}".format(e)) - logging.info("Stopping tests...") + log.critical("Critical error: {}".format(e)) + log.info("Stopping tests...") return lint_obj # Print the results lint_obj.print_results() + # Save results to Markdown file + if md_fn is not None: + log.info("Writing lint results to {}".format(md_fn)) + markdown = lint_obj.get_results_md() + with open(md_fn, "w") as fh: + fh.write(markdown) + + # Save results to JSON file + if json_fn is not None: + lint_obj.save_json_results(json_fn) + + # Try to post comment to a GitHub PR + lint_obj.github_comment() + # Exit code if len(lint_obj.failed) > 0: - logging.error( - "Sorry, some tests failed - exiting with a non-zero error code...{}\n\n" - .format("\n\tReminder: Lint tests were run in --release mode." if release_mode else '') - ) + if release_mode: + log.info("Reminder: Lint tests were run in --release mode.") return lint_obj @@ -118,10 +144,12 @@ class PipelineLint(object): params.clusterOptions = false ... """ + def __init__(self, path): """ Initialise linting object """ self.release_mode = False self.path = path + self.git_sha = None self.files = [] self.config = {} self.pipeline_name = None @@ -129,10 +157,21 @@ def __init__(self, path): self.dockerfile = [] self.conda_config = {} self.conda_package_info = {} + self.schema_obj = None self.passed = [] self.warned = [] self.failed = [] + try: + repo = git.Repo(self.path) + self.git_sha = repo.head.object.hexsha + except: + pass + + # Overwrite if we have the last commit from the PR - otherwise we get a merge commit hash + if os.environ.get("GITHUB_PR_COMMIT", "") != "": + self.git_sha = os.environ["GITHUB_PR_COMMIT"] + def lint_pipeline(self, release_mode=False): """Main linting function. @@ -161,30 +200,47 @@ def lint_pipeline(self, release_mode=False): Raises: If a critical problem is found, an ``AssertionError`` is raised. """ + log.info("Testing pipeline: [magenta]{}".format(self.path)) + if self.release_mode: + log.info("Including --release mode tests") check_functions = [ - 'check_files_exist', - 'check_licence', - 'check_docker', - 'check_nextflow_config', - 'check_actions_branch_protection', - 'check_actions_ci', - 'check_actions_lint', - 'check_readme', - 'check_conda_env_yaml', - 'check_conda_dockerfile', - 'check_pipeline_todos', - 'check_pipeline_name' + "check_files_exist", + "check_licence", + "check_docker", + "check_nextflow_config", + "check_actions_branch_protection", + "check_actions_ci", + "check_actions_lint", + "check_actions_awstest", + "check_actions_awsfulltest", + "check_readme", + "check_conda_env_yaml", + "check_conda_dockerfile", + "check_pipeline_todos", + "check_pipeline_name", + "check_cookiecutter_strings", + "check_schema_lint", + "check_schema_params", ] if release_mode: self.release_mode = True - check_functions.extend([ - 'check_version_consistency' - ]) - with click.progressbar(check_functions, label='Running pipeline tests', item_show_func=repr) as fun_names: - for fun_name in fun_names: + check_functions.extend(["check_version_consistency"]) + + progress = rich.progress.Progress( + "[bold blue]{task.description}", + rich.progress.BarColumn(bar_width=None), + "[magenta]{task.completed} of {task.total}[reset] » [bold yellow]{task.fields[func_name]}", + transient=True, + ) + with progress: + lint_progress = progress.add_task( + "Running lint checks", total=len(check_functions), func_name=check_functions[0] + ) + for fun_name in check_functions: + progress.update(lint_progress, advance=1, func_name=fun_name) getattr(self, fun_name)() if len(self.failed) > 0: - logging.error("Found test failures in '{}', halting lint run.".format(fun_name)) + log.error("Found test failures in `{}`, halting lint run.".format(fun_name)) break def check_files_exist(self): @@ -195,22 +251,25 @@ def check_files_exist(self): Files that **must** be present:: 'nextflow.config', + 'nextflow_schema.json', 'Dockerfile', ['LICENSE', 'LICENSE.md', 'LICENCE', 'LICENCE.md'], # NB: British / American spelling 'README.md', 'CHANGELOG.md', 'docs/README.md', 'docs/output.md', - 'docs/usage.md' + 'docs/usage.md', + '.github/workflows/branch.yml', + '.github/workflows/ci.yml', + '.github/workflows/linting.yml' Files that *should* be present:: 'main.nf', 'environment.yml', 'conf/base.config', - '.github/workflows/branch.yml', - '.github/workflows/ci.yml', - '.github/workfows/linting.yml' + '.github/workflows/awstest.yml', + '.github/workflows/awsfulltest.yml' Files that *must not* be present:: @@ -227,82 +286,83 @@ def check_files_exist(self): # NB: Should all be files, not directories # List of lists. Passes if any of the files in the sublist are found. files_fail = [ - ['nextflow.config'], - ['Dockerfile'], - ['LICENSE', 'LICENSE.md', 'LICENCE', 'LICENCE.md'], # NB: British / American spelling - ['README.md'], - ['CHANGELOG.md'], - [os.path.join('docs','README.md')], - [os.path.join('docs','output.md')], - [os.path.join('docs','usage.md')], - [os.path.join('.github', 'workflows', 'branch.yml')], - [os.path.join('.github', 'workflows','ci.yml')], - [os.path.join('.github', 'workflows', 'linting.yml')] + ["nextflow.config"], + ["nextflow_schema.json"], + ["Dockerfile"], + ["LICENSE", "LICENSE.md", "LICENCE", "LICENCE.md"], # NB: British / American spelling + ["README.md"], + ["CHANGELOG.md"], + [os.path.join("docs", "README.md")], + [os.path.join("docs", "output.md")], + [os.path.join("docs", "usage.md")], + [os.path.join(".github", "workflows", "branch.yml")], + [os.path.join(".github", "workflows", "ci.yml")], + [os.path.join(".github", "workflows", "linting.yml")], ] files_warn = [ - ['main.nf'], - ['environment.yml'], - [os.path.join('conf','base.config')] + ["main.nf"], + ["environment.yml"], + [os.path.join("conf", "base.config")], + [os.path.join(".github", "workflows", "awstest.yml")], + [os.path.join(".github", "workflows", "awsfulltest.yml")], ] # List of strings. Dails / warns if any of the strings exist. - files_fail_ifexists = [ - 'Singularity' - ] - files_warn_ifexists = [ - '.travis.yml' - ] + files_fail_ifexists = ["Singularity", "parameters.settings.json"] + files_warn_ifexists = [".travis.yml"] def pf(file_path): return os.path.join(self.path, file_path) # First - critical files. Check that this is actually a Nextflow pipeline - if not os.path.isfile(pf('nextflow.config')) and not os.path.isfile(pf('main.nf')): - raise AssertionError('Neither nextflow.config or main.nf found! Is this a Nextflow pipeline?') + if not os.path.isfile(pf("nextflow.config")) and not os.path.isfile(pf("main.nf")): + self.failed.append((1, "File not found: nextflow.config or main.nf")) + raise AssertionError("Neither nextflow.config or main.nf found! Is this a Nextflow pipeline?") # Files that cause an error if they don't exist for files in files_fail: if any([os.path.isfile(pf(f)) for f in files]): - self.passed.append((1, "File found: {}".format(self._bold_list_items(files)))) + self.passed.append((1, "File found: {}".format(self._wrap_quotes(files)))) self.files.extend(files) else: - self.failed.append((1, "File not found: {}".format(self._bold_list_items(files)))) + self.failed.append((1, "File not found: {}".format(self._wrap_quotes(files)))) # Files that cause a warning if they don't exist for files in files_warn: if any([os.path.isfile(pf(f)) for f in files]): - self.passed.append((1, "File found: {}".format(self._bold_list_items(files)))) + self.passed.append((1, "File found: {}".format(self._wrap_quotes(files)))) self.files.extend(files) else: - self.warned.append((1, "File not found: {}".format(self._bold_list_items(files)))) + self.warned.append((1, "File not found: {}".format(self._wrap_quotes(files)))) # Files that cause an error if they exist for file in files_fail_ifexists: if os.path.isfile(pf(file)): - self.failed.append((1, "File must be removed: {}".format(self._bold_list_items(file)))) + self.failed.append((1, "File must be removed: {}".format(self._wrap_quotes(file)))) else: - self.passed.append((1, "File not found check: {}".format(self._bold_list_items(file)))) + self.passed.append((1, "File not found check: {}".format(self._wrap_quotes(file)))) # Files that cause a warning if they exist for file in files_warn_ifexists: if os.path.isfile(pf(file)): - self.warned.append((1, "File should be removed: {}".format(self._bold_list_items(file)))) + self.warned.append((1, "File should be removed: {}".format(self._wrap_quotes(file)))) else: - self.passed.append((1, "File not found check: {}".format(self._bold_list_items(file)))) + self.passed.append((1, "File not found check: {}".format(self._wrap_quotes(file)))) # Load and parse files for later - if 'environment.yml' in self.files: - with open(os.path.join(self.path, 'environment.yml'), 'r') as fh: + if "environment.yml" in self.files: + with open(os.path.join(self.path, "environment.yml"), "r") as fh: self.conda_config = yaml.safe_load(fh) def check_docker(self): """Checks that Dockerfile contains the string ``FROM``.""" fn = os.path.join(self.path, "Dockerfile") content = "" - with open(fn, 'r') as fh: content = fh.read() + with open(fn, "r") as fh: + content = fh.read() # Implicitly also checks if empty. - if 'FROM ' in content: + if "FROM " in content: self.passed.append((2, "Dockerfile check passed")) self.dockerfile = [line.strip() for line in content.splitlines()] return @@ -317,11 +377,12 @@ def check_licence(self): * licence contains the string *without restriction* * licence doesn't have any placeholder variables """ - for l in ['LICENSE', 'LICENSE.md', 'LICENCE', 'LICENCE.md']: + for l in ["LICENSE", "LICENSE.md", "LICENCE", "LICENCE.md"]: fn = os.path.join(self.path, l) if os.path.isfile(fn): content = "" - with open(fn, 'r') as fh: content = fh.read() + with open(fn, "r") as fh: + content = fh.read() # needs at least copyright, permission, notice and "as-is" lines nl = content.count("\n") @@ -333,7 +394,7 @@ def check_licence(self): # license. Most variations actually don't contain the # string MIT Searching for 'without restriction' # instead (a crutch). - if not 'without restriction' in content: + if not "without restriction" in content: self.failed.append((3, "Licence file did not look like MIT: {}".format(fn))) return @@ -341,7 +402,7 @@ def check_licence(self): # - https://choosealicense.com/licenses/mit/ # - https://opensource.org/licenses/MIT # - https://en.wikipedia.org/wiki/MIT_License - placeholders = {'[year]', '[fullname]', '', '', '', ''} + placeholders = {"[year]", "[fullname]", "", "", "", ""} if any([ph in content for ph in placeholders]): self.failed.append((3, "Licence file contains placeholders: {}".format(fn))) return @@ -364,38 +425,37 @@ def check_nextflow_config(self): # Fail tests if these are missing config_fail = [ - ['manifest.name'], - ['manifest.nextflowVersion'], - ['manifest.description'], - ['manifest.version'], - ['manifest.homePage'], - ['timeline.enabled'], - ['trace.enabled'], - ['report.enabled'], - ['dag.enabled'], - ['process.cpus'], - ['process.memory'], - ['process.time'], - ['params.outdir'] + ["manifest.name"], + ["manifest.nextflowVersion"], + ["manifest.description"], + ["manifest.version"], + ["manifest.homePage"], + ["timeline.enabled"], + ["trace.enabled"], + ["report.enabled"], + ["dag.enabled"], + ["process.cpus"], + ["process.memory"], + ["process.time"], + ["params.outdir"], + ["params.input"], ] # Throw a warning if these are missing config_warn = [ - ['manifest.mainScript'], - ['timeline.file'], - ['trace.file'], - ['report.file'], - ['dag.file'], - ['params.reads','params.input','params.design'], - ['process.container'], - ['params.single_end'] + ["manifest.mainScript"], + ["timeline.file"], + ["trace.file"], + ["report.file"], + ["dag.file"], + ["process.container"], ] # Old depreciated vars - fail if present config_fail_ifdefined = [ - 'params.version', - 'params.nf_required_version', - 'params.container', - 'params.singleEnd', - 'params.igenomesIgnore' + "params.version", + "params.nf_required_version", + "params.container", + "params.singleEnd", + "params.igenomesIgnore", ] # Get the nextflow config for this pipeline @@ -403,226 +463,396 @@ def check_nextflow_config(self): for cfs in config_fail: for cf in cfs: if cf in self.config.keys(): - self.passed.append((4, "Config variable found: {}".format(self._bold_list_items(cf)))) + self.passed.append((4, "Config variable found: {}".format(self._wrap_quotes(cf)))) break else: - self.failed.append((4, "Config variable not found: {}".format(self._bold_list_items(cfs)))) + self.failed.append((4, "Config variable not found: {}".format(self._wrap_quotes(cfs)))) for cfs in config_warn: for cf in cfs: if cf in self.config.keys(): - self.passed.append((4, "Config variable found: {}".format(self._bold_list_items(cf)))) + self.passed.append((4, "Config variable found: {}".format(self._wrap_quotes(cf)))) break else: - self.warned.append((4, "Config variable not found: {}".format(self._bold_list_items(cfs)))) + self.warned.append((4, "Config variable not found: {}".format(self._wrap_quotes(cfs)))) for cf in config_fail_ifdefined: if cf not in self.config.keys(): - self.passed.append((4, "Config variable (correctly) not found: {}".format(self._bold_list_items(cf)))) + self.passed.append((4, "Config variable (correctly) not found: {}".format(self._wrap_quotes(cf)))) else: - self.failed.append((4, "Config variable (incorrectly) found: {}".format(self._bold_list_items(cf)))) + self.failed.append((4, "Config variable (incorrectly) found: {}".format(self._wrap_quotes(cf)))) # Check and warn if the process configuration is done with deprecated syntax - process_with_deprecated_syntax = list(set([re.search('^(process\.\$.*?)\.+.*$', ck).group(1) for ck in self.config.keys() if re.match(r'^(process\.\$.*?)\.+.*$', ck)])) + process_with_deprecated_syntax = list( + set( + [ + re.search(r"^(process\.\$.*?)\.+.*$", ck).group(1) + for ck in self.config.keys() + if re.match(r"^(process\.\$.*?)\.+.*$", ck) + ] + ) + ) for pd in process_with_deprecated_syntax: self.warned.append((4, "Process configuration is done with deprecated_syntax: {}".format(pd))) # Check the variables that should be set to 'true' - for k in ['timeline.enabled', 'report.enabled', 'trace.enabled', 'dag.enabled']: - if self.config.get(k) == 'true': - self.passed.append((4, "Config variable '{}' had correct value: {}".format(k, self.config.get(k)))) + for k in ["timeline.enabled", "report.enabled", "trace.enabled", "dag.enabled"]: + if self.config.get(k) == "true": + self.passed.append((4, "Config `{}` had correct value: `{}`".format(k, self.config.get(k)))) else: - self.failed.append((4, "Config variable '{}' did not have correct value: {}".format(k, self.config.get(k)))) + self.failed.append((4, "Config `{}` did not have correct value: `{}`".format(k, self.config.get(k)))) # Check that the pipeline name starts with nf-core try: - assert self.config.get('manifest.name', '').strip('\'"').startswith('nf-core/') + assert self.config.get("manifest.name", "").strip("'\"").startswith("nf-core/") except (AssertionError, IndexError): - self.failed.append((4, "Config variable 'manifest.name' did not begin with nf-core/:\n {}".format(self.config.get('manifest.name', '').strip('\'"')))) + self.failed.append( + ( + 4, + "Config `manifest.name` did not begin with `nf-core/`:\n {}".format( + self.config.get("manifest.name", "").strip("'\"") + ), + ) + ) else: - self.passed.append((4, "Config variable 'manifest.name' began with 'nf-core/'")) - self.pipeline_name = self.config.get('manifest.name', '').strip("'").replace('nf-core/', '') + self.passed.append((4, "Config `manifest.name` began with `nf-core/`")) + self.pipeline_name = self.config.get("manifest.name", "").strip("'").replace("nf-core/", "") # Check that the homePage is set to the GitHub URL try: - assert self.config.get('manifest.homePage', '').strip('\'"').startswith('https://github.com/nf-core/') + assert self.config.get("manifest.homePage", "").strip("'\"").startswith("https://github.com/nf-core/") except (AssertionError, IndexError): - self.failed.append((4, "Config variable 'manifest.homePage' did not begin with https://github.com/nf-core/:\n {}".format(self.config.get('manifest.homePage', '').strip('\'"')))) + self.failed.append( + ( + 4, + "Config variable `manifest.homePage` did not begin with https://github.com/nf-core/:\n {}".format( + self.config.get("manifest.homePage", "").strip("'\"") + ), + ) + ) else: - self.passed.append((4, "Config variable 'manifest.homePage' began with 'https://github.com/nf-core/'")) + self.passed.append((4, "Config variable `manifest.homePage` began with https://github.com/nf-core/")) # Check that the DAG filename ends in `.svg` - if 'dag.file' in self.config: - if self.config['dag.file'].strip('\'"').endswith('.svg'): - self.passed.append((4, "Config variable 'dag.file' ended with .svg")) + if "dag.file" in self.config: + if self.config["dag.file"].strip("'\"").endswith(".svg"): + self.passed.append((4, "Config `dag.file` ended with `.svg`")) else: - self.failed.append((4, "Config variable 'dag.file' did not end with .svg")) + self.failed.append((4, "Config `dag.file` did not end with `.svg`")) # Check that the minimum nextflowVersion is set properly - if 'manifest.nextflowVersion' in self.config: - if self.config.get('manifest.nextflowVersion', '').strip('"\'').lstrip('!').startswith('>='): - self.passed.append((4, "Config variable 'manifest.nextflowVersion' started with >= or !>=")) + if "manifest.nextflowVersion" in self.config: + if self.config.get("manifest.nextflowVersion", "").strip("\"'").lstrip("!").startswith(">="): + self.passed.append((4, "Config variable `manifest.nextflowVersion` started with >= or !>=")) # Save self.minNextflowVersion for convenience - nextflowVersionMatch = re.search(r'[0-9\.]+(-edge)?', self.config.get('manifest.nextflowVersion', '')) + nextflowVersionMatch = re.search(r"[0-9\.]+(-edge)?", self.config.get("manifest.nextflowVersion", "")) if nextflowVersionMatch: self.minNextflowVersion = nextflowVersionMatch.group(0) else: self.minNextflowVersion = None else: - self.failed.append((4, "Config variable 'manifest.nextflowVersion' did not start with '>=' or '!>=' : '{}'".format(self.config.get('manifest.nextflowVersion', '')).strip('"\''))) + self.failed.append( + ( + 4, + "Config `manifest.nextflowVersion` did not start with `>=` or `!>=` : `{}`".format( + self.config.get("manifest.nextflowVersion", "") + ).strip("\"'"), + ) + ) # Check that the process.container name is pulling the version tag or :dev - if self.config.get('process.container'): - container_name = '{}:{}'.format(self.config.get('manifest.name').replace('nf-core','nfcore').strip("'"), self.config.get('manifest.version', '').strip("'")) - if 'dev' in self.config.get('manifest.version', '') or not self.config.get('manifest.version'): - container_name = '{}:dev'.format(self.config.get('manifest.name').replace('nf-core','nfcore').strip("'")) + if self.config.get("process.container"): + container_name = "{}:{}".format( + self.config.get("manifest.name").replace("nf-core", "nfcore").strip("'"), + self.config.get("manifest.version", "").strip("'"), + ) + if "dev" in self.config.get("manifest.version", "") or not self.config.get("manifest.version"): + container_name = "{}:dev".format( + self.config.get("manifest.name").replace("nf-core", "nfcore").strip("'") + ) try: - assert self.config.get('process.container', '').strip("'") == container_name + assert self.config.get("process.container", "").strip("'") == container_name except AssertionError: if self.release_mode: - self.failed.append((4, "Config variable process.container looks wrong. Should be '{}' but is '{}'".format(container_name, self.config.get('process.container', '').strip("'")))) + self.failed.append( + ( + 4, + "Config `process.container` looks wrong. Should be `{}` but is `{}`".format( + container_name, self.config.get("process.container", "").strip("'") + ), + ) + ) else: - self.warned.append((4, "Config variable process.container looks wrong. Should be '{}' but is '{}'. Fix this before you make a release of your pipeline!".format(container_name, self.config.get('process.container', '').strip("'")))) + self.warned.append( + ( + 4, + "Config `process.container` looks wrong. Should be `{}` but is `{}`".format( + container_name, self.config.get("process.container", "").strip("'") + ), + ) + ) else: - self.passed.append((4, "Config variable process.container looks correct: '{}'".format(container_name))) + self.passed.append((4, "Config `process.container` looks correct: `{}`".format(container_name))) # Check that the pipeline version contains `dev` - if not self.release_mode and 'manifest.version' in self.config: - if self.config['manifest.version'].strip(' \'"').endswith('dev'): - self.passed.append((4, "Config variable manifest.version ends in 'dev': '{}'".format(self.config['manifest.version']))) + if not self.release_mode and "manifest.version" in self.config: + if self.config["manifest.version"].strip(" '\"").endswith("dev"): + self.passed.append( + (4, "Config `manifest.version` ends in `dev`: `{}`".format(self.config["manifest.version"])) + ) else: - self.warned.append((4, "Config variable manifest.version should end in 'dev': '{}'".format(self.config['manifest.version']))) - elif 'manifest.version' in self.config: - if 'dev' in self.config['manifest.version']: - self.failed.append((4, "Config variable manifest.version should not contain 'dev' for a release: '{}'".format(self.config['manifest.version']))) + self.warned.append( + (4, "Config `manifest.version` should end in `dev`: `{}`".format(self.config["manifest.version"]),) + ) + elif "manifest.version" in self.config: + if "dev" in self.config["manifest.version"]: + self.failed.append( + ( + 4, + "Config `manifest.version` should not contain `dev` for a release: `{}`".format( + self.config["manifest.version"] + ), + ) + ) else: - self.passed.append((4, "Config variable manifest.version does not contain 'dev' for release: '{}'".format(self.config['manifest.version']))) + self.passed.append( + ( + 4, + "Config `manifest.version` does not contain `dev` for release: `{}`".format( + self.config["manifest.version"] + ), + ) + ) def check_actions_branch_protection(self): """Checks that the GitHub Actions branch protection workflow is valid. Makes sure PRs can only come from nf-core dev or 'patch' of a fork. """ - fn = os.path.join(self.path, '.github', 'workflows', 'branch.yml') + fn = os.path.join(self.path, ".github", "workflows", "branch.yml") if os.path.isfile(fn): - with open(fn, 'r') as fh: + with open(fn, "r") as fh: branchwf = yaml.safe_load(fh) # Check that the action is turned on for PRs to master try: - assert('master' in branchwf[True]['pull_request']['branches']) + # Yaml 'on' parses as True - super weird + assert "master" in branchwf[True]["pull_request"]["branches"] except (AssertionError, KeyError): - self.failed.append((5, "GitHub Actions 'branch' workflow should be triggered for PRs to master: '{}'".format(fn))) + self.failed.append( + (5, "GitHub Actions 'branch' workflow should be triggered for PRs to master: `{}`".format(fn)) + ) else: - self.passed.append((5, "GitHub Actions 'branch' workflow is triggered for PRs to master: '{}'".format(fn))) + self.passed.append( + (5, "GitHub Actions 'branch' workflow is triggered for PRs to master: `{}`".format(fn)) + ) # Check that PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch - PRMasterCheck = "{{ [[ $(git remote get-url origin) == *nf-core/{} ]] && [[ ${{GITHUB_HEAD_REF}} = \"dev\" ]]; }} || [[ ${{GITHUB_HEAD_REF}} == \"patch\" ]]".format(self.pipeline_name.lower()) - steps = branchwf['jobs']['test']['steps'] - try: - steps = branchwf['jobs']['test']['steps'] - assert(any([PRMasterCheck in step.get('run', []) for step in steps])) - except (AssertionError, KeyError): - self.failed.append((5, "GitHub Actions 'branch' workflow should check that forks don't submit PRs to master: '{}'".format(fn))) + steps = branchwf.get("jobs", {}).get("test", {}).get("steps", []) + for step in steps: + has_name = step.get("name", "").strip() == "Check PRs" + has_if = step.get("if", "").strip() == "github.repository == 'nf-core/{}'".format( + self.pipeline_name.lower() + ) + # Don't use .format() as the squiggly brackets get ridiculous + has_run = step.get( + "run", "" + ).strip() == '{ [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/PIPELINENAME ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]'.replace( + "PIPELINENAME", self.pipeline_name.lower() + ) + if has_name and has_if and has_run: + self.passed.append((5, "GitHub Actions 'branch' workflow looks good: `{}`".format(fn),)) + break else: - self.passed.append((5, "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format(fn))) + self.failed.append( + (5, "Couldn't find GitHub Actions 'branch' check for PRs to master: `{}`".format(fn),) + ) def check_actions_ci(self): """Checks that the GitHub Actions CI workflow is valid Makes sure tests run with the required nextflow version. """ - fn = os.path.join(self.path, '.github', 'workflows', 'ci.yml') + fn = os.path.join(self.path, ".github", "workflows", "ci.yml") if os.path.isfile(fn): - with open(fn, 'r') as fh: + with open(fn, "r") as fh: ciwf = yaml.safe_load(fh) - # Check that the action is turned on for push and pull requests + # Check that the action is turned on for the correct events try: - assert('push' in ciwf[True]) - assert('pull_request' in ciwf[True]) + expected = {"push": {"branches": ["dev"]}, "pull_request": None, "release": {"types": ["published"]}} + # NB: YAML dict key 'on' is evaluated to a Python dict key True + assert ciwf[True] == expected except (AssertionError, KeyError, TypeError): - self.failed.append((5, "GitHub Actions CI workflow must be triggered on PR and push: '{}'".format(fn))) + self.failed.append((5, "GitHub Actions CI is not triggered on expected events: `{}`".format(fn),)) else: - self.passed.append((5, "GitHub Actions CI workflow is triggered on PR and push: '{}'".format(fn))) + self.passed.append((5, "GitHub Actions CI is triggered on expected events: `{}`".format(fn))) # Check that we're pulling the right docker image and tagging it properly - if self.config.get('process.container', ''): - docker_notag = re.sub(r':(?:[\.\d]+|dev)$', '', self.config.get('process.container', '').strip('"\'')) - docker_withtag = self.config.get('process.container', '').strip('"\'') - docker_pull_cmd = 'docker pull {}:dev'.format(docker_notag) + if self.config.get("process.container", ""): + docker_notag = re.sub(r":(?:[\.\d]+|dev)$", "", self.config.get("process.container", "").strip("\"'")) + docker_withtag = self.config.get("process.container", "").strip("\"'") + + # docker build + docker_build_cmd = "docker build --no-cache . -t {}".format(docker_withtag) try: - steps = ciwf['jobs']['test']['steps'] - assert(any([docker_pull_cmd in step['run'] for step in steps if 'run' in step.keys()])) + steps = ciwf["jobs"]["test"]["steps"] + assert any([docker_build_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "CI is not pulling the correct docker image. Should be:\n '{}'".format(docker_pull_cmd))) + self.failed.append( + (5, "CI is not building the correct docker image. Should be: `{}`".format(docker_build_cmd),) + ) + else: + self.passed.append((5, "CI is building the correct docker image: `{}`".format(docker_build_cmd))) + + # docker pull + docker_pull_cmd = "docker pull {}:dev".format(docker_notag) + try: + steps = ciwf["jobs"]["test"]["steps"] + assert any([docker_pull_cmd in step["run"] for step in steps if "run" in step.keys()]) + except (AssertionError, KeyError, TypeError): + self.failed.append( + (5, "CI is not pulling the correct docker image. Should be: `{}`".format(docker_pull_cmd)) + ) else: self.passed.append((5, "CI is pulling the correct docker image: {}".format(docker_pull_cmd))) - docker_tag_cmd = 'docker tag {}:dev {}'.format(docker_notag, docker_withtag) + # docker tag + docker_tag_cmd = "docker tag {}:dev {}".format(docker_notag, docker_withtag) try: - steps = ciwf['jobs']['test']['steps'] - assert(any([docker_tag_cmd in step['run'] for step in steps if 'run' in step.keys()])) + steps = ciwf["jobs"]["test"]["steps"] + assert any([docker_tag_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "CI is not tagging docker image correctly. Should be:\n '{}'".format(docker_tag_cmd))) + self.failed.append( + (5, "CI is not tagging docker image correctly. Should be: `{}`".format(docker_tag_cmd)) + ) else: self.passed.append((5, "CI is tagging docker image correctly: {}".format(docker_tag_cmd))) # Check that we are testing the minimum nextflow version try: - matrix = ciwf['jobs']['test']['strategy']['matrix']['nxf_ver'] - assert(any([self.minNextflowVersion in matrix])) + matrix = ciwf["jobs"]["test"]["strategy"]["matrix"]["nxf_ver"] + assert any([self.minNextflowVersion in matrix]) except (KeyError, TypeError): - self.failed.append((5, "Continuous integration does not check minimum NF version: '{}'".format(fn))) + self.failed.append((5, "Continuous integration does not check minimum NF version: `{}`".format(fn))) except AssertionError: - self.failed.append((5, "Minimum NF version differed from CI and what was set in the pipelines manifest: {}".format(fn))) + self.failed.append((5, "Minimum NF version different in CI and pipelines manifest: `{}`".format(fn))) else: - self.passed.append((5, "Continuous integration checks minimum NF version: '{}'".format(fn))) + self.passed.append((5, "Continuous integration checks minimum NF version: `{}`".format(fn))) def check_actions_lint(self): """Checks that the GitHub Actions lint workflow is valid Makes sure ``nf-core lint`` and ``markdownlint`` runs. """ - fn = os.path.join(self.path, '.github', 'workflows', 'linting.yml') + fn = os.path.join(self.path, ".github", "workflows", "linting.yml") if os.path.isfile(fn): - with open(fn, 'r') as fh: + with open(fn, "r") as fh: lintwf = yaml.safe_load(fh) # Check that the action is turned on for push and pull requests try: - assert('push' in lintwf[True]) - assert('pull_request' in lintwf[True]) + assert "push" in lintwf[True] + assert "pull_request" in lintwf[True] except (AssertionError, KeyError, TypeError): - self.failed.append((5, "GitHub Actions linting workflow must be triggered on PR and push: '{}'".format(fn))) + self.failed.append( + (5, "GitHub Actions linting workflow must be triggered on PR and push: `{}`".format(fn)) + ) else: - self.passed.append((5, "GitHub Actions linting workflow is triggered on PR and push: '{}'".format(fn))) + self.passed.append((5, "GitHub Actions linting workflow is triggered on PR and push: `{}`".format(fn))) # Check that the Markdown linting runs - Markdownlint_cmd = 'markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml' + Markdownlint_cmd = "markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml" try: - steps = lintwf['jobs']['Markdown']['steps'] - assert(any([Markdownlint_cmd in step['run'] for step in steps if 'run' in step.keys()])) + steps = lintwf["jobs"]["Markdown"]["steps"] + assert any([Markdownlint_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "Continuous integration must run Markdown lint Tests: '{}'".format(fn))) + self.failed.append((5, "Continuous integration must run Markdown lint Tests: `{}`".format(fn))) else: - self.passed.append((5, "Continuous integration runs Markdown lint Tests: '{}'".format(fn))) - + self.passed.append((5, "Continuous integration runs Markdown lint Tests: `{}`".format(fn))) # Check that the nf-core linting runs - nfcore_lint_cmd = 'nf-core lint ${GITHUB_WORKSPACE}' + nfcore_lint_cmd = "nf-core lint ${GITHUB_WORKSPACE}" + try: + steps = lintwf["jobs"]["nf-core"]["steps"] + assert any([nfcore_lint_cmd in step["run"] for step in steps if "run" in step.keys()]) + except (AssertionError, KeyError, TypeError): + self.failed.append((5, "Continuous integration must run nf-core lint Tests: `{}`".format(fn))) + else: + self.passed.append((5, "Continuous integration runs nf-core lint Tests: `{}`".format(fn))) + + def check_actions_awstest(self): + """Checks the GitHub Actions awstest is valid. + + Makes sure it is triggered only on ``push`` to ``master``. + """ + fn = os.path.join(self.path, ".github", "workflows", "awstest.yml") + if os.path.isfile(fn): + with open(fn, "r") as fh: + wf = yaml.safe_load(fh) + + # Check that the action is only turned on for push + try: + assert "push" in wf[True] + assert "pull_request" not in wf[True] + except (AssertionError, KeyError, TypeError): + self.failed.append( + (5, "GitHub Actions AWS test should be triggered on push and not PRs: `{}`".format(fn)) + ) + else: + self.passed.append((5, "GitHub Actions AWS test is triggered on push and not PRs: `{}`".format(fn))) + + # Check that the action is only turned on for push to master + try: + assert "master" in wf[True]["push"]["branches"] + assert "dev" not in wf[True]["push"]["branches"] + except (AssertionError, KeyError, TypeError): + self.failed.append( + (5, "GitHub Actions AWS test should be triggered only on push to master: `{}`".format(fn)) + ) + else: + self.passed.append((5, "GitHub Actions AWS test is triggered only on push to master: `{}`".format(fn))) + + def check_actions_awsfulltest(self): + """Checks the GitHub Actions awsfulltest is valid. + + Makes sure it is triggered only on ``release``. + """ + fn = os.path.join(self.path, ".github", "workflows", "awsfulltest.yml") + if os.path.isfile(fn): + with open(fn, "r") as fh: + wf = yaml.safe_load(fh) + + aws_profile = "-profile test " + + # Check that the action is only turned on for published releases try: - steps = lintwf['jobs']['nf-core']['steps'] - assert(any([ nfcore_lint_cmd in step['run'] for step in steps if 'run' in step.keys()])) + assert "release" in wf[True] + assert "published" in wf[True]["release"]["types"] + assert "push" not in wf[True] + assert "pull_request" not in wf[True] except (AssertionError, KeyError, TypeError): - self.failed.append((5, "Continuous integration must run nf-core lint Tests: '{}'".format(fn))) + self.failed.append( + (5, "GitHub Actions AWS full test should be triggered only on published release: `{}`".format(fn)) + ) else: - self.passed.append((5, "Continuous integration runs nf-core lint Tests: '{}'".format(fn))) + self.passed.append( + (5, "GitHub Actions AWS full test is triggered only on published release: `{}`".format(fn)) + ) + + # Warn if `-profile test` is still unchanged + try: + steps = wf["jobs"]["run-awstest"]["steps"] + assert any([aws_profile in step["run"] for step in steps if "run" in step.keys()]) + except (AssertionError, KeyError, TypeError): + self.passed.append((5, "GitHub Actions AWS full test should test full datasets: `{}`".format(fn))) + else: + self.warned.append((5, "GitHub Actions AWS full test should test full datasets: `{}`".format(fn))) def check_readme(self): """Checks the repository README file for errors. Currently just checks the badges at the top of the README. """ - with open(os.path.join(self.path, 'README.md'), 'r') as fh: + with open(os.path.join(self.path, "README.md"), "r") as fh: content = fh.read() # Check that there is a readme badge showing the minimum required version of Nextflow @@ -630,24 +860,37 @@ def check_readme(self): nf_badge_re = r"\[!\[Nextflow\]\(https://img\.shields\.io/badge/nextflow-%E2%89%A5([\d\.]+)-brightgreen\.svg\)\]\(https://www\.nextflow\.io/\)" match = re.search(nf_badge_re, content) if match: - nf_badge_version = match.group(1).strip('\'"') + nf_badge_version = match.group(1).strip("'\"") try: assert nf_badge_version == self.minNextflowVersion except (AssertionError, KeyError): - self.failed.append((6, "README Nextflow minimum version badge does not match config. Badge: '{}', Config: '{}'".format(nf_badge_version, self.minNextflowVersion))) + self.failed.append( + ( + 6, + "README Nextflow minimum version badge does not match config. Badge: `{}`, Config: `{}`".format( + nf_badge_version, self.minNextflowVersion + ), + ) + ) else: - self.passed.append((6, "README Nextflow minimum version badge matched config. Badge: '{}', Config: '{}'".format(nf_badge_version, self.minNextflowVersion))) + self.passed.append( + ( + 6, + "README Nextflow minimum version badge matched config. Badge: `{}`, Config: `{}`".format( + nf_badge_version, self.minNextflowVersion + ), + ) + ) else: self.warned.append((6, "README did not have a Nextflow minimum version badge.")) # Check that we have a bioconda badge if we have a bioconda environment file - if 'environment.yml' in self.files: - bioconda_badge = '[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/)' + if "environment.yml" in self.files: + bioconda_badge = "[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/)" if bioconda_badge in content: self.passed.append((6, "README had a bioconda badge")) else: - self.failed.append((6, "Found a bioconda environment.yml file but no badge in the README")) - + self.warned.append((6, "Found a bioconda environment.yml file but no badge in the README")) def check_version_consistency(self): """Checks container tags versions. @@ -662,37 +905,47 @@ def check_version_consistency(self): versions = {} # Get the version definitions # Get version from nextflow.config - versions['manifest.version'] = self.config.get('manifest.version', '').strip(' \'"') + versions["manifest.version"] = self.config.get("manifest.version", "").strip(" '\"") # Get version from the docker slug - if self.config.get('process.container', '') and \ - not ':' in self.config.get('process.container', ''): - self.failed.append((7, "Docker slug seems not to have " - "a version tag: {}".format(self.config.get('process.container', '')))) + if self.config.get("process.container", "") and not ":" in self.config.get("process.container", ""): + self.failed.append( + ( + 7, + "Docker slug seems not to have " + "a version tag: {}".format(self.config.get("process.container", "")), + ) + ) return # Get config container slugs, (if set; one container per workflow) - if self.config.get('process.container', ''): - versions['process.container'] = self.config.get('process.container', '').strip(' \'"').split(':')[-1] - if self.config.get('process.container', ''): - versions['process.container'] = self.config.get('process.container', '').strip(' \'"').split(':')[-1] + if self.config.get("process.container", ""): + versions["process.container"] = self.config.get("process.container", "").strip(" '\"").split(":")[-1] + if self.config.get("process.container", ""): + versions["process.container"] = self.config.get("process.container", "").strip(" '\"").split(":")[-1] # Get version from the GITHUB_REF env var if this is a release - if os.environ.get('GITHUB_REF', '').startswith('refs/tags/') and os.environ.get('GITHUB_REPOSITORY', '') != 'nf-core/tools': - versions['GITHUB_REF'] = os.path.basename(os.environ['GITHUB_REF'].strip(' \'"')) + if ( + os.environ.get("GITHUB_REF", "").startswith("refs/tags/") + and os.environ.get("GITHUB_REPOSITORY", "") != "nf-core/tools" + ): + versions["GITHUB_REF"] = os.path.basename(os.environ["GITHUB_REF"].strip(" '\"")) # Check if they are all numeric for v_type, version in versions.items(): - if not version.replace('.', '').isdigit(): + if not version.replace(".", "").isdigit(): self.failed.append((7, "{} was not numeric: {}!".format(v_type, version))) return # Check if they are consistent if len(set(versions.values())) != 1: - self.failed.append((7, "The versioning is not consistent between container, release tag " - "and config. Found {}".format( - ", ".join(["{} = {}".format(k, v) for k,v in versions.items()]) - ))) + self.failed.append( + ( + 7, + "The versioning is not consistent between container, release tag " + "and config. Found {}".format(", ".join(["{} = {}".format(k, v) for k, v in versions.items()])), + ) + ) return self.passed.append((7, "Version tags are numeric and consistent between container, release tag and config.")) @@ -705,68 +958,84 @@ def check_conda_env_yaml(self): * check that dependency versions are pinned * dependency versions are the latest available """ - if 'environment.yml' not in self.files: + if "environment.yml" not in self.files: return # Check that the environment name matches the pipeline name - pipeline_version = self.config.get('manifest.version', '').strip(' \'"') - expected_env_name = 'nf-core-{}-{}'.format(self.pipeline_name.lower(), pipeline_version) - if self.conda_config['name'] != expected_env_name: - self.failed.append((8, "Conda environment name is incorrect ({}, should be {})".format(self.conda_config['name'], expected_env_name))) + pipeline_version = self.config.get("manifest.version", "").strip(" '\"") + expected_env_name = "nf-core-{}-{}".format(self.pipeline_name.lower(), pipeline_version) + if self.conda_config["name"] != expected_env_name: + self.failed.append( + ( + 8, + "Conda environment name is incorrect ({}, should be {})".format( + self.conda_config["name"], expected_env_name + ), + ) + ) else: self.passed.append((8, "Conda environment name was correct ({})".format(expected_env_name))) # Check conda dependency list - for dep in self.conda_config.get('dependencies', []): + for dep in self.conda_config.get("dependencies", []): if isinstance(dep, str): # Check that each dependency has a version number try: - assert dep.count('=') == 1 + assert dep.count("=") in [1, 2] except AssertionError: - self.failed.append((8, "Conda dependency did not have pinned version number: {}".format(dep))) + self.failed.append((8, "Conda dependency did not have pinned version number: `{}`".format(dep))) else: - self.passed.append((8, "Conda dependency had pinned version number: {}".format(dep))) + self.passed.append((8, "Conda dependency had pinned version number: `{}`".format(dep))) try: - depname, depver = dep.split('=', 1) + depname, depver = dep.split("=")[:2] self.check_anaconda_package(dep) except ValueError: pass else: # Check that required version is available at all - if depver not in self.conda_package_info[dep].get('versions'): + if depver not in self.conda_package_info[dep].get("versions"): self.failed.append((8, "Conda dependency had an unknown version: {}".format(dep))) continue # No need to test for latest version, continue linting # Check version is latest available - last_ver = self.conda_package_info[dep].get('latest_version') + last_ver = self.conda_package_info[dep].get("latest_version") if last_ver is not None and last_ver != depver: - self.warned.append((8, "Conda package is not latest available: {}, {} available".format(dep, last_ver))) + self.warned.append( + (8, "Conda package is not latest available: `{}`, `{}` available".format(dep, last_ver)) + ) else: - self.passed.append((8, "Conda package is latest available: {}".format(dep))) + self.passed.append((8, "Conda package is latest available: `{}`".format(dep))) elif isinstance(dep, dict): - for pip_dep in dep.get('pip', []): + for pip_dep in dep.get("pip", []): # Check that each pip dependency has a version number try: - assert pip_dep.count('=') == 2 + assert pip_dep.count("=") == 2 except AssertionError: self.failed.append((8, "Pip dependency did not have pinned version number: {}".format(pip_dep))) else: self.passed.append((8, "Pip dependency had pinned version number: {}".format(pip_dep))) try: - pip_depname, pip_depver = pip_dep.split('==', 1) + pip_depname, pip_depver = pip_dep.split("==", 1) self.check_pip_package(pip_dep) except ValueError: pass else: # Check, if PyPi package version is available at all - if pip_depver not in self.conda_package_info[pip_dep].get('releases').keys(): + if pip_depver not in self.conda_package_info[pip_dep].get("releases").keys(): self.failed.append((8, "PyPi package had an unknown version: {}".format(pip_depver))) continue # No need to test latest version, if not available - last_ver = self.conda_package_info[pip_dep].get('info').get('version') + last_ver = self.conda_package_info[pip_dep].get("info").get("version") if last_ver is not None and last_ver != pip_depver: - self.warned.append((8, "PyPi package is not latest available: {}, {} available".format(pip_depver, last_ver))) + self.warned.append( + ( + 8, + "PyPi package is not latest available: {}, {} available".format( + pip_depver, last_ver + ), + ) + ) else: self.passed.append((8, "PyPi package is latest available: {}".format(pip_depver))) @@ -782,24 +1051,17 @@ def check_anaconda_package(self, dep): A ValueError, if the package name can not be resolved. """ # Check if each dependency is the latest available version - depname, depver = dep.split('=', 1) - dep_channels = self.conda_config.get('channels', []) + depname, depver = dep.split("=", 1) + dep_channels = self.conda_config.get("channels", []) # 'defaults' isn't actually a channel name. See https://docs.anaconda.com/anaconda/user-guide/tasks/using-repositories/ - if 'defaults' in dep_channels: - dep_channels.remove('defaults') - dep_channels.extend([ - 'main', - 'anaconda', - 'r', - 'free', - 'archive', - 'anaconda-extras' - ]) - if '::' in depname: - dep_channels = [depname.split('::')[0]] - depname = depname.split('::')[1] + if "defaults" in dep_channels: + dep_channels.remove("defaults") + dep_channels.extend(["main", "anaconda", "r", "free", "archive", "anaconda-extras"]) + if "::" in depname: + dep_channels = [depname.split("::")[0]] + depname = depname.split("::")[1] for ch in dep_channels: - anaconda_api_url = 'https://api.anaconda.org/package/{}/{}'.format(ch, depname) + anaconda_api_url = "https://api.anaconda.org/package/{}/{}".format(ch, depname) try: response = requests.get(anaconda_api_url, timeout=10) except (requests.exceptions.Timeout): @@ -814,10 +1076,17 @@ def check_anaconda_package(self, dep): self.conda_package_info[dep] = dep_json return elif response.status_code != 404: - self.warned.append((8, "Anaconda API returned unexpected response code '{}' for: {}\n{}".format(response.status_code, anaconda_api_url, response))) + self.warned.append( + ( + 8, + "Anaconda API returned unexpected response code `{}` for: {}\n{}".format( + response.status_code, anaconda_api_url, response + ), + ) + ) raise ValueError elif response.status_code == 404: - logging.debug("Could not find {} in conda channel {}".format(dep, ch)) + log.debug("Could not find {} in conda channel {}".format(dep, ch)) else: # We have looped through each channel and had a 404 response code on everything self.failed.append((8, "Could not find Conda dependency using the Anaconda API: {}".format(dep))) @@ -834,8 +1103,8 @@ def check_pip_package(self, dep): Raises: A ValueError, if the package name can not be resolved or the connection timed out. """ - pip_depname, pip_depver = dep.split('=', 1) - pip_api_url = 'https://pypi.python.org/pypi/{}/json'.format(pip_depname) + pip_depname, pip_depver = dep.split("=", 1) + pip_api_url = "https://pypi.python.org/pypi/{}/json".format(pip_depname) try: response = requests.get(pip_api_url, timeout=10) except (requests.exceptions.Timeout): @@ -860,15 +1129,15 @@ def check_conda_dockerfile(self): * dependency versions are pinned * dependency versions are the latest available """ - if 'environment.yml' not in self.files or len(self.dockerfile) == 0: + if "environment.yml" not in self.files or len(self.dockerfile) == 0: return expected_strings = [ - "FROM nfcore/base:{}".format('dev' if 'dev' in nf_core.__version__ else nf_core.__version__), - 'COPY environment.yml /', - 'RUN conda env create -f /environment.yml && conda clean -a', - 'RUN conda env export --name {} > {}.yml'.format(self.conda_config['name'], self.conda_config['name']), - 'ENV PATH /opt/conda/envs/{}/bin:$PATH'.format(self.conda_config['name']) + "FROM nfcore/base:{}".format("dev" if "dev" in nf_core.__version__ else nf_core.__version__), + "COPY environment.yml /", + "RUN conda env create --quiet -f /environment.yml && conda clean -a", + "RUN conda env export --name {} > {}.yml".format(self.conda_config["name"], self.conda_config["name"]), + "ENV PATH /opt/conda/envs/{}/bin:$PATH".format(self.conda_config["name"]), ] difference = set(expected_strings) - set(self.dockerfile) @@ -880,70 +1149,336 @@ def check_conda_dockerfile(self): def check_pipeline_todos(self): """ Go through all template files looking for the string 'TODO nf-core:' """ - ignore = ['.git'] - if os.path.isfile(os.path.join(self.path, '.gitignore')): - with io.open(os.path.join(self.path, '.gitignore'), 'rt', encoding='latin1') as fh: + ignore = [".git"] + if os.path.isfile(os.path.join(self.path, ".gitignore")): + with io.open(os.path.join(self.path, ".gitignore"), "rt", encoding="latin1") as fh: for l in fh: - ignore.append(os.path.basename(l.strip().rstrip('/'))) + ignore.append(os.path.basename(l.strip().rstrip("/"))) for root, dirs, files in os.walk(self.path): # Ignore files for i in ignore: - if i in dirs: - dirs.remove(i) - if i in files: - files.remove(i) + dirs = [d for d in dirs if not fnmatch.fnmatch(os.path.join(root, d), i)] + files = [f for f in files if not fnmatch.fnmatch(os.path.join(root, f), i)] for fname in files: - with io.open(os.path.join(root, fname), 'rt', encoding='latin1') as fh: + with io.open(os.path.join(root, fname), "rt", encoding="latin1") as fh: for l in fh: - if 'TODO nf-core' in l: - l = l.replace('', '').replace('# TODO nf-core: ', '').replace('// TODO nf-core: ', '').replace('TODO nf-core: ', '').strip() - if len(fname) + len(l) > 50: - l = '{}..'.format(l[:50-len(fname)]) - self.warned.append((10, "TODO string found in '{}': {}".format(fname,l))) + if "TODO nf-core" in l: + l = ( + l.replace("", "") + .replace("# TODO nf-core: ", "") + .replace("// TODO nf-core: ", "") + .replace("TODO nf-core: ", "") + .strip() + ) + self.warned.append((10, "TODO string found in `{}`: _{}_".format(fname, l))) def check_pipeline_name(self): """Check whether pipeline name adheres to lower case/no hyphen naming convention""" - if self.pipeline_name.islower() and self.pipeline_name.isalpha(): + if self.pipeline_name.islower() and self.pipeline_name.isalnum(): self.passed.append((12, "Name adheres to nf-core convention")) if not self.pipeline_name.islower(): self.warned.append((12, "Naming does not adhere to nf-core conventions: Contains uppercase letters")) - if not self.pipeline_name.isalpha(): - self.warned.append((12, "Naming does not adhere to nf-core conventions: Contains non alphabetical characters")) + if not self.pipeline_name.isalnum(): + self.warned.append( + (12, "Naming does not adhere to nf-core conventions: Contains non alphanumeric characters") + ) + + def check_cookiecutter_strings(self): + """ + Look for the string 'cookiecutter' in all pipeline files. + Finding it probably means that there has been a copy+paste error from the template. + """ + try: + # First, try to get the list of files using git + git_ls_files = subprocess.check_output(["git", "ls-files"], cwd=self.path).splitlines() + list_of_files = [os.path.join(self.path, s.decode("utf-8")) for s in git_ls_files] + except subprocess.CalledProcessError as e: + # Failed, so probably not initialised as a git repository - just a list of all files + log.debug("Couldn't call 'git ls-files': {}".format(e)) + list_of_files = [] + for subdir, dirs, files in os.walk(self.path): + for file in files: + list_of_files.append(os.path.join(subdir, file)) + + # Loop through files, searching for string + num_matches = 0 + num_files = 0 + for fn in list_of_files: + num_files += 1 + with io.open(fn, "r", encoding="latin1") as fh: + lnum = 0 + for l in fh: + lnum += 1 + cc_matches = re.findall(r"{{\s*cookiecutter[^}]*}}", l) + if len(cc_matches) > 0: + for cc_match in cc_matches: + self.failed.append( + (13, "Found a cookiecutter template string in `{}` L{}: {}".format(fn, lnum, cc_match)) + ) + num_matches += 1 + if num_matches == 0: + self.passed.append((13, "Did not find any cookiecutter template strings ({} files)".format(num_files))) + + def check_schema_lint(self): + """ Lint the pipeline schema """ + + # Only show error messages from schema + if log.getEffectiveLevel() == logging.INFO: + logging.getLogger("nf_core.schema").setLevel(logging.ERROR) + + # Lint the schema + self.schema_obj = nf_core.schema.PipelineSchema() + self.schema_obj.get_schema_path(self.path) + try: + self.schema_obj.load_lint_schema() + self.passed.append((14, "Schema lint passed")) + except AssertionError as e: + self.failed.append((14, "Schema lint failed: {}".format(e))) + + # Check the title and description - gives warnings instead of fail + if self.schema_obj.schema is not None: + try: + self.schema_obj.validate_schema_title_description() + self.passed.append((14, "Schema title + description lint passed")) + except AssertionError as e: + self.warned.append((14, e)) + + def check_schema_params(self): + """ Check that the schema describes all flat params in the pipeline """ + + # First, get the top-level config options for the pipeline + # Schema object already created in the previous test + self.schema_obj.get_schema_path(self.path) + self.schema_obj.get_wf_params() + self.schema_obj.no_prompts = True + # Remove any schema params not found in the config + removed_params = self.schema_obj.remove_schema_notfound_configs() + # Add schema params found in the config but not the schema + added_params = self.schema_obj.add_schema_found_configs() + + if len(removed_params) > 0: + for param in removed_params: + self.warned.append((15, "Schema param `{}` not found from nextflow config".format(param))) + + if len(added_params) > 0: + for param in added_params: + self.failed.append( + (15, "Param `{}` from `nextflow config` not found in nextflow_schema.json".format(param)) + ) + + if len(removed_params) == 0 and len(added_params) == 0: + self.passed.append((15, "Schema matched params returned from nextflow config")) def print_results(self): - # Print results - rl = "\n Using --release mode linting tests" if self.release_mode else '' - logging.info("{}\n LINTING RESULTS\n{}\n".format(click.style('='*29, dim=True), click.style('='*35, dim=True)) + - click.style(" [{}] {:>4} tests passed\n".format(u'\u2714', len(self.passed)), fg='green') + - click.style(" [!] {:>4} tests had warnings\n".format(len(self.warned)), fg='yellow') + - click.style(" [{}] {:>4} tests failed".format(u'\u2717', len(self.failed)), fg='red') + rl - ) + console = Console() # Helper function to format test links nicely - def format_result(test_results): + def format_result(test_results, table): """ - Given an error message ID and the message text, return a nicely formatted + Given an list of error message IDs and the message texts, return a nicely formatted string for the terminal with appropriate ASCII colours. """ - print_results = [] for eid, msg in test_results: - url = click.style("http://nf-co.re/errors#{}".format(eid), fg='blue') - print_results.append('{} : {}'.format(url, msg)) - return "\n ".join(print_results) - - if len(self.passed) > 0: - logging.debug("{}\n {}".format(click.style("Test Passed:", fg='green'), format_result(self.passed))) + table.add_row( + Markdown("[https://nf-co.re/errors#{0}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) + ) + return table + + def _s(some_list): + if len(some_list) > 1: + return "s" + return "" + + # Table of passed tests + if len(self.passed) > 0 and log.getEffectiveLevel() == logging.DEBUG: + table = Table(style="green", box=rich.box.ROUNDED) + table.add_column( + "[[\u2714]] {} Test{} Passed".format(len(self.passed), _s(self.passed)), no_wrap=True, + ) + table = format_result(self.passed, table) + console.print(table) + + # Table of warning tests if len(self.warned) > 0: - logging.warning("{}\n {}".format(click.style("Test Warnings:", fg='yellow'), format_result(self.warned))) + table = Table(style="yellow", box=rich.box.ROUNDED) + table.add_column("[[!]] {} Test Warning{}".format(len(self.warned), _s(self.warned)), no_wrap=True) + table = format_result(self.warned, table) + console.print(table) + + # Table of failing tests if len(self.failed) > 0: - logging.error("{}\n {}".format(click.style("Test Failures:", fg='red'), format_result(self.failed))) + table = Table(style="red", box=rich.box.ROUNDED) + table.add_column( + "[[\u2717]] {} Test{} Failed".format(len(self.failed), _s(self.failed)), no_wrap=True, + ) + table = format_result(self.failed, table) + console.print(table) + + # Summary table + + table = Table(box=rich.box.ROUNDED) + table.add_column("[bold green]LINT RESULTS SUMMARY".format(len(self.passed)), no_wrap=True) + table.add_row( + "[[\u2714]] {:>3} Test{} Passed".format(len(self.passed), _s(self.passed)), style="green", + ) + table.add_row("[[!]] {:>3} Test Warning{}".format(len(self.warned), _s(self.warned)), style="yellow") + table.add_row("[[\u2717]] {:>3} Test{} Failed".format(len(self.failed), _s(self.failed)), style="red") + console.print(table) + + def get_results_md(self): + """ + Function to create a markdown file suitable for posting in a GitHub comment + """ + # Overall header + overall_result = "Passed :white_check_mark:" + if len(self.failed) > 0: + overall_result = "Failed :x:" + + # List of tests for details + test_failures = "" + if len(self.failed) > 0: + test_failures = "### :x: Test failures:\n\n{}\n\n".format( + "\n".join( + [ + "* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, "`")) + for eid, msg in self.failed + ] + ) + ) + + test_warnings = "" + if len(self.warned) > 0: + test_warnings = "### :heavy_exclamation_mark: Test warnings:\n\n{}\n\n".format( + "\n".join( + [ + "* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, "`")) + for eid, msg in self.warned + ] + ) + ) + + test_passes = "" + if len(self.passed) > 0: + test_passes = "### :white_check_mark: Tests passed:\n\n{}\n\n".format( + "\n".join( + [ + "* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, "`")) + for eid, msg in self.passed + ] + ) + ) + + now = datetime.datetime.now() + + markdown = textwrap.dedent( + """ + #### `nf-core lint` overall result: {} + + {} + + ```diff + +| ✅ {:2d} tests passed |+ + !| ❗ {:2d} tests had warnings |! + -| ❌ {:2d} tests failed |- + ``` +
- def _bold_list_items(self, files): + {}{}{}### Run details: + + * nf-core/tools version {} + * Run at `{}` + +
+ """ + ).format( + overall_result, + "Posted for pipeline commit {}".format(self.git_sha[:7]) if self.git_sha is not None else "", + len(self.passed), + len(self.warned), + len(self.failed), + test_failures, + test_warnings, + test_passes, + nf_core.__version__, + now.strftime("%Y-%m-%d %H:%M:%S"), + ) + + return markdown + + def save_json_results(self, json_fn): + """ + Function to dump lint results to a JSON file for downstream use + """ + + log.info("Writing lint results to {}".format(json_fn)) + now = datetime.datetime.now() + results = { + "nf_core_tools_version": nf_core.__version__, + "date_run": now.strftime("%Y-%m-%d %H:%M:%S"), + "tests_pass": [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.passed], + "tests_warned": [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.warned], + "tests_failed": [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.failed], + "num_tests_pass": len(self.passed), + "num_tests_warned": len(self.warned), + "num_tests_failed": len(self.failed), + "has_tests_pass": len(self.passed) > 0, + "has_tests_warned": len(self.warned) > 0, + "has_tests_failed": len(self.failed) > 0, + "markdown_result": self.get_results_md(), + } + with open(json_fn, "w") as fh: + json.dump(results, fh, indent=4) + + def github_comment(self): + """ + If we are running in a GitHub PR, try to post results as a comment + """ + if os.environ.get("GITHUB_TOKEN", "") != "" and os.environ.get("GITHUB_COMMENTS_URL", "") != "": + try: + headers = {"Authorization": "token {}".format(os.environ["GITHUB_TOKEN"])} + # Get existing comments - GET + get_r = requests.get(url=os.environ["GITHUB_COMMENTS_URL"], headers=headers) + if get_r.status_code == 200: + + # Look for an existing comment to update + update_url = False + for comment in get_r.json(): + if comment["user"]["login"] == "github-actions[bot]" and comment["body"].startswith( + "\n#### `nf-core lint` overall result" + ): + # Update existing comment - PATCH + log.info("Updating GitHub comment") + update_r = requests.patch( + url=comment["url"], + data=json.dumps({"body": self.get_results_md().replace("Posted", "**Updated**")}), + headers=headers, + ) + return + + # Create new comment - POST + if len(self.warned) > 0 or len(self.failed) > 0: + log.info("Posting GitHub comment") + post_r = requests.post( + url=os.environ["GITHUB_COMMENTS_URL"], + data=json.dumps({"body": self.get_results_md()}), + headers=headers, + ) + + except Exception as e: + log.warning("Could not post GitHub comment: {}\n{}".format(os.environ["GITHUB_COMMENTS_URL"], e)) + + def _wrap_quotes(self, files): if not isinstance(files, list): files = [files] - bfiles = [click.style(f, bold=True) for f in files] - return ' or '.join(bfiles) + bfiles = ["`{}`".format(f) for f in files] + return " or ".join(bfiles) + + def _strip_ansi_codes(self, string, replace_with=""): + # https://stackoverflow.com/a/14693789/713980 + ansi_escape = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])") + return ansi_escape.sub(replace_with, string) diff --git a/nf_core/list.py b/nf_core/list.py index e64c4d2225..c81d35bd42 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -7,24 +7,26 @@ import click import datetime import errno +import git import json import logging import os import re +import requests +import rich.console +import rich.table import subprocess import sys -import git -import requests -import tabulate - import nf_core.utils +log = logging.getLogger(__name__) + # Set up local caching for requests to speed up remote queries nf_core.utils.setup_requests_cachedir() -def list_workflows(filter_by=None, sort_by='release', as_json=False): +def list_workflows(filter_by=None, sort_by="release", as_json=False, show_archived=False): """Prints out a list of all nf-core workflows. Args: @@ -33,14 +35,55 @@ def list_workflows(filter_by=None, sort_by='release', as_json=False): `release` (default), `name`, `stars`. as_json (boolean): Set to true, if the lists should be printed in JSON. """ - wfs = Workflows(filter_by, sort_by) + wfs = Workflows(filter_by, sort_by, show_archived) wfs.get_remote_workflows() wfs.get_local_nf_workflows() wfs.compare_remote_local() if as_json: - wfs.print_json() + return wfs.print_json() + else: + return wfs.print_summary() + + +def get_local_wf(workflow, revision=None): + """ + Check if this workflow has a local copy and use nextflow to pull it if not + """ + # Assume nf-core if no org given + if workflow.count("/") == 0: + workflow = "nf-core/{}".format(workflow) + + wfs = Workflows() + wfs.get_local_nf_workflows() + for wf in wfs.local_workflows: + if workflow == wf.full_name: + if revision is None or revision == wf.commit_sha or revision == wf.branch or revision == wf.active_tag: + if wf.active_tag: + print_revision = "v{}".format(wf.active_tag) + elif wf.branch: + print_revision = "{} - {}".format(wf.branch, wf.commit_sha[:7]) + else: + print_revision = wf.commit_sha + log.info("Using local workflow: {} ({})".format(workflow, print_revision)) + return wf.local_path + + # Wasn't local, fetch it + log.info("Downloading workflow: {} ({})".format(workflow, revision)) + try: + with open(os.devnull, "w") as devnull: + cmd = ["nextflow", "pull", workflow] + if revision is not None: + cmd.extend(["-r", revision]) + subprocess.check_output(cmd, stderr=devnull) + except OSError as e: + if e.errno == errno.ENOENT: + raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") + except subprocess.CalledProcessError as e: + raise AssertionError("`nextflow pull` returned non-zero error code: %s,\n %s", e.returncode, e.output) else: - wfs.print_summary() + local_wf = LocalWorkflow(workflow) + local_wf.get_local_nf_workflow_details() + return local_wf.local_path class Workflows(object): @@ -54,24 +97,26 @@ class Workflows(object): sort_by (str): workflows can be sorted by keywords. Keyword must be one of `release` (default), `name`, `stars`. """ - def __init__(self, filter_by=None, sort_by='release'): + + def __init__(self, filter_by=None, sort_by="release", show_archived=False): self.remote_workflows = list() self.local_workflows = list() self.local_unmatched = list() self.keyword_filters = filter_by if filter_by is not None else [] self.sort_workflows_by = sort_by + self.show_archived = show_archived def get_remote_workflows(self): - """Retrieves remote workflows from `nf-co.re `_. + """Retrieves remote workflows from `nf-co.re `_. Remote workflows are stored in :attr:`self.remote_workflows` list. """ # List all repositories at nf-core - logging.debug("Fetching list of nf-core workflows") - nfcore_url = 'http://nf-co.re/pipelines.json' + log.debug("Fetching list of nf-core workflows") + nfcore_url = "https://nf-co.re/pipelines.json" response = requests.get(nfcore_url, timeout=10) if response.status_code == 200: - repos = response.json()['remote_workflows'] + repos = response.json()["remote_workflows"] for repo in repos: self.remote_workflows.append(RemoteWorkflow(repo)) @@ -81,35 +126,38 @@ def get_local_nf_workflows(self): Local workflows are stored in :attr:`self.local_workflows` list. """ # Try to guess the local cache directory (much faster than calling nextflow) - if os.environ.get('NXF_ASSETS'): - nf_wfdir = os.path.join(os.environ.get('NXF_ASSETS'), 'nf-core') + if len(os.environ.get("NXF_ASSETS", "")) > 0: + nextflow_wfdir = os.environ.get("NXF_ASSETS") else: - nf_wfdir = os.path.join(os.getenv("HOME"), '.nextflow', 'assets', 'nf-core') - if os.path.isdir(nf_wfdir): - logging.debug("Guessed nextflow assets directory - pulling nf-core dirnames") - for wf_name in os.listdir(nf_wfdir): - self.local_workflows.append( LocalWorkflow('nf-core/{}'.format(wf_name)) ) + nextflow_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets") + if os.path.isdir(nextflow_wfdir): + log.debug("Guessed nextflow assets directory - pulling pipeline dirnames") + for org_name in os.listdir(nextflow_wfdir): + for wf_name in os.listdir(os.path.join(nextflow_wfdir, org_name)): + self.local_workflows.append(LocalWorkflow("{}/{}".format(org_name, wf_name))) # Fetch details about local cached pipelines with `nextflow list` else: - logging.debug("Getting list of local nextflow workflows") + log.debug("Getting list of local nextflow workflows") try: - with open(os.devnull, 'w') as devnull: - nflist_raw = subprocess.check_output(['nextflow', 'list'], stderr=devnull) + with open(os.devnull, "w") as devnull: + nflist_raw = subprocess.check_output(["nextflow", "list"], stderr=devnull) except OSError as e: if e.errno == errno.ENOENT: - raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") + raise AssertionError( + "It looks like Nextflow is not installed. It is required for most nf-core functions." + ) except subprocess.CalledProcessError as e: raise AssertionError("`nextflow list` returned non-zero error code: %s,\n %s", e.returncode, e.output) else: for wf_name in nflist_raw.splitlines(): - if not str(wf_name).startswith('nf-core/'): + if not str(wf_name).startswith("nf-core/"): self.local_unmatched.append(wf_name) else: - self.local_workflows.append( LocalWorkflow(wf_name) ) + self.local_workflows.append(LocalWorkflow(wf_name)) # Find additional information about each workflow by checking its git history - logging.debug("Fetching extra info about {} local workflows".format(len(self.local_workflows))) + log.debug("Fetching extra info about {} local workflows".format(len(self.local_workflows))) for wf in self.local_workflows: wf.get_local_nf_workflow_details() @@ -127,7 +175,7 @@ def compare_remote_local(self): if rwf.full_name == lwf.full_name: rwf.local_wf = lwf if rwf.releases: - if rwf.releases[-1]['tag_sha'] == lwf.commit_sha: + if rwf.releases[-1]["tag_sha"] == lwf.commit_sha: rwf.local_is_latest = True else: rwf.local_is_latest = False @@ -138,21 +186,22 @@ def filtered_workflows(self): Returns: list: Filtered remote workflows. """ - # If no keywords, don't filter - if not self.keyword_filters: - return self.remote_workflows - filtered_workflows = [] for wf in self.remote_workflows: + # Skip archived pipelines + if not self.show_archived and wf.archived: + continue + # Search through any supplied keywords for k in self.keyword_filters: in_name = k in wf.name if wf.name else False in_desc = k in wf.description if wf.description else False - in_topics = any([ k in t for t in wf.topics]) + in_topics = any([k in t for t in wf.topics]) if not in_name and not in_desc and not in_topics: break else: # We didn't hit a break, so all keywords were found filtered_workflows.append(wf) + return filtered_workflows def print_summary(self): @@ -160,63 +209,83 @@ def print_summary(self): filtered_workflows = self.filtered_workflows() - # Sort by released / dev, then alphabetical - if not self.sort_workflows_by or self.sort_workflows_by == 'release': + # Sort by released / dev / archived, then alphabetical + if not self.sort_workflows_by or self.sort_workflows_by == "release": filtered_workflows.sort( key=lambda wf: ( - (wf.releases[-1].get('published_at_timestamp', 0) if len(wf.releases) > 0 else 0) * -1, - wf.full_name.lower() + (wf.releases[-1].get("published_at_timestamp", 0) if len(wf.releases) > 0 else 0) * -1, + wf.archived, + wf.full_name.lower(), ) ) # Sort by date pulled - elif self.sort_workflows_by == 'pulled': + elif self.sort_workflows_by == "pulled": + def sort_pulled_date(wf): try: return wf.local_wf.last_pull * -1 except: return 0 + filtered_workflows.sort(key=sort_pulled_date) # Sort by name - elif self.sort_workflows_by == 'name': - filtered_workflows.sort( key=lambda wf: wf.full_name.lower() ) + elif self.sort_workflows_by == "name": + filtered_workflows.sort(key=lambda wf: wf.full_name.lower()) # Sort by stars, then name - elif self.sort_workflows_by == 'stars': - filtered_workflows.sort( - key=lambda wf: ( - wf.stargazers_count * -1, - wf.full_name.lower() - ) - ) + elif self.sort_workflows_by == "stars": + filtered_workflows.sort(key=lambda wf: (wf.stargazers_count * -1, wf.full_name.lower())) # Build summary list to print - summary = list() + table = rich.table.Table() + table.add_column("Pipeline Name") + table.add_column("Stars", justify="right") + table.add_column("Latest Release", justify="right") + table.add_column("Released", justify="right") + table.add_column("Last Pulled", justify="right") + table.add_column("Have latest release?") for wf in filtered_workflows: - version = click.style(wf.releases[-1]['tag_name'], fg='blue') if len(wf.releases) > 0 else click.style('dev', fg='yellow') - published = wf.releases[-1]['published_at_pretty'] if len(wf.releases) > 0 else '-' - pulled = wf.local_wf.last_pull_pretty if wf.local_wf is not None else '-' + wf_name = "[bold][link=https://nf-co.re/{0}]{0}[/link]".format(wf.name, wf.full_name) + version = "[yellow]dev" + if len(wf.releases) > 0: + version = "[blue]{}".format(wf.releases[-1]["tag_name"]) + published = wf.releases[-1]["published_at_pretty"] if len(wf.releases) > 0 else "[dim]-" + pulled = wf.local_wf.last_pull_pretty if wf.local_wf is not None else "[dim]-" if wf.local_wf is not None: - is_latest = click.style('Yes', fg='green') if wf.local_is_latest else click.style('No', fg='red') + revision = "" + if wf.local_wf.active_tag is not None: + revision = "v{}".format(wf.local_wf.active_tag) + elif wf.local_wf.branch is not None: + revision = "{} - {}".format(wf.local_wf.branch, wf.local_wf.commit_sha[:7]) + else: + revision = wf.local_wf.commit_sha + if wf.local_is_latest: + is_latest = "[green]Yes ({})".format(revision) + else: + is_latest = "[red]No ({})".format(revision) + else: + is_latest = "[dim]-" + + rowdata = [wf_name, str(wf.stargazers_count), version, published, pulled, is_latest] + + # Handle archived pipelines + if wf.archived: + rowdata[1] = "archived" + rowdata = [re.sub("\[\w+\]", "", k) for k in rowdata] + table.add_row(*rowdata, style="dim") else: - is_latest = '-' - rowdata = [ wf.full_name, version, published, pulled, is_latest ] - if self.sort_workflows_by == 'stars': - rowdata.insert(1, wf.stargazers_count) - summary.append(rowdata) - t_headers = ['Name', 'Version', 'Released', 'Last Pulled', 'Have latest release?'] - if self.sort_workflows_by == 'stars': - t_headers.insert(1, 'Stargazers') + table.add_row(*rowdata) + t_headers = ["Name", "Latest Release", "Released", "Last Pulled", "Have latest release?"] # Print summary table - print("", file=sys.stderr) - print(tabulate.tabulate(summary, headers=t_headers)) - print("", file=sys.stderr) + return table def print_json(self): """ Dump JSON of all parsed information """ - print(json.dumps({ - 'local_workflows': self.local_workflows, - 'remote_workflows': self.remote_workflows - }, default=lambda o: o.__dict__, indent=4)) + return json.dumps( + {"local_workflows": self.local_workflows, "remote_workflows": self.remote_workflows}, + default=lambda o: o.__dict__, + indent=4, + ) class RemoteWorkflow(object): @@ -229,17 +298,17 @@ class RemoteWorkflow(object): def __init__(self, data): # Vars from the initial data payload - self.name = data.get('name') - self.full_name = data.get('full_name') - self.description = data.get('description') - self.topics = data.get('topics', []) - self.archived = data.get('archived') - self.stargazers_count = data.get('stargazers_count') - self.watchers_count = data.get('watchers_count') - self.forks_count = data.get('forks_count') + self.name = data.get("name") + self.full_name = data.get("full_name") + self.description = data.get("description") + self.topics = data.get("topics", []) + self.archived = data.get("archived") + self.stargazers_count = data.get("stargazers_count") + self.watchers_count = data.get("watchers_count") + self.forks_count = data.get("forks_count") # Placeholder vars for releases info (ignore pre-releases) - self.releases = [ r for r in data.get('releases', []) if r.get('published_at') is not None ] + self.releases = [r for r in data.get("releases", []) if r.get("published_at") is not None] # Placeholder vars for local comparison self.local_wf = None @@ -247,10 +316,12 @@ def __init__(self, data): # Beautify date for release in self.releases: - release['published_at_pretty'] = pretty_date( - datetime.datetime.strptime(release.get('published_at'), "%Y-%m-%dT%H:%M:%SZ") + release["published_at_pretty"] = pretty_date( + datetime.datetime.strptime(release.get("published_at"), "%Y-%m-%dT%H:%M:%SZ") + ) + release["published_at_timestamp"] = int( + datetime.datetime.strptime(release.get("published_at"), "%Y-%m-%dT%H:%M:%SZ").strftime("%s") ) - release['published_at_timestamp'] = int(datetime.datetime.strptime(release.get('published_at'), "%Y-%m-%dT%H:%M:%SZ").strftime("%s")) class LocalWorkflow(object): @@ -264,6 +335,7 @@ def __init__(self, name): self.commit_sha = None self.remote_url = None self.branch = None + self.active_tag = None self.last_pull = None self.last_pull_date = None self.last_pull_pretty = None @@ -274,29 +346,31 @@ def get_local_nf_workflow_details(self): if self.local_path is None: # Try to guess the local cache directory - if os.environ.get('NXF_ASSETS'): - nf_wfdir = os.path.join(os.environ.get('NXF_ASSETS'), self.full_name) + if len(os.environ.get("NXF_ASSETS", "")) > 0: + nf_wfdir = os.path.join(os.environ.get("NXF_ASSETS"), self.full_name) else: - nf_wfdir = os.path.join(os.getenv("HOME"), '.nextflow', 'assets', self.full_name) + nf_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets", self.full_name) if os.path.isdir(nf_wfdir): - logging.debug("Guessed nextflow assets workflow directory") + log.debug("Guessed nextflow assets workflow directory: {}".format(nf_wfdir)) self.local_path = nf_wfdir # Use `nextflow info` to get more details about the workflow else: try: - with open(os.devnull, 'w') as devnull: - nfinfo_raw = subprocess.check_output(['nextflow', 'info', '-d', self.full_name], stderr=devnull) + with open(os.devnull, "w") as devnull: + nfinfo_raw = subprocess.check_output(["nextflow", "info", "-d", self.full_name], stderr=devnull) + nfinfo_raw = str(nfinfo_raw) except OSError as e: if e.errno == errno.ENOENT: - raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") + raise AssertionError( + "It looks like Nextflow is not installed. It is required for most nf-core functions." + ) except subprocess.CalledProcessError as e: - raise AssertionError("`nextflow list` returned non-zero error code: %s,\n %s", e.returncode, e.output) + raise AssertionError( + "`nextflow list` returned non-zero error code: %s,\n %s", e.returncode, e.output + ) else: - re_patterns = { - 'repository': r"repository\s*: (.*)", - 'local_path': r"local path\s*: (.*)" - } + re_patterns = {"repository": r"repository\s*: (.*)", "local_path": r"local path\s*: (.*)"} for key, pattern in re_patterns.items(): m = re.search(pattern, nfinfo_raw) if m: @@ -304,21 +378,35 @@ def get_local_nf_workflow_details(self): # Pull information from the local git repository if self.local_path is not None: + log.debug("Pulling git info from {}".format(self.local_path)) try: repo = git.Repo(self.local_path) self.commit_sha = str(repo.head.commit.hexsha) self.remote_url = str(repo.remotes.origin.url) - self.branch = str(repo.active_branch) - self.last_pull = os.stat(os.path.join(self.local_path, '.git', 'FETCH_HEAD')).st_mtime + self.last_pull = os.stat(os.path.join(self.local_path, ".git", "FETCH_HEAD")).st_mtime self.last_pull_date = datetime.datetime.fromtimestamp(self.last_pull).strftime("%Y-%m-%d %H:%M:%S") self.last_pull_pretty = pretty_date(self.last_pull) + + # Get the checked out branch if we can + try: + self.branch = str(repo.active_branch) + except TypeError: + self.branch = None + + # See if we are on a tag (release) + self.active_tag = None + for tag in repo.tags: + if str(tag.commit) == str(self.commit_sha): + self.active_tag = str(tag) + + # I'm not sure that we need this any more, it predated the self.branch catch above for detacted HEAD except TypeError as e: - logging.error( - "Could not fetch status of local Nextflow copy of {}:".format(self.full_name) + - "\n {}".format(str(e)) + - "\n\nIt's probably a good idea to delete this local copy and pull again:".format(self.local_path) + - "\n rm -rf {}".format(self.local_path) + - "\n nextflow pull {}".format(self.full_name) + log.error( + "Could not fetch status of local Nextflow copy of {}:".format(self.full_name) + + "\n {}".format(str(e)) + + "\n\nIt's probably a good idea to delete this local copy and pull again:".format(self.local_path) + + "\n rm -rf {}".format(self.local_path) + + "\n nextflow pull {}".format(self.full_name) ) @@ -331,6 +419,7 @@ def pretty_date(time): Adapted by sven1103 """ from datetime import datetime + now = datetime.now() if isinstance(time, datetime): diff = now - time @@ -340,28 +429,26 @@ def pretty_date(time): day_diff = diff.days pretty_msg = OrderedDict() - pretty_msg[0] = [(float('inf'), 1, 'from the future')] + pretty_msg[0] = [(float("inf"), 1, "from the future")] pretty_msg[1] = [ - (10, 1, "just now"), - (60, 1, "{sec:.0f} seconds ago"), - (120, 1, "a minute ago"), - (3600, 60, "{sec:.0f} minutes ago"), - (7200, 1, "an hour ago"), - (86400, 3600, "{sec:.0f} hours ago") - ] - pretty_msg[2] = [(float('inf'), 1, 'yesterday')] - pretty_msg[7] = [(float('inf'), 1, '{days:.0f} day{day_s} ago')] - pretty_msg[31] = [(float('inf'), 7, '{days:.0f} week{day_s} ago')] - pretty_msg[365] = [(float('inf'), 30, '{days:.0f} months ago')] - pretty_msg[float('inf')] = [(float('inf'), 365, '{days:.0f} year{day_s} ago')] + (10, 1, "just now"), + (60, 1, "{sec:.0f} seconds ago"), + (120, 1, "a minute ago"), + (3600, 60, "{sec:.0f} minutes ago"), + (7200, 1, "an hour ago"), + (86400, 3600, "{sec:.0f} hours ago"), + ] + pretty_msg[2] = [(float("inf"), 1, "yesterday")] + pretty_msg[7] = [(float("inf"), 1, "{days:.0f} day{day_s} ago")] + pretty_msg[31] = [(float("inf"), 7, "{days:.0f} week{day_s} ago")] + pretty_msg[365] = [(float("inf"), 30, "{days:.0f} months ago")] + pretty_msg[float("inf")] = [(float("inf"), 365, "{days:.0f} year{day_s} ago")] for days, seconds in pretty_msg.items(): if day_diff < days: for sec in seconds: if second_diff < sec[0]: return sec[2].format( - days = day_diff/sec[1], - sec = second_diff/sec[1], - day_s = 's' if day_diff/sec[1] > 1 else '' - ) - return '... time is relative anyway' + days=day_diff / sec[1], sec=second_diff / sec[1], day_s="s" if day_diff / sec[1] > 1 else "" + ) + return "... time is relative anyway" diff --git a/nf_core/modules.py b/nf_core/modules.py new file mode 100644 index 0000000000..e498b2460f --- /dev/null +++ b/nf_core/modules.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python +""" +Code to handle DSL2 module imports from a GitHub repository +""" + +from __future__ import print_function + +import base64 +import logging +import os +import requests +import sys +import tempfile + +log = logging.getLogger(__name__) + + +class ModulesRepo(object): + """ + An object to store details about the repository being used for modules. + + Used by the `nf-core modules` top-level command with -r and -b flags, + so that this can be used in the same way by all sucommands. + """ + + def __init__(self, repo="nf-core/modules", branch="master"): + self.name = repo + self.branch = branch + + +class PipelineModules(object): + def __init__(self): + """ + Initialise the PipelineModules object + """ + self.modules_repo = ModulesRepo() + self.pipeline_dir = None + self.modules_file_tree = {} + self.modules_current_hash = None + self.modules_avail_module_names = [] + + def list_modules(self): + """ + Get available module names from GitHub tree for repo + and print as list to stdout + """ + self.get_modules_file_tree() + return_str = "" + + if len(self.modules_avail_module_names) > 0: + log.info("Modules available from {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch)) + # Print results to stdout + return_str += "\n".join(self.modules_avail_module_names) + else: + log.info( + "No available modules found in {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch) + ) + return return_str + + def install(self, module): + + log.info("Installing {}".format(module)) + + # Check that we were given a pipeline + if self.pipeline_dir is None or not os.path.exists(self.pipeline_dir): + log.error("Could not find pipeline: {}".format(self.pipeline_dir)) + return False + main_nf = os.path.join(self.pipeline_dir, "main.nf") + nf_config = os.path.join(self.pipeline_dir, "nextflow.config") + if not os.path.exists(main_nf) and not os.path.exists(nf_config): + log.error("Could not find a main.nf or nextfow.config file in: {}".format(self.pipeline_dir)) + return False + + # Get the available modules + self.get_modules_file_tree() + + # Check that the supplied name is an available module + if module not in self.modules_avail_module_names: + log.error("Module '{}' not found in list of available modules.".format(module)) + log.info("Use the command 'nf-core modules list' to view available software") + return False + log.debug("Installing module '{}' at modules hash {}".format(module, self.modules_current_hash)) + + # Check that we don't already have a folder for this module + module_dir = os.path.join(self.pipeline_dir, "modules", "nf-core", "software", module) + if os.path.exists(module_dir): + log.error("Module directory already exists: {}".format(module_dir)) + log.info("To update an existing module, use the commands 'nf-core update' or 'nf-core fix'") + return False + + # Download module files + files = self.get_module_file_urls(module) + log.debug("Fetching module files:\n - {}".format("\n - ".join(files.keys()))) + for filename, api_url in files.items(): + dl_filename = os.path.join(self.pipeline_dir, "modules", "nf-core", filename) + self.download_gh_file(dl_filename, api_url) + log.info("Downloaded {} files to {}".format(len(files), module_dir)) + + def update(self, module, force=False): + log.error("This command is not yet implemented") + pass + + def remove(self, module): + log.error("This command is not yet implemented") + pass + + def check_modules(self): + log.error("This command is not yet implemented") + pass + + def get_modules_file_tree(self): + """ + Fetch the file list from the repo, using the GitHub API + + Sets self.modules_file_tree + self.modules_current_hash + self.modules_avail_module_names + """ + api_url = "https://api.github.com/repos/{}/git/trees/{}?recursive=1".format( + self.modules_repo.name, self.modules_repo.branch + ) + r = requests.get(api_url) + if r.status_code == 404: + log.error( + "Repository / branch not found: {} ({})\n{}".format( + self.modules_repo.name, self.modules_repo.branch, api_url + ) + ) + sys.exit(1) + elif r.status_code != 200: + raise SystemError( + "Could not fetch {} ({}) tree: {}\n{}".format( + self.modules_repo.name, self.modules_repo.branch, r.status_code, api_url + ) + ) + + result = r.json() + assert result["truncated"] == False + + self.modules_current_hash = result["sha"] + self.modules_file_tree = result["tree"] + for f in result["tree"]: + if f["path"].startswith("software/") and f["path"].endswith("/main.nf") and "/test/" not in f["path"]: + # remove software/ and /main.nf + self.modules_avail_module_names.append(f["path"][9:-8]) + + def get_module_file_urls(self, module): + """Fetch list of URLs for a specific module + + Takes the name of a module and iterates over the GitHub repo file tree. + Loops over items that are prefixed with the path 'software/' and ignores + anything that's not a blob. Also ignores the test/ subfolder. + + Returns a dictionary with keys as filenames and values as GitHub API URIs. + These can be used to then download file contents. + + Args: + module (string): Name of module for which to fetch a set of URLs + + Returns: + dict: Set of files and associated URLs as follows: + + { + 'software/fastqc/main.nf': 'https://api.github.com/repos/nf-core/modules/git/blobs/65ba598119206a2b851b86a9b5880b5476e263c3', + 'software/fastqc/meta.yml': 'https://api.github.com/repos/nf-core/modules/git/blobs/0d5afc23ba44d44a805c35902febc0a382b17651' + } + """ + results = {} + for f in self.modules_file_tree: + if not f["path"].startswith("software/{}".format(module)): + continue + if f["type"] != "blob": + continue + if "/test/" in f["path"]: + continue + results[f["path"]] = f["url"] + return results + + def download_gh_file(self, dl_filename, api_url): + """Download a file from GitHub using the GitHub API + + Args: + dl_filename (string): Path to save file to + api_url (string): GitHub API URL for file + + Raises: + If a problem, raises an error + """ + + # Make target directory if it doesn't already exist + dl_directory = os.path.dirname(dl_filename) + if not os.path.exists(dl_directory): + os.makedirs(dl_directory) + + # Call the GitHub API + r = requests.get(api_url) + if r.status_code != 200: + raise SystemError("Could not fetch {} file: {}\n {}".format(self.modules_repo.name, r.status_code, api_url)) + result = r.json() + file_contents = base64.b64decode(result["content"]) + + # Write the file contents + with open(dl_filename, "wb") as fh: + fh.write(file_contents) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/.dockstore.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/.dockstore.yml new file mode 100644 index 0000000000..030138a0ca --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/.dockstore.yml @@ -0,0 +1,5 @@ +# Dockstore config version, not pipeline version +version: 1.2 +workflows: + - subclass: nfl + primaryDescriptorPath: /nextflow.config diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index 3486863845..bd292ca180 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -54,4 +54,4 @@ These tests are run both with the latest available version of `Nextflow` and als ## Getting help -For further information/help, please consult the [{{ cookiecutter.name }} documentation](https://nf-co.re/{{ cookiecutter.name }}/docs) and don't hesitate to get in touch on the nf-core Slack [#{{ cookiecutter.short_name }}](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) channel ([join our Slack here](https://nf-co.re/join/slack)). +For further information/help, please consult the [{{ cookiecutter.name }} documentation](https://nf-co.re/{{ cookiecutter.short_name }}/docs) and don't hesitate to get in touch on the nf-core Slack [#{{ cookiecutter.short_name }}](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) channel ([join our Slack here](https://nf-co.re/join/slack)). diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md index a732734304..3b98a62ca1 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,24 +1,27 @@ + -## Describe the bug +## Description of the bug -A clear and concise description of what the bug is. + ## Steps to reproduce Steps to reproduce the behaviour: -1. Command line: `nextflow run ...` -2. See error: _Please provide your error message_ +1. Command line: +2. See error: ## Expected behaviour -A clear and concise description of what you expected to happen. + ## System @@ -39,4 +42,4 @@ A clear and concise description of what you expected to happen. ## Additional context -Add any other context about the problem here. + diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/feature_request.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/feature_request.md index 148df5999c..199fb5d52b 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/feature_request.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/feature_request.md @@ -1,24 +1,26 @@ + ## Is your feature request related to a problem? Please describe -A clear and concise description of what the problem is. + -Ex. I'm always frustrated when [...] + ## Describe the solution you'd like -A clear and concise description of what you want to happen. + ## Describe alternatives you've considered -A clear and concise description of any alternative solutions or features you've considered. + ## Additional context -Add any other context about the feature request here. + diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md index 3143db9604..25f24d6c1f 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md @@ -1,3 +1,4 @@ + + ## PR checklist - [ ] This comment contains a description of changes (with reason) +- [ ] `CHANGELOG.md` is updated - [ ] If you've fixed a bug or added code that should be tested, add tests! -- [ ] If necessary, also make a PR on the [{{ cookiecutter.name }} branch on the nf-core/test-datasets repo](https://github.com/nf-core/test-datasets/pull/new/{{ cookiecutter.name }}) -- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`). -- [ ] Make sure your code lints (`nf-core lint .`). - [ ] Documentation in `docs` is updated -- [ ] `CHANGELOG.md` is updated -- [ ] `README.md` is updated - -**Learn more about contributing:** [CONTRIBUTING.md](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) \ No newline at end of file +- [ ] If necessary, also make a PR on the [{{ cookiecutter.name }} branch on the nf-core/test-datasets repo](https://github.com/nf-core/test-datasets/pull/new/{{ cookiecutter.name }}) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml new file mode 100644 index 0000000000..9da1209356 --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml @@ -0,0 +1,40 @@ +name: nf-core AWS full size tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test_full' on AWS batch + +on: + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS full tests + if: github.repository == '{{ cookiecutter.name }}' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + # TODO nf-core: You can customise AWS full pipeline tests as required + # Add full size test data (but still relatively small datasets for few samples) + # on the `test_full.config` test runs with only one set of parameters + # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command + {% raw %}env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}{% endraw %} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-{{ cookiecutter.short_name }} \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml new file mode 100644 index 0000000000..6a2759edbe --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml @@ -0,0 +1,40 @@ +name: nf-core AWS test +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test' on AWS batch + +on: + push: + branches: + - master + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == '{{ cookiecutter.name }}' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + # TODO nf-core: You can customise CI pipeline run tests as required + # For example: adding multiple test runs with different parameters + # Remember that you can parallelise this by using strategy.matrix + {% raw %}env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}{% endraw %} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-{{ cookiecutter.short_name }} \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 1018f6d253..94ec0a87ca 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -3,14 +3,34 @@ name: nf-core branch protection # It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev` on: pull_request: - branches: - - master + branches: [master] jobs: test: - runs-on: ubuntu-18.04 + runs-on: ubuntu-latest steps: - # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches - name: Check PRs + if: github.repository == '{{cookiecutter.name}}' run: | - { [[ $(git remote get-url origin) == *{{cookiecutter.name}} ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + { [[ {% raw %}${{github.event.pull_request.head.repo.full_name}}{% endraw %} == {{cookiecutter.name}} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + +{% raw %} + # If the above check failed, post a comment on the PR explaining the failure + - name: Post PR comment + if: failure() + uses: mshick/add-pr-comment@v1 + with: + message: | + Hi @${{ github.event.pull_request.user.login }}, + + It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. + The `master` branch on nf-core repositories should always contain code from the latest release. + Because of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch. + + You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page. + + Thanks again for your contribution! + repo-token: ${{ secrets.GITHUB_TOKEN }} + allow-repeats: false +{% endraw %} diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml index 7701e5f3b5..ce1d565692 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml @@ -1,30 +1,55 @@ name: nf-core CI -# This workflow is triggered on pushes and PRs to the repository. -# It runs the pipeline with the minimal test dataset to check that it completes without any syntax errors -on: [push, pull_request] +# This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors +on: + push: + branches: + - dev + pull_request: + release: + types: [published] jobs: test: + name: Run workflow tests + # Only run on push if this is the nf-core dev branch (merged PRs) + if: {% raw %}${{{% endraw %} github.event_name != 'push' || (github.event_name == 'push' && github.repository == '{{ cookiecutter.name }}') {% raw %}}}{% endraw %} + runs-on: ubuntu-latest env: NXF_VER: {% raw %}${{ matrix.nxf_ver }}{% endraw %} NXF_ANSI_LOG: false - runs-on: ubuntu-latest strategy: matrix: # Nextflow versions: check pipeline minimum and current latest nxf_ver: ['19.10.0', ''] steps: - - uses: actions/checkout@v2 - - name: Install Nextflow - run: | - wget -qO- get.nextflow.io | bash - sudo mv nextflow /usr/local/bin/ + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Check if Dockerfile or Conda environment changed + uses: technote-space/get-diff-action@v1 + with: + PREFIX_FILTER: | + Dockerfile + environment.yml + + - name: Build new docker image + if: env.GIT_DIFF + run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:dev + - name: Pull docker image + if: {% raw %}${{ !env.GIT_DIFF }}{% endraw %} run: | docker pull {{ cookiecutter.name_docker }}:dev docker tag {{ cookiecutter.name_docker }}:dev {{ cookiecutter.name_docker }}:dev + + - name: Install Nextflow + run: | + wget -qO- get.nextflow.io | bash + sudo mv nextflow /usr/local/bin/ + - name: Run pipeline with test data + # TODO nf-core: You can customise CI pipeline run tests as required + # For example: adding multiple test runs with different parameters + # Remember that you can parallelise this by using strategy.matrix run: | - # TODO nf-core: You can customise CI pipeline run tests as required - # (eg. adding multiple test runs with different parameters) nextflow run ${GITHUB_WORKSPACE} -profile test,docker diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml index 1e0827a800..d10a057a9b 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml @@ -33,18 +33,29 @@ jobs: nf-core: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + + - name: Check out pipeline code + uses: actions/checkout@v2 + - name: Install Nextflow run: | wget -qO- get.nextflow.io | bash sudo mv nextflow /usr/local/bin/ + - uses: actions/setup-python@v1 with: python-version: '3.6' architecture: 'x64' + - name: Install dependencies run: | python -m pip install --upgrade pip pip install nf-core +{% raw %} - name: Run nf-core lint + env: + GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }} run: nf-core lint ${GITHUB_WORKSPACE} +{% endraw %} diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml new file mode 100644 index 0000000000..65079085c5 --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml @@ -0,0 +1,39 @@ +name: nf-core Docker push +# This builds the docker image and pushes it to DockerHub +# Runs on nf-core repo releases and push event to 'dev' branch (PR merges) +on: + push: + branches: + - dev + release: + types: [published] + +push_dockerhub: + name: Push new Docker image to Docker Hub + runs-on: ubuntu-latest + # Only run for the nf-core repo, for releases and merged PRs + if: {% raw %}${{{% endraw %} github.repository == '{{ cookiecutter.name }}' {% raw %}}}{% endraw %} + env: + DOCKERHUB_USERNAME: {% raw %}${{ secrets.DOCKERHUB_USERNAME }}{% endraw %} + DOCKERHUB_PASS: {% raw %}${{ secrets.DOCKERHUB_PASS }}{% endraw %} + steps: + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Build new docker image + run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:latest + + - name: Push Docker image to DockerHub (dev) + if: {% raw %}${{ github.event_name == 'push' }}{% endraw %} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:dev + docker push {{ cookiecutter.name_docker }}:dev + + - name: Push Docker image to DockerHub (release) + if: {% raw %}${{ github.event_name == 'release' }}{% endraw %} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker push {{ cookiecutter.name_docker }}:latest + docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:{% raw %}${{ github.event.release.tag_name }}{% endraw %} + docker push {{ cookiecutter.name_docker }}:{% raw %}${{ github.event.release.tag_name }}{% endraw %} diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.gitignore b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.gitignore index 6354f3708f..aa4bb5b375 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.gitignore +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.gitignore @@ -5,4 +5,5 @@ results/ .DS_Store tests/ testing/ +testing* *.pyc diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CHANGELOG.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CHANGELOG.md index fcbbfe48f9..b401036075 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CHANGELOG.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CHANGELOG.md @@ -1,11 +1,11 @@ # {{ cookiecutter.name }}: Changelog -The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) -and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## v{{ cookiecutter.version }} - [date] -Initial release of {{ cookiecutter.name }}, created with the [nf-core](http://nf-co.re/) template. +Initial release of {{ cookiecutter.name }}, created with the [nf-core](https://nf-co.re/) template. ### `Added` diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md index cf930c8acf..405fb1bfd7 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md @@ -40,7 +40,7 @@ Project maintainers who do not follow or enforce the Code of Conduct in good fai ## Attribution -This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://www.contributor-covenant.org/version/1/4/code-of-conduct/][version] -[homepage]: http://contributor-covenant.org -[version]: http://contributor-covenant.org/version/1/4/ +[homepage]: https://contributor-covenant.org +[version]: https://www.contributor-covenant.org/version/1/4/code-of-conduct/ diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile index 69d24cdc31..bd07987b6e 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile @@ -4,10 +4,14 @@ LABEL authors="{{ cookiecutter.author }}" \ # Install the conda environment COPY environment.yml / -RUN conda env create -f /environment.yml && conda clean -a +RUN conda env create --quiet -f /environment.yml && conda clean -a # Add conda installation dir to PATH (instead of doing 'conda activate') ENV PATH /opt/conda/envs/{{ cookiecutter.name_noslash }}-{{ cookiecutter.version }}/bin:$PATH # Dump the details of the installed packages to a file for posterity RUN conda env export --name {{ cookiecutter.name_noslash }}-{{ cookiecutter.version }} > {{ cookiecutter.name_noslash }}-{{ cookiecutter.version }}.yml + +# Instruct R processes to use these empty files instead of clashing with a local version +RUN touch .Rprofile +RUN touch .Renviron diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index bcae7fc382..169587892b 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -6,8 +6,9 @@ [![GitHub Actions Linting Status](https://github.com/{{ cookiecutter.name }}/workflows/nf-core%20linting/badge.svg)](https://github.com/{{ cookiecutter.name }}/actions) [![Nextflow](https://img.shields.io/badge/nextflow-%E2%89%A519.10.0-brightgreen.svg)](https://www.nextflow.io/) -[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/) +[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) [![Docker](https://img.shields.io/docker/automated/{{ cookiecutter.name_docker }}.svg)](https://hub.docker.com/r/{{ cookiecutter.name_docker }}) +[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23{{ cookiecutter.short_name }}-4A154B?logo=slack)](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) ## Introduction @@ -15,40 +16,31 @@ The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool ## Quick Start -i. Install [`nextflow`](https://nf-co.re/usage/installation) +1. Install [`nextflow`](https://nf-co.re/usage/installation) -ii. Install either [`Docker`](https://docs.docker.com/engine/installation/) or [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) for full pipeline reproducibility (please only use [`Conda`](https://conda.io/miniconda.html) as a last resort; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles)) +2. Install either [`Docker`](https://docs.docker.com/engine/installation/) or [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) for full pipeline reproducibility _(please only use [`Conda`](https://conda.io/miniconda.html) as a last resort; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))_ -iii. Download the pipeline and test it on a minimal dataset with a single command +3. Download the pipeline and test it on a minimal dataset with a single command: -```bash -nextflow run {{ cookiecutter.name }} -profile test, -``` + ```bash + nextflow run {{ cookiecutter.name }} -profile test, + ``` -> Please check [nf-core/configs](https://github.com/nf-core/configs#documentation) to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use `-profile ` in your command. This will enable either `docker` or `singularity` and set the appropriate execution settings for your local compute environment. + > Please check [nf-core/configs](https://github.com/nf-core/configs#documentation) to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use `-profile ` in your command. This will enable either `docker` or `singularity` and set the appropriate execution settings for your local compute environment. -iv. Start running your own analysis! +4. Start running your own analysis! - + -```bash -nextflow run {{ cookiecutter.name }} -profile --reads '*_R{1,2}.fastq.gz' --genome GRCh37 -``` + ```bash + nextflow run {{ cookiecutter.name }} -profile --input '*_R{1,2}.fastq.gz' --genome GRCh37 + ``` See [usage docs](docs/usage.md) for all of the available options when running the pipeline. ## Documentation -The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline, found in the `docs/` directory: - -1. [Installation](https://nf-co.re/usage/installation) -2. Pipeline configuration - * [Local installation](https://nf-co.re/usage/local_installation) - * [Adding your own system config](https://nf-co.re/usage/adding_own_config) - * [Reference genomes](https://nf-co.re/usage/reference_genomes) -3. [Running the pipeline](docs/usage.md) -4. [Output and how to interpret the results](docs/output.md) -5. [Troubleshooting](https://nf-co.re/usage/troubleshooting) +The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline which you can read at [https://nf-core/{{ cookiecutter.short_name }}/docs](https://nf-core/{{ cookiecutter.short_name }}/docs) or find in the [`docs/` directory](docs). @@ -60,7 +52,7 @@ The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline If you would like to contribute to this pipeline, please see the [contributing guidelines](.github/CONTRIBUTING.md). -For further information or help, don't hesitate to get in touch on [Slack](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) (you can join with [this invite](https://nf-co.re/join/slack)). +For further information or help, don't hesitate to get in touch on the [Slack `#{{ cookiecutter.short_name }}` channel](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) (you can join with [this invite](https://nf-co.re/join/slack)). ## Citation @@ -73,5 +65,5 @@ You can cite the `nf-core` publication as follows: > > Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. > -> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). +> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). > ReadCube: [Full Access Link](https://rdcu.be/b1GjZ) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py index 57cc4263fe..a26d1ff5e6 100755 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py @@ -4,33 +4,23 @@ import markdown import os import sys +import io + def convert_markdown(in_fn): - input_md = open(in_fn, mode="r", encoding="utf-8").read() + input_md = io.open(in_fn, mode="r", encoding="utf-8").read() html = markdown.markdown( "[TOC]\n" + input_md, - extensions = [ - 'pymdownx.extra', - 'pymdownx.b64', - 'pymdownx.highlight', - 'pymdownx.emoji', - 'pymdownx.tilde', - 'toc' - ], - extension_configs = { - 'pymdownx.b64': { - 'base_path': os.path.dirname(in_fn) - }, - 'pymdownx.highlight': { - 'noclasses': True - }, - 'toc': { - 'title': 'Table of Contents' - } - } + extensions=["pymdownx.extra", "pymdownx.b64", "pymdownx.highlight", "pymdownx.emoji", "pymdownx.tilde", "toc"], + extension_configs={ + "pymdownx.b64": {"base_path": os.path.dirname(in_fn)}, + "pymdownx.highlight": {"noclasses": True}, + "toc": {"title": "Table of Contents"}, + }, ) return html + def wrap_html(contents): header = """ @@ -83,18 +73,19 @@ def wrap_html(contents): def parse_args(args=None): parser = argparse.ArgumentParser() - parser.add_argument('mdfile', type=argparse.FileType('r'), nargs='?', - help='File to convert. Defaults to stdin.') - parser.add_argument('-o', '--out', type=argparse.FileType('w'), - default=sys.stdout, - help='Output file name. Defaults to stdout.') + parser.add_argument("mdfile", type=argparse.FileType("r"), nargs="?", help="File to convert. Defaults to stdin.") + parser.add_argument( + "-o", "--out", type=argparse.FileType("w"), default=sys.stdout, help="Output file name. Defaults to stdout." + ) return parser.parse_args(args) + def main(args=None): args = parse_args(args) converted_md = convert_markdown(args.mdfile.name) html = wrap_html(converted_md) args.out.write(html) -if __name__ == '__main__': + +if __name__ == "__main__": sys.exit(main()) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/scrape_software_versions.py b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/scrape_software_versions.py index 9f5a6a64a3..a6d687ce1b 100755 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/scrape_software_versions.py +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/scrape_software_versions.py @@ -5,16 +5,16 @@ # TODO nf-core: Add additional regexes for new tools in process get_software_versions regexes = { - '{{ cookiecutter.name }}': ['v_pipeline.txt', r"(\S+)"], - 'Nextflow': ['v_nextflow.txt', r"(\S+)"], - 'FastQC': ['v_fastqc.txt', r"FastQC v(\S+)"], - 'MultiQC': ['v_multiqc.txt', r"multiqc, version (\S+)"], + "{{ cookiecutter.name }}": ["v_pipeline.txt", r"(\S+)"], + "Nextflow": ["v_nextflow.txt", r"(\S+)"], + "FastQC": ["v_fastqc.txt", r"FastQC v(\S+)"], + "MultiQC": ["v_multiqc.txt", r"multiqc, version (\S+)"], } results = OrderedDict() -results['{{ cookiecutter.name }}'] = 'N/A' -results['Nextflow'] = 'N/A' -results['FastQC'] = 'N/A' -results['MultiQC'] = 'N/A' +results["{{ cookiecutter.name }}"] = 'N/A' +results["Nextflow"] = 'N/A' +results["FastQC"] = 'N/A' +results["MultiQC"] = 'N/A' # Search each file using its regex for k, v in regexes.items(): @@ -30,10 +30,11 @@ # Remove software set to false in results for k in list(results): if not results[k]: - del(results[k]) + del results[k] # Dump to YAML -print (''' +print( + """ id: 'software_versions' section_name: '{{ cookiecutter.name }} Software Versions' section_href: 'https://github.com/{{ cookiecutter.name }}' @@ -41,12 +42,13 @@ description: 'are collected at run time from the software output.' data: |
-''') -for k,v in results.items(): - print("
{}
{}
".format(k,v)) -print ("
") +""" +) +for k, v in results.items(): + print("
{}
{}
".format(k, v)) +print(" ") # Write out regexes as csv file: -with open('software_versions.csv', 'w') as f: - for k,v in results.items(): - f.write("{}\t{}\n".format(k,v)) +with open("software_versions.csv", "w") as f: + for k, v in results.items(): + f.write("{}\t{}\n".format(k, v)) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config index 2de924228f..caeafceb25 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config @@ -340,6 +340,7 @@ params { gtf = "${params.igenomes_base}/Danio_rerio/UCSC/danRer10/Annotation/Genes/genes.gtf" bed12 = "${params.igenomes_base}/Danio_rerio/UCSC/danRer10/Annotation/Genes/genes.bed" mito_name = "chrM" + macs_gsize = "1.37e9" } 'dm6' { fasta = "${params.igenomes_base}/Drosophila_melanogaster/UCSC/dm6/Sequence/WholeGenomeFasta/genome.fa" diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test.config index 1d54fac2c9..7840d28846 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test.config @@ -19,7 +19,7 @@ params { // TODO nf-core: Specify the paths to your test data on nf-core/test-datasets // TODO nf-core: Give any required params for the test so that command line flags are not needed single_end = false - readPaths = [ + input_paths = [ ['Testdata', ['https://github.com/nf-core/test-datasets/raw/exoseq/testdata/Testdata_R1.tiny.fastq.gz', 'https://github.com/nf-core/test-datasets/raw/exoseq/testdata/Testdata_R2.tiny.fastq.gz']], ['SRR389222', ['https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub1.fastq.gz', 'https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub2.fastq.gz']] ] diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test_full.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test_full.config new file mode 100644 index 0000000000..170c6bd4fc --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test_full.config @@ -0,0 +1,22 @@ +/* + * ------------------------------------------------- + * Nextflow config file for running full-size tests + * ------------------------------------------------- + * Defines bundled input files and everything required + * to run a full size pipeline test. Use as follows: + * nextflow run {{ cookiecutter.name }} -profile test_full, + */ + +params { + config_profile_name = 'Full test profile' + config_profile_description = 'Full test dataset to check pipeline function' + + // Input data for full size test + // TODO nf-core: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA) + // TODO nf-core: Give any required params for the test so that command line flags are not needed + single_end = false + readPaths = [ + ['Testdata', ['https://github.com/nf-core/test-datasets/raw/exoseq/testdata/Testdata_R1.tiny.fastq.gz', 'https://github.com/nf-core/test-datasets/raw/exoseq/testdata/Testdata_R2.tiny.fastq.gz']], + ['SRR389222', ['https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub1.fastq.gz', 'https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub2.fastq.gz']] + ] +} diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/README.md index 7dd10924f9..ef2bb5200a 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/README.md @@ -1,12 +1,12 @@ # {{ cookiecutter.name }}: Documentation -The {{ cookiecutter.name }} documentation is split into the following files: - -1. [Installation](https://nf-co.re/usage/installation) -2. Pipeline configuration - * [Local installation](https://nf-co.re/usage/local_installation) - * [Adding your own system config](https://nf-co.re/usage/adding_own_config) - * [Reference genomes](https://nf-co.re/usage/reference_genomes) -3. [Running the pipeline](usage.md) -4. [Output and how to interpret the results](output.md) -5. [Troubleshooting](https://nf-co.re/usage/troubleshooting) +The {{ cookiecutter.name }} documentation is split into the following pages: + + + +* [Usage](usage.md) + * An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. +* [Output](output.md) + * An overview of the different results produced by the pipeline and how to interpret them. + +You can find a lot more documentation about installing, configuring and running nf-core pipelines on the website: [https://nf-co.re](https://nf-co.re) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md index f6bfa82bf7..4722747c35 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md @@ -2,6 +2,8 @@ This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. +The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. + ## Pipeline overview @@ -9,35 +11,47 @@ This document describes the output produced by the pipeline. Most of the plots a The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps: -* [FastQC](#fastqc) - read quality control -* [MultiQC](#multiqc) - aggregate report, describing results of the whole pipeline +* [FastQC](#fastqc) - Read quality control +* [MultiQC](#multiqc) - Aggregate report describing results from the whole pipeline +* [Pipeline information](#pipeline-information) - Report metrics generated during the workflow execution ## FastQC -[FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your reads. It provides information about the quality score distribution across your reads, the per base sequence content (%T/A/G/C). You get information about adapter contamination and other overrepresented sequences. +[FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your sequenced reads. It provides information about the quality score distribution across your reads, per base sequence content (%A/T/G/C), adapter contamination and overrepresented sequences. -For further reading and documentation see the [FastQC help](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/). +For further reading and documentation see the [FastQC help pages](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/). -> **NB:** The FastQC plots displayed in the MultiQC report shows _untrimmed_ reads. They may contain adapter sequence and potentially regions with low quality. To see how your reads look after trimming, look at the FastQC reports in the `trim_galore` directory. +**Output files:** -**Output directory: `results/fastqc`** +* `fastqc/` + * `*_fastqc.html`: FastQC report containing quality metrics for your untrimmed raw fastq files. +* `fastqc/zips/` + * `*_fastqc.zip`: Zip archive containing the FastQC report, tab-delimited data file and plot images. -* `sample_fastqc.html` - * FastQC report, containing quality metrics for your untrimmed raw fastq files -* `zips/sample_fastqc.zip` - * zip file containing the FastQC report, tab-delimited data file and plot images +> **NB:** The FastQC plots displayed in the MultiQC report shows _untrimmed_ reads. They may contain adapter sequence and potentially regions with low quality. ## MultiQC -[MultiQC](http://multiqc.info) is a visualisation tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in within the report data directory. +[MultiQC](http://multiqc.info) is a visualization tool that generates a single HTML report summarizing all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. + +The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. + +For more information about how to use MultiQC reports, see [https://multiqc.info](https://multiqc.info). + +**Output files:** + +* `multiqc/` + * `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. + * `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. + * `multiqc_plots/`: directory containing static images from the report in various formats. -The pipeline has special steps which allow the software versions used to be reported in the MultiQC output for future traceability. +## Pipeline information -**Output directory: `results/multiqc`** +[Nextflow](https://www.nextflow.io/docs/latest/tracing.html) provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage. -* `Project_multiqc_report.html` - * MultiQC report - a standalone HTML file that can be viewed in your web browser -* `Project_multiqc_data/` - * Directory containing parsed statistics from the different tools used in the pipeline +**Output files:** -For more information about how to use MultiQC reports, see [http://multiqc.info](http://multiqc.info) +* `pipeline_info/` + * Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. + * Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.csv`. + * Documentation for interpretation of results in HTML format: `results_description.html`. diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index c665f10518..3517a308c9 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -1,62 +1,15 @@ # {{ cookiecutter.name }}: Usage -## Table of contents - -* [Table of contents](#table-of-contents) -* [Introduction](#introduction) -* [Running the pipeline](#running-the-pipeline) - * [Updating the pipeline](#updating-the-pipeline) - * [Reproducibility](#reproducibility) -* [Main arguments](#main-arguments) - * [`-profile`](#-profile) - * [`--reads`](#--reads) - * [`--single_end`](#--single_end) -* [Reference genomes](#reference-genomes) - * [`--genome` (using iGenomes)](#--genome-using-igenomes) - * [`--fasta`](#--fasta) - * [`--igenomes_ignore`](#--igenomes_ignore) -* [Job resources](#job-resources) - * [Automatic resubmission](#automatic-resubmission) - * [Custom resource requests](#custom-resource-requests) -* [AWS Batch specific parameters](#aws-batch-specific-parameters) - * [`--awsqueue`](#--awsqueue) - * [`--awsregion`](#--awsregion) - * [`--awscli`](#--awscli) -* [Other command line parameters](#other-command-line-parameters) - * [`--outdir`](#--outdir) - * [`--email`](#--email) - * [`--email_on_fail`](#--email_on_fail) - * [`--max_multiqc_email_size`](#--max_multiqc_email_size) - * [`-name`](#-name) - * [`-resume`](#-resume) - * [`-c`](#-c) - * [`--custom_config_version`](#--custom_config_version) - * [`--custom_config_base`](#--custom_config_base) - * [`--max_memory`](#--max_memory) - * [`--max_time`](#--max_time) - * [`--max_cpus`](#--max_cpus) - * [`--plaintext_email`](#--plaintext_email) - * [`--monochrome_logs`](#--monochrome_logs) - * [`--multiqc_config`](#--multiqc_config) - ## Introduction -Nextflow handles job submissions on SLURM or other environments, and supervises running the jobs. Thus the Nextflow process must run until the pipeline is finished. We recommend that you put the process running in the background through `screen` / `tmux` or similar tool. Alternatively you can run nextflow within a cluster job submitted your job scheduler. - -It is recommended to limit the Nextflow Java virtual machines memory. We recommend adding the following line to your environment (typically in `~/.bashrc` or `~./bash_profile`): - -```bash -NXF_OPTS='-Xms1g -Xmx4g' -``` - - + ## Running the pipeline The typical command for running the pipeline is as follows: ```bash -nextflow run {{ cookiecutter.name }} --reads '*_R{1,2}.fastq.gz' -profile docker +nextflow run {{ cookiecutter.name }} --input '*_R{1,2}.fastq.gz' -profile docker ``` This will launch the pipeline with the `docker` configuration profile. See below for more information about profiles. @@ -86,7 +39,9 @@ First, go to the [{{ cookiecutter.name }} releases page](https://github.com/{{ c This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. -## Main arguments +## Core Nextflow arguments + +> **NB:** These options are part of Nextflow and use a _single_ hyphen (pipeline parameters use a double-hyphen). ### `-profile` @@ -104,11 +59,11 @@ They are loaded in sequence, so later profiles can overwrite earlier profiles. If `-profile` is not specified, the pipeline will run locally and expect all software to be installed and available on the `PATH`. This is _not_ recommended. * `docker` - * A generic configuration profile to be used with [Docker](http://docker.com/) - * Pulls software from dockerhub: [`{{ cookiecutter.name_docker }}`](http://hub.docker.com/r/{{ cookiecutter.name_docker }}/) + * A generic configuration profile to be used with [Docker](https://docker.com/) + * Pulls software from Docker Hub: [`{{ cookiecutter.name_docker }}`](https://hub.docker.com/r/{{ cookiecutter.name_docker }}/) * `singularity` - * A generic configuration profile to be used with [Singularity](http://singularity.lbl.gov/) - * Pulls software from DockerHub: [`{{ cookiecutter.name_docker }}`](http://hub.docker.com/r/{{ cookiecutter.name_docker }}/) + * A generic configuration profile to be used with [Singularity](https://sylabs.io/docs/) + * Pulls software from Docker Hub: [`{{ cookiecutter.name_docker }}`](https://hub.docker.com/r/{{ cookiecutter.name_docker }}/) * `conda` * Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker or Singularity. * A generic configuration profile to be used with [Conda](https://conda.io/docs/) @@ -117,215 +72,50 @@ If `-profile` is not specified, the pipeline will run locally and expect all sof * A profile with a complete configuration for automated testing * Includes links to test data so needs no other parameters - - -### `--reads` - -Use this to specify the location of your input FastQ files. For example: - -```bash ---reads 'path/to/data/sample_*_{1,2}.fastq' -``` - -Please note the following requirements: - -1. The path must be enclosed in quotes -2. The path must have at least one `*` wildcard character -3. When using the pipeline with paired end data, the path must use `{1,2}` notation to specify read pairs. - -If left unspecified, a default pattern is used: `data/*{1,2}.fastq.gz` - -### `--single_end` - -By default, the pipeline expects paired-end data. If you have single-end data, you need to specify `--single_end` on the command line when you launch the pipeline. A normal glob pattern, enclosed in quotation marks, can then be used for `--reads`. For example: - -```bash ---single_end --reads '*.fastq' -``` - -It is not possible to run a mixture of single-end and paired-end files in one run. - -## Reference genomes - -The pipeline config files come bundled with paths to the illumina iGenomes reference index files. If running with docker or AWS, the configuration is set up to use the [AWS-iGenomes](https://ewels.github.io/AWS-iGenomes/) resource. - -### `--genome` (using iGenomes) +### `-resume` -There are 31 different species supported in the iGenomes references. To run the pipeline, you must specify which to use with the `--genome` flag. +Specify this when restarting a pipeline. Nextflow will used cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. -You can find the keys to specify the genomes in the [iGenomes config file](../conf/igenomes.config). Common genomes that are supported are: +You can also supply a run name to resume a specific run: `-resume [run-name]`. Use the `nextflow log` command to show previous run names. -* Human - * `--genome GRCh37` -* Mouse - * `--genome GRCm38` -* _Drosophila_ - * `--genome BDGP6` -* _S. cerevisiae_ - * `--genome 'R64-1-1'` +### `-c` -> There are numerous others - check the config file for more. +Specify the path to a specific config file (this is a core NextFlow command). See the [nf-core website documentation](https://nf-co.re/usage/configuration) for more information. -Note that you can use the same configuration setup to save sets of reference files for your own use, even if they are not part of the iGenomes resource. See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for instructions on where to save such a file. +#### Custom resource requests -The syntax for this reference configuration is as follows: +Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with an error code of `143` (exceeded requested resources) it will automatically resubmit with higher requests (2 x original, then 3 x original). If it still fails after three times then the pipeline is stopped. - +Whilst these default requirements will hopefully work for most people with most data, you may find that you want to customise the compute resources that the pipeline requests. You can do this by creating a custom config file. For example, to give the workflow process `star` 32GB of memory, you could use the following config: ```nextflow -params { - genomes { - 'GRCh37' { - fasta = '' // Used if no star index given - } - // Any number of additional genomes, key is used with --genome +process { + withName: star { + memory = 32.GB } } ``` - - -### `--fasta` - -If you prefer, you can specify the full path to your reference genome when you run the pipeline: - -```bash ---fasta '[path to Fasta reference]' -``` - -### `--igenomes_ignore` - -Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`. - -## Job resources - -### Automatic resubmission - -Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with an error code of `143` (exceeded requested resources) it will automatically resubmit with higher requests (2 x original, then 3 x original). If it still fails after three times then the pipeline is stopped. - -### Custom resource requests - -Wherever process-specific requirements are set in the pipeline, the default value can be changed by creating a custom config file. See the files hosted at [`nf-core/configs`](https://github.com/nf-core/configs/tree/master/conf) for examples. +See the main [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for more information. If you are likely to be running `nf-core` pipelines regularly it may be a good idea to request that your custom config file is uploaded to the `nf-core/configs` git repository. Before you do this please can you test that the config file works with your pipeline of choice using the `-c` parameter (see definition below). You can then create a pull request to the `nf-core/configs` repository with the addition of your config file, associated documentation file (see examples in [`nf-core/configs/docs`](https://github.com/nf-core/configs/tree/master/docs)), and amending [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) to include your custom profile. -If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack). - -## AWS Batch specific parameters - -Running the pipeline on AWS Batch requires a couple of specific parameters to be set according to your AWS Batch configuration. Please use [`-profile awsbatch`](https://github.com/nf-core/configs/blob/master/conf/awsbatch.config) and then specify all of the following parameters. - -### `--awsqueue` - -The JobQueue that you intend to use on AWS Batch. +If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack) on the [`#configs` channel](https://nfcore.slack.com/channels/configs). -### `--awsregion` +### Running in the background -The AWS region in which to run your job. Default is set to `eu-west-1` but can be adjusted to your needs. +Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. -### `--awscli` +The Nextflow `-bg` flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. -The [AWS CLI](https://www.nextflow.io/docs/latest/awscloud.html#aws-cli-installation) path in your custom AMI. Default: `/home/ec2-user/miniconda/bin/aws`. +Alternatively, you can use `screen` / `tmux` or similar tool to create a detached session which you can log back into at a later time. +Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs). -Please make sure to also set the `-w/--work-dir` and `--outdir` parameters to a S3 storage bucket of your choice - you'll get an error message notifying you if you didn't. +#### Nextflow memory requirements -## Other command line parameters - - - -### `--outdir` - -The output directory where the results will be saved. - -### `--email` - -Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run. - -### `--email_on_fail` - -This works exactly as with `--email`, except emails are only sent if the workflow is not successful. - -### `--max_multiqc_email_size` - -Threshold size for MultiQC report to be attached in notification email. If file generated by pipeline exceeds the threshold, it will not be attached (Default: 25MB). - -### `-name` - -Name for the pipeline run. If not specified, Nextflow will automatically generate a random mnemonic. - -This is used in the MultiQC report (if not default) and in the summary HTML / e-mail (always). - -**NB:** Single hyphen (core Nextflow option) - -### `-resume` - -Specify this when restarting a pipeline. Nextflow will used cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. - -You can also supply a run name to resume a specific run: `-resume [run-name]`. Use the `nextflow log` command to show previous run names. - -**NB:** Single hyphen (core Nextflow option) - -### `-c` - -Specify the path to a specific config file (this is a core NextFlow command). - -**NB:** Single hyphen (core Nextflow option) - -Note - you can use this to override pipeline defaults. - -### `--custom_config_version` - -Provide git commit id for custom Institutional configs hosted at `nf-core/configs`. This was implemented for reproducibility purposes. Default: `master`. - -```bash -## Download and use config file with following git commid id ---custom_config_version d52db660777c4bf36546ddb188ec530c3ada1b96 -``` - -### `--custom_config_base` - -If you're running offline, nextflow will not be able to fetch the institutional config files -from the internet. If you don't need them, then this is not a problem. If you do need them, -you should download the files from the repo and tell nextflow where to find them with the -`custom_config_base` option. For example: +In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. +We recommend adding the following line to your environment to limit this (typically in `~/.bashrc` or `~./bash_profile`): ```bash -## Download and unzip the config files -cd /path/to/my/configs -wget https://github.com/nf-core/configs/archive/master.zip -unzip master.zip - -## Run the pipeline -cd /path/to/my/data -nextflow run /path/to/pipeline/ --custom_config_base /path/to/my/configs/configs-master/ +NXF_OPTS='-Xms1g -Xmx4g' ``` - -> Note that the nf-core/tools helper package has a `download` command to download all required pipeline -> files + singularity containers + institutional configs in one go for you, to make this process easier. - -### `--max_memory` - -Use to set a top-limit for the default memory requirement for each process. -Should be a string in the format integer-unit. eg. `--max_memory '8.GB'` - -### `--max_time` - -Use to set a top-limit for the default time requirement for each process. -Should be a string in the format integer-unit. eg. `--max_time '2.h'` - -### `--max_cpus` - -Use to set a top-limit for the default CPU requirement for each process. -Should be a string in the format integer-unit. eg. `--max_cpus 1` - -### `--plaintext_email` - -Set to receive plain-text e-mails instead of HTML formatted. - -### `--monochrome_logs` - -Set to disable colourful command line output and live life in monochrome. - -### `--multiqc_config` - -Specify a path to a custom MultiQC configuration file. diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 8e0f77b206..715d15aaea 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -18,12 +18,12 @@ def helpMessage() { The typical command for running the pipeline is as follows: - nextflow run {{ cookiecutter.name }} --reads '*_R{1,2}.fastq.gz' -profile docker + nextflow run {{ cookiecutter.name }} --input '*_R{1,2}.fastq.gz' -profile docker Mandatory arguments: - --reads [file] Path to input data (must be surrounded with quotes) - -profile [str] Configuration profile to use. Can use multiple (comma separated) - Available: conda, docker, singularity, test, awsbatch, and more + --input [file] Path to input data (must be surrounded with quotes) + -profile [str] Configuration profile to use. Can use multiple (comma separated) + Available: conda, docker, singularity, test, awsbatch, and more Options: --genome [str] Name of iGenomes reference @@ -34,9 +34,10 @@ def helpMessage() { Other options: --outdir [file] The output directory where the results will be saved + --publish_dir_mode [str] Mode for publishing results in the output directory. Available: symlink, rellink, link, copy, copyNoFollow, move (Default: copy) --email [email] Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits --email_on_fail [email] Same as --email, except only send mail if the workflow is not successful - --max_multiqc_email_size [str] Theshold size for MultiQC report to be attached in notification email. If file generated by pipeline exceeds the threshold, it will not be attached (Default: 25MB) + --max_multiqc_email_size [str] Threshold size for MultiQC report to be attached in notification email. If file generated by pipeline exceeds the threshold, it will not be attached (Default: 25MB) -name [str] Name for the pipeline run. If not specified, Nextflow will automatically generate a random mnemonic AWSBatch options: @@ -73,12 +74,13 @@ params.fasta = params.genome ? params.genomes[ params.genome ].fasta ?: false : if (params.fasta) { ch_fasta = file(params.fasta, checkIfExists: true) } // Has the run name been specified by the user? -// this has the bonus effect of catching both -name and --name +// this has the bonus effect of catching both -name and --name custom_runName = params.name if (!(workflow.runName ==~ /[a-z]+_[a-z]+/)) { custom_runName = workflow.runName } +// Check AWS batch settings if (workflow.profile.contains('awsbatch')) { // AWSBatch sanity checking if (!params.awsqueue || !params.awsregion) exit 1, "Specify correct --awsqueue and --awsregion parameters on AWSBatch!" @@ -93,28 +95,29 @@ if (workflow.profile.contains('awsbatch')) { ch_multiqc_config = file("$baseDir/assets/multiqc_config.yaml", checkIfExists: true) ch_multiqc_custom_config = params.multiqc_config ? Channel.fromPath(params.multiqc_config, checkIfExists: true) : Channel.empty() ch_output_docs = file("$baseDir/docs/output.md", checkIfExists: true) +ch_output_docs_images = file("$baseDir/docs/images/", checkIfExists: true) /* * Create a channel for input read files */ -if (params.readPaths) { +if (params.input_paths) { if (params.single_end) { Channel - .from(params.readPaths) + .from(params.input_paths) .map { row -> [ row[0], [ file(row[1][0], checkIfExists: true) ] ] } - .ifEmpty { exit 1, "params.readPaths was empty - no input files supplied" } + .ifEmpty { exit 1, "params.input_paths was empty - no input files supplied" } .into { ch_read_files_fastqc; ch_read_files_trimming } } else { Channel - .from(params.readPaths) + .from(params.input_paths) .map { row -> [ row[0], [ file(row[1][0], checkIfExists: true), file(row[1][1], checkIfExists: true) ] ] } - .ifEmpty { exit 1, "params.readPaths was empty - no input files supplied" } + .ifEmpty { exit 1, "params.input_paths was empty - no input files supplied" } .into { ch_read_files_fastqc; ch_read_files_trimming } } } else { Channel - .fromFilePairs(params.reads, size: params.single_end ? 1 : 2) - .ifEmpty { exit 1, "Cannot find any reads matching: ${params.reads}\nNB: Path needs to be enclosed in quotes!\nIf this is single-end data, please specify --single_end on the command line." } + .fromFilePairs(params.input, size: params.single_end ? 1 : 2) + .ifEmpty { exit 1, "Cannot find any reads matching: ${params.input}\nNB: Path needs to be enclosed in quotes!\nIf this is single-end data, please specify --single_end on the command line." } .into { ch_read_files_fastqc; ch_read_files_trimming } } @@ -124,7 +127,7 @@ def summary = [:] if (workflow.revision) summary['Pipeline Release'] = workflow.revision summary['Run Name'] = custom_runName ?: workflow.runName // TODO nf-core: Report custom parameters here -summary['Reads'] = params.reads +summary['Reads'] = params.input summary['Fasta Ref'] = params.fasta summary['Data Type'] = params.single_end ? 'Single-End' : 'Paired-End' summary['Max Resources'] = "$params.max_memory memory, $params.max_cpus cpus, $params.max_time time per job" @@ -140,9 +143,10 @@ if (workflow.profile.contains('awsbatch')) { summary['AWS CLI'] = params.awscli } summary['Config Profile'] = workflow.profile -if (params.config_profile_description) summary['Config Description'] = params.config_profile_description -if (params.config_profile_contact) summary['Config Contact'] = params.config_profile_contact -if (params.config_profile_url) summary['Config URL'] = params.config_profile_url +if (params.config_profile_description) summary['Config Profile Description'] = params.config_profile_description +if (params.config_profile_contact) summary['Config Profile Contact'] = params.config_profile_contact +if (params.config_profile_url) summary['Config Profile URL'] = params.config_profile_url +summary['Config Files'] = workflow.configFiles.join(', ') if (params.email || params.email_on_fail) { summary['E-mail Address'] = params.email summary['E-mail on failure'] = params.email_on_fail @@ -174,7 +178,7 @@ Channel.from(summary.collect{ [it.key, it.value] }) * Parse software version numbers */ process get_software_versions { - publishDir "${params.outdir}/pipeline_info", mode: 'copy', + publishDir "${params.outdir}/pipeline_info", mode: params.publish_dir_mode, saveAs: { filename -> if (filename.indexOf(".csv") > 0) filename else null @@ -201,7 +205,7 @@ process get_software_versions { process fastqc { tag "$name" label 'process_medium' - publishDir "${params.outdir}/fastqc", mode: 'copy', + publishDir "${params.outdir}/fastqc", mode: params.publish_dir_mode, saveAs: { filename -> filename.indexOf(".zip") > 0 ? "zips/$filename" : "$filename" } @@ -222,7 +226,7 @@ process fastqc { * STEP 2 - MultiQC */ process multiqc { - publishDir "${params.outdir}/MultiQC", mode: 'copy' + publishDir "${params.outdir}/MultiQC", mode: params.publish_dir_mode input: file (multiqc_config) from ch_multiqc_config @@ -251,10 +255,11 @@ process multiqc { * STEP 3 - Output Description HTML */ process output_documentation { - publishDir "${params.outdir}/pipeline_info", mode: 'copy' + publishDir "${params.outdir}/pipeline_info", mode: params.publish_dir_mode input: file output_docs from ch_output_docs + file images from ch_output_docs_images output: file "results_description.html" @@ -345,7 +350,11 @@ workflow.onComplete { log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (sendmail)" } catch (all) { // Catch failures and try with plaintext - [ 'mail', '-s', subject, email_address ].execute() << email_txt + def mail_cmd = [ 'mail', '-s', subject, '--content-type=text/html', email_address ] + if ( mqc_report.size() <= params.max_multiqc_email_size.toBytes() ) { + mail_cmd += [ '-A', mqc_report ] + } + mail_cmd.execute() << email_html log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (mail)" } } diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config index c8d4ea682a..30f5260f67 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config @@ -11,9 +11,10 @@ params { // Workflow flags // TODO nf-core: Specify your pipeline's command line flags genome = false - reads = "data/*{1,2}.fastq.gz" + input = "data/*{1,2}.fastq.gz" single_end = false outdir = './results' + publish_dir_mode = 'copy' // Boilerplate options name = false @@ -78,9 +79,11 @@ if (!params.igenomes_ignore) { includeConfig 'conf/igenomes.config' } -// Export this variable to prevent local Python libraries from conflicting with those in the container +// Export these variables to prevent local Python/R libraries from conflicting with those in the container env { PYTHONNOUSERSITE = 1 + R_PROFILE_USER = "/.Rprofile" + R_ENVIRON_USER = "/.Renviron" } // Capture exit codes from upstream processes when piping diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json new file mode 100644 index 0000000000..b12212e80b --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -0,0 +1,259 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", + "title": "{{ cookiecutter.name }} pipeline parameters", + "description": "{{ cookiecutter.description }}", + "type": "object", + "definitions": { + "input_output_options": { + "title": "Input/output options", + "type": "object", + "fa_icon": "fas fa-terminal", + "description": "Define where the pipeline should find input data and save output data.", + "required": [ + "input" + ], + "properties": { + "input": { + "type": "string", + "fa_icon": "fas fa-dna", + "description": "Input FastQ files.", + "help_text": "Use this to specify the location of your input FastQ files. For example:\n\n```bash\n--input 'path/to/data/sample_*_{1,2}.fastq'\n```\n\nPlease note the following requirements:\n\n1. The path must be enclosed in quotes\n2. The path must have at least one `*` wildcard character\n3. When using the pipeline with paired end data, the path must use `{1,2}` notation to specify read pairs.\n\nIf left unspecified, a default pattern is used: `data/*{1,2}.fastq.gz`" + }, + "single_end": { + "type": "boolean", + "description": "Specifies that the input is single-end reads.", + "fa_icon": "fas fa-align-center", + "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, you need to specify `--single_end` on the command line when you launch the pipeline. A normal glob pattern, enclosed in quotation marks, can then be used for `--input`. For example:\n\n```bash\n--single_end --input '*.fastq'\n```\n\nIt is not possible to run a mixture of single-end and paired-end files in one run." + }, + "outdir": { + "type": "string", + "description": "The output directory where the results will be saved.", + "default": "./results", + "fa_icon": "fas fa-folder-open" + }, + "email": { + "type": "string", + "description": "Email address for completion summary.", + "fa_icon": "fas fa-envelope", + "help_text": "Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + } + } + }, + "reference_genome_options": { + "title": "Reference genome options", + "type": "object", + "fa_icon": "fas fa-dna", + "description": "Options for the reference genome indices used to align reads.", + "properties": { + "genome": { + "type": "string", + "description": "Name of iGenomes reference.", + "fa_icon": "fas fa-book", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`.\n\nSee the [nf-core website docs](https://nf-co.re/usage/reference_genomes) for more details." + }, + "fasta": { + "type": "string", + "fa_icon": "fas fa-font", + "description": "Path to FASTA genome file.", + "help_text": "If you have no genome reference available, the pipeline can build one using a FASTA file. This requires additional time and resources, so it's better to use a pre-build index if possible." + }, + "igenomes_base": { + "type": "string", + "description": "Directory / URL base for iGenomes references.", + "default": "s3://ngi-igenomes/igenomes/", + "fa_icon": "fas fa-cloud-download-alt", + "hidden": true + }, + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config.", + "fa_icon": "fas fa-ban", + "hidden": true, + "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." + } + } + }, + "generic_options": { + "title": "Generic options", + "type": "object", + "fa_icon": "fas fa-file-import", + "description": "Less common options for the pipeline, typically set in a config file.", + "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", + "properties": { + "help": { + "type": "boolean", + "description": "Display help text.", + "hidden": true, + "fa_icon": "fas fa-question-circle" + }, + "publish_dir_mode": { + "type": "string", + "default": "copy", + "hidden": true, + "description": "Method used to save pipeline results to output directory.", + "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", + "fa_icon": "fas fa-copy", + "enum": [ + "symlink", + "rellink", + "link", + "copy", + "copyNoFollow", + "mov" + ] + }, + "name": { + "type": "string", + "description": "Workflow name.", + "fa_icon": "fas fa-fingerprint", + "hidden": true, + "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." + }, + "email_on_fail": { + "type": "string", + "description": "Email address for completion summary, only when pipeline fails.", + "fa_icon": "fas fa-exclamation-triangle", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "hidden": true, + "help_text": "This works exactly as with `--email`, except emails are only sent if the workflow is not successful." + }, + "plaintext_email": { + "type": "boolean", + "description": "Send plain-text email instead of HTML.", + "fa_icon": "fas fa-remove-format", + "hidden": true, + "help_text": "Set to receive plain-text e-mails instead of HTML formatted." + }, + "max_multiqc_email_size": { + "type": "string", + "description": "File size limit when attaching MultiQC reports to summary emails.", + "default": "25.MB", + "fa_icon": "fas fa-file-upload", + "hidden": true, + "help_text": "If file generated by pipeline exceeds the threshold, it will not be attached." + }, + "monochrome_logs": { + "type": "boolean", + "description": "Do not use coloured log outputs.", + "fa_icon": "fas fa-palette", + "hidden": true, + "help_text": "Set to disable colourful command line output and live life in monochrome." + }, + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC.", + "fa_icon": "fas fa-cog", + "hidden": true + }, + "tracedir": { + "type": "string", + "description": "Directory to keep pipeline Nextflow logs and reports.", + "default": "${params.outdir}/pipeline_info", + "fa_icon": "fas fa-cogs", + "hidden": true + } + } + }, + "max_job_request_options": { + "title": "Max job request options", + "type": "object", + "fa_icon": "fab fa-acquisitions-incorporated", + "description": "Set the top limit for requested resources for any single job.", + "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", + "properties": { + "max_cpus": { + "type": "integer", + "description": "Maximum number of CPUs that can be requested for any single job.", + "default": 16, + "fa_icon": "fas fa-microchip", + "hidden": true, + "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" + }, + "max_memory": { + "type": "string", + "description": "Maximum amount of memory that can be requested for any single job.", + "default": "128.GB", + "fa_icon": "fas fa-memory", + "hidden": true, + "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" + }, + "max_time": { + "type": "string", + "description": "Maximum amount of time that can be requested for any single job.", + "default": "240.h", + "fa_icon": "far fa-clock", + "hidden": true, + "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" + } + } + }, + "institutional_config_options": { + "title": "Institutional config options", + "type": "object", + "fa_icon": "fas fa-university", + "description": "Parameters used to describe centralised config profiles. These should not be edited.", + "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", + "properties": { + "custom_config_version": { + "type": "string", + "description": "Git commit id for Institutional configs.", + "default": "master", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "Provide git commit id for custom Institutional configs hosted at `nf-core/configs`. This was implemented for reproducibility purposes. Default: `master`.\n\n```bash\n## Download and use config file with following git commit id\n--custom_config_version d52db660777c4bf36546ddb188ec530c3ada1b96\n```" + }, + "custom_config_base": { + "type": "string", + "description": "Base directory for Institutional configs.", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true, + "help_text": "If you're running offline, nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell nextflow where to find them with the `custom_config_base` option. For example:\n\n```bash\n## Download and unzip the config files\ncd /path/to/my/configs\nwget https://github.com/nf-core/configs/archive/master.zip\nunzip master.zip\n\n## Run the pipeline\ncd /path/to/my/data\nnextflow run /path/to/pipeline/ --custom_config_base /path/to/my/configs/configs-master/\n```\n\n> Note that the nf-core/tools helper package has a `download` command to download all required pipeline files + singularity containers + institutional configs in one go for you, to make this process easier.", + "fa_icon": "fas fa-users-cog" + }, + "hostnames": { + "type": "string", + "description": "Institutional configs hostname.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + }, + "config_profile_description": { + "type": "string", + "description": "Institutional config description.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + }, + "config_profile_contact": { + "type": "string", + "description": "Institutional config contact information.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + }, + "config_profile_url": { + "type": "string", + "description": "Institutional config URL link.", + "hidden": true, + "fa_icon": "fas fa-users-cog" + } + } + } + }, + "allOf": [ + { + "$ref": "#/definitions/input_output_options" + }, + { + "$ref": "#/definitions/reference_genome_options" + }, + { + "$ref": "#/definitions/generic_options" + }, + { + "$ref": "#/definitions/max_job_request_options" + }, + { + "$ref": "#/definitions/institutional_config_options" + } + ] +} diff --git a/nf_core/schema.py b/nf_core/schema.py new file mode 100644 index 0000000000..177de0a0f9 --- /dev/null +++ b/nf_core/schema.py @@ -0,0 +1,546 @@ +#!/usr/bin/env python +""" Code to deal with pipeline JSON Schema """ + +from __future__ import print_function +from rich.prompt import Confirm + +import copy +import jinja2 +import json +import jsonschema +import logging +import os +import requests +import requests_cache +import sys +import time +import webbrowser +import yaml + +import nf_core.list, nf_core.utils + +log = logging.getLogger(__name__) + + +class PipelineSchema(object): + """ Class to generate a schema object with + functions to handle pipeline JSON Schema """ + + def __init__(self): + """ Initialise the object """ + + self.schema = None + self.pipeline_dir = None + self.schema_filename = None + self.schema_defaults = {} + self.schema_params = [] + self.input_params = {} + self.pipeline_params = {} + self.pipeline_manifest = {} + self.schema_from_scratch = False + self.no_prompts = False + self.web_only = False + self.web_schema_build_url = "https://nf-co.re/pipeline_schema_builder" + self.web_schema_build_web_url = None + self.web_schema_build_api_url = None + + def get_schema_path(self, path, local_only=False, revision=None): + """ Given a pipeline name, directory, or path, set self.schema_filename """ + + # Supplied path exists - assume a local pipeline directory or schema + if os.path.exists(path): + if revision is not None: + log.warning("Local workflow supplied, ignoring revision '{}'".format(revision)) + if os.path.isdir(path): + self.pipeline_dir = path + self.schema_filename = os.path.join(path, "nextflow_schema.json") + else: + self.pipeline_dir = os.path.dirname(path) + self.schema_filename = path + + # Path does not exist - assume a name of a remote workflow + elif not local_only: + self.pipeline_dir = nf_core.list.get_local_wf(path, revision=revision) + self.schema_filename = os.path.join(self.pipeline_dir, "nextflow_schema.json") + + # Only looking for local paths, overwrite with None to be safe + else: + self.schema_filename = None + + # Check that the schema file exists + if self.schema_filename is None or not os.path.exists(self.schema_filename): + error = "Could not find pipeline schema for '{}': {}".format(path, self.schema_filename) + log.error(error) + raise AssertionError(error) + + def load_lint_schema(self): + """ Load and lint a given schema to see if it looks valid """ + try: + self.load_schema() + num_params = self.validate_schema() + self.get_schema_defaults() + log.info("[green][[✓]] Pipeline schema looks valid[/] [dim](found {} params)".format(num_params)) + except json.decoder.JSONDecodeError as e: + error_msg = "[bold red]Could not parse schema JSON:[/] {}".format(e) + log.error(error_msg) + raise AssertionError(error_msg) + except AssertionError as e: + error_msg = "[red][[✗]] Pipeline schema does not follow nf-core specs:\n {}".format(e) + log.error(error_msg) + raise AssertionError(error_msg) + + def load_schema(self): + """ Load a pipeline schema from a file """ + with open(self.schema_filename, "r") as fh: + self.schema = json.load(fh) + self.schema_defaults = {} + self.schema_params = [] + log.debug("JSON file loaded: {}".format(self.schema_filename)) + + def get_schema_defaults(self): + """ + Generate set of default input parameters from schema. + + Saves defaults to self.schema_defaults + Returns count of how many parameters were found (with or without a default value) + """ + # Top level schema-properties (ungrouped) + for p_key, param in self.schema.get("properties", {}).items(): + self.schema_params.append(p_key) + if "default" in param: + self.schema_defaults[p_key] = param["default"] + + # Grouped schema properties in subschema definitions + for d_key, definition in self.schema.get("definitions", {}).items(): + for p_key, param in definition.get("properties", {}).items(): + self.schema_params.append(p_key) + if "default" in param: + self.schema_defaults[p_key] = param["default"] + + def save_schema(self): + """ Save a pipeline schema to a file """ + # Write results to a JSON file + num_params = len(self.schema.get("properties", {})) + num_params += sum([len(d.get("properties", {})) for d in self.schema.get("definitions", {}).values()]) + log.info("Writing schema with {} params: '{}'".format(num_params, self.schema_filename)) + with open(self.schema_filename, "w") as fh: + json.dump(self.schema, fh, indent=4) + + def load_input_params(self, params_path): + """ Load a given a path to a parameters file (JSON/YAML) + + These should be input parameters used to run a pipeline with + the Nextflow -params-file option. + """ + # First, try to load as JSON + try: + with open(params_path, "r") as fh: + params = json.load(fh) + self.input_params.update(params) + log.debug("Loaded JSON input params: {}".format(params_path)) + except Exception as json_e: + log.debug("Could not load input params as JSON: {}".format(json_e)) + # This failed, try to load as YAML + try: + with open(params_path, "r") as fh: + params = yaml.safe_load(fh) + self.input_params.update(params) + log.debug("Loaded YAML input params: {}".format(params_path)) + except Exception as yaml_e: + error_msg = "Could not load params file as either JSON or YAML:\n JSON: {}\n YAML: {}".format( + json_e, yaml_e + ) + log.error(error_msg) + raise AssertionError(error_msg) + + def validate_params(self): + """ Check given parameters against a schema and validate """ + try: + assert self.schema is not None + jsonschema.validate(self.input_params, self.schema) + except AssertionError: + log.error("[red][[✗]] Pipeline schema not found") + return False + except jsonschema.exceptions.ValidationError as e: + log.error("[red][[✗]] Input parameters are invalid: {}".format(e.message)) + return False + log.info("[green][[✓]] Input parameters look valid") + return True + + def validate_schema(self, schema=None): + """ + Check that the Schema is valid + + Returns: Number of parameters found + """ + if schema is None: + schema = self.schema + try: + jsonschema.Draft7Validator.check_schema(schema) + log.debug("JSON Schema Draft7 validated") + except jsonschema.exceptions.SchemaError as e: + raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) + + param_keys = list(schema.get("properties", {}).keys()) + num_params = len(param_keys) + for d_key, d_schema in schema.get("definitions", {}).items(): + # Check that this definition is mentioned in allOf + assert "allOf" in schema, "Schema has definitions, but no allOf key" + in_allOf = False + for allOf in schema["allOf"]: + if allOf["$ref"] == "#/definitions/{}".format(d_key): + in_allOf = True + assert in_allOf, "Definition subschema `{}` not included in schema `allOf`".format(d_key) + + for d_param_id in d_schema.get("properties", {}): + # Check that we don't have any duplicate parameter IDs in different definitions + assert d_param_id not in param_keys, "Duplicate parameter found in schema `definitions`: `{}`".format( + d_param_id + ) + param_keys.append(d_param_id) + num_params += 1 + + # Check that everything in allOf exists + for allOf in schema.get("allOf", []): + assert "definitions" in schema, "Schema has allOf, but no definitions" + def_key = allOf["$ref"][14:] + assert def_key in schema["definitions"], "Subschema `{}` found in `allOf` but not `definitions`".format( + def_key + ) + + # Check that the schema describes at least one parameter + assert num_params > 0, "No parameters found in schema" + + return num_params + + def validate_schema_title_description(self, schema=None): + """ + Extra validation command for linting. + Checks that the schema "$id", "title" and "description" attributes match the piipeline config. + """ + if schema is None: + schema = self.schema + if schema is None: + log.debug("Pipeline schema not set - skipping validation of top-level attributes") + return None + + assert "$schema" in self.schema, "Schema missing top-level `$schema` attribute" + schema_attr = "https://json-schema.org/draft-07/schema" + assert self.schema["$schema"] == schema_attr, "Schema `$schema` should be `{}`\n Found `{}`".format( + schema_attr, self.schema["$schema"] + ) + + if self.pipeline_manifest == {}: + self.get_wf_params() + + if "name" not in self.pipeline_manifest: + log.debug("Pipeline manifest `name` not known - skipping validation of schema id and title") + else: + assert "$id" in self.schema, "Schema missing top-level `$id` attribute" + assert "title" in self.schema, "Schema missing top-level `title` attribute" + # Validate that id, title and description match the pipeline manifest + id_attr = "https://raw.githubusercontent.com/{}/master/nextflow_schema.json".format( + self.pipeline_manifest["name"].strip("\"'") + ) + assert self.schema["$id"] == id_attr, "Schema `$id` should be `{}`\n Found `{}`".format( + id_attr, self.schema["$id"] + ) + + title_attr = "{} pipeline parameters".format(self.pipeline_manifest["name"].strip("\"'")) + assert self.schema["title"] == title_attr, "Schema `title` should be `{}`\n Found: `{}`".format( + title_attr, self.schema["title"] + ) + + if "description" not in self.pipeline_manifest: + log.debug("Pipeline manifest 'description' not known - skipping validation of schema description") + else: + assert "description" in self.schema, "Schema missing top-level 'description' attribute" + desc_attr = self.pipeline_manifest["description"].strip("\"'") + assert self.schema["description"] == desc_attr, "Schema 'description' should be '{}'\n Found: '{}'".format( + desc_attr, self.schema["description"] + ) + + def make_skeleton_schema(self): + """ Make a new pipeline schema from the template """ + self.schema_from_scratch = True + # Use Jinja to render the template schema file to a variable + # Bit confusing sorry, but cookiecutter only works with directories etc so this saves a bunch of code + templateLoader = jinja2.FileSystemLoader( + searchpath=os.path.join( + os.path.dirname(os.path.realpath(__file__)), "pipeline-template", "{{cookiecutter.name_noslash}}" + ) + ) + templateEnv = jinja2.Environment(loader=templateLoader) + schema_template = templateEnv.get_template("nextflow_schema.json") + cookiecutter_vars = { + "name": self.pipeline_manifest.get("name", os.path.dirname(self.schema_filename)).strip("'"), + "description": self.pipeline_manifest.get("description", "").strip("'"), + } + self.schema = json.loads(schema_template.render(cookiecutter=cookiecutter_vars)) + self.get_schema_defaults() + + def build_schema(self, pipeline_dir, no_prompts, web_only, url): + """ Interactively build a new pipeline schema for a pipeline """ + + if no_prompts: + self.no_prompts = True + if web_only: + self.web_only = True + if url: + self.web_schema_build_url = url + + # Get pipeline schema filename + try: + self.get_schema_path(pipeline_dir, local_only=True) + except AssertionError: + log.info("No existing schema found - creating a new one from the nf-core template") + self.get_wf_params() + self.make_skeleton_schema() + self.remove_schema_notfound_configs() + self.add_schema_found_configs() + try: + self.validate_schema() + except AssertionError as e: + log.error("[red]Something went wrong when building a new schema:[/] {}".format(e)) + log.info("Please ask for help on the nf-core Slack") + return False + else: + # Schema found - load and validate + try: + self.load_lint_schema() + except AssertionError as e: + log.error("Existing pipeline schema found, but it is invalid: {}".format(self.schema_filename)) + log.info("Please fix or delete this file, then try again.") + return False + + if not self.web_only: + self.get_wf_params() + self.remove_schema_notfound_configs() + self.add_schema_found_configs() + self.save_schema() + + # If running interactively, send to the web for customisation + if not self.no_prompts: + if Confirm.ask(":rocket: Launch web builder for customisation and editing?"): + try: + self.launch_web_builder() + except AssertionError as e: + log.error(e.args[0]) + # Extra help for people running offline + if "Could not connect" in e.args[0]: + log.info( + "If you're working offline, now copy your schema ({}) and paste at https://nf-co.re/pipeline_schema_builder".format( + self.schema_filename + ) + ) + log.info("When you're finished, you can paste the edited schema back into the same file") + if self.web_schema_build_web_url: + log.info( + "To save your work, open {}\n" + "Click the blue 'Finished' button, copy the schema and paste into this file: {}".format( + self.web_schema_build_web_url, self.schema_filename + ) + ) + return False + + def get_wf_params(self): + """ + Load the pipeline parameter defaults using `nextflow config` + Strip out only the params. values and ignore anything that is not a flat variable + """ + # Check that we haven't already pulled these (eg. skeleton schema) + if len(self.pipeline_params) > 0 and len(self.pipeline_manifest) > 0: + log.debug("Skipping get_wf_params as we already have them") + return + + log.debug("Collecting pipeline parameter defaults\n") + config = nf_core.utils.fetch_wf_config(os.path.dirname(self.schema_filename)) + skipped_params = [] + # Pull out just the params. values + for ckey, cval in config.items(): + if ckey.startswith("params."): + # skip anything that's not a flat variable + if "." in ckey[7:]: + skipped_params.append(ckey) + continue + self.pipeline_params[ckey[7:]] = cval + if ckey.startswith("manifest."): + self.pipeline_manifest[ckey[9:]] = cval + # Log skipped params + if len(skipped_params) > 0: + log.debug( + "Skipped following pipeline params because they had nested parameter values:\n{}".format( + ", ".join(skipped_params) + ) + ) + + def remove_schema_notfound_configs(self): + """ + Go through top-level schema and all definitions sub-schemas to remove + anything that's not in the nextflow config. + """ + # Top-level properties + self.schema, params_removed = self.remove_schema_notfound_configs_single_schema(self.schema) + # Sub-schemas in definitions + for d_key, definition in self.schema.get("definitions", {}).items(): + cleaned_schema, p_removed = self.remove_schema_notfound_configs_single_schema(definition) + self.schema["definitions"][d_key] = cleaned_schema + params_removed.extend(p_removed) + return params_removed + + def remove_schema_notfound_configs_single_schema(self, schema): + """ + Go through a single schema / set of properties and strip out + anything that's not in the nextflow config. + + Takes: Schema or sub-schema with properties key + Returns: Cleaned schema / sub-schema + """ + # Make a deep copy so as not to edit in place + schema = copy.deepcopy(schema) + params_removed = [] + # Use iterator so that we can delete the key whilst iterating + for p_key in [k for k in schema.get("properties", {}).keys()]: + if self.prompt_remove_schema_notfound_config(p_key): + del schema["properties"][p_key] + # Remove required flag if set + if p_key in schema.get("required", []): + schema["required"].remove(p_key) + # Remove required list if now empty + if "required" in schema and len(schema["required"]) == 0: + del schema["required"] + log.debug("Removing '{}' from pipeline schema".format(p_key)) + params_removed.append(p_key) + + return schema, params_removed + + def prompt_remove_schema_notfound_config(self, p_key): + """ + Check if a given key is found in the nextflow config params and prompt to remove it if note + + Returns True if it should be removed, False if not. + """ + if p_key not in self.pipeline_params.keys(): + if self.no_prompts or self.schema_from_scratch: + return True + if Confirm.ask( + ":question: Unrecognised [white bold]'params.{}'[/] found in schema but not pipeline! [yellow]Remove it?".format( + p_key + ) + ): + return True + return False + + def add_schema_found_configs(self): + """ + Add anything that's found in the Nextflow params that's missing in the pipeline schema + """ + params_added = [] + for p_key, p_val in self.pipeline_params.items(): + # Check if key is in schema parameters + if not p_key in self.schema_params: + if ( + self.no_prompts + or self.schema_from_scratch + or Confirm.ask( + ":sparkles: Found [white bold]'params.{}'[/] in pipeline but not in schema. [blue]Add to pipeline schema?".format( + p_key + ) + ) + ): + if "properties" not in self.schema: + self.schema["properties"] = {} + self.schema["properties"][p_key] = self.build_schema_param(p_val) + log.debug("Adding '{}' to pipeline schema".format(p_key)) + params_added.append(p_key) + + return params_added + + def build_schema_param(self, p_val): + """ + Build a pipeline schema dictionary for an param interactively + """ + p_val = p_val.strip("\"'") + # p_val is always a string as it is parsed from nextflow config this way + try: + p_val = float(p_val) + if p_val == int(p_val): + p_val = int(p_val) + p_type = "integer" + else: + p_type = "number" + except ValueError: + p_type = "string" + + # NB: Only test "True" for booleans, as it is very common to initialise + # an empty param as false when really we expect a string at a later date.. + if p_val == "True": + p_val = True + p_type = "boolean" + + p_schema = {"type": p_type, "default": p_val} + + # Assume that false and empty strings shouldn't be a default + if p_val == "false" or p_val == "": + del p_schema["default"] + + return p_schema + + def launch_web_builder(self): + """ + Send pipeline schema to web builder and wait for response + """ + content = { + "post_content": "json_schema", + "api": "true", + "version": nf_core.__version__, + "status": "waiting_for_user", + "schema": json.dumps(self.schema), + } + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_url, content) + try: + assert "api_url" in web_response + assert "web_url" in web_response + assert web_response["status"] == "recieved" + except (AssertionError) as e: + log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + raise AssertionError( + "Pipeline schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format( + self.web_schema_build_url + ) + ) + else: + self.web_schema_build_web_url = web_response["web_url"] + self.web_schema_build_api_url = web_response["api_url"] + log.info("Opening URL: {}".format(web_response["web_url"])) + webbrowser.open(web_response["web_url"]) + log.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") + nf_core.utils.wait_cli_function(self.get_web_builder_response) + + def get_web_builder_response(self): + """ + Given a URL for a Schema build response, recursively query it until results are ready. + Once ready, validate Schema and write to disk. + """ + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_api_url) + if web_response["status"] == "error": + raise AssertionError("Got error from schema builder: '{}'".format(web_response.get("message"))) + elif web_response["status"] == "waiting_for_user": + return False + elif web_response["status"] == "web_builder_edited": + log.info("Found saved status from nf-core schema builder") + try: + self.schema = web_response["schema"] + self.validate_schema() + except AssertionError as e: + raise AssertionError("Response from schema builder did not pass validation:\n {}".format(e)) + else: + self.save_schema() + return True + else: + log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + raise AssertionError( + "Pipeline schema builder returned unexpected status ({}): {}\n See verbose log for full response".format( + web_response["status"], self.web_schema_build_api_url + ) + ) diff --git a/nf_core/sync.py b/nf_core/sync.py index d0ecd720d6..5dc109cbb8 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -6,30 +6,40 @@ import git import json import logging -import nf_core import os import re import requests import shutil -import sys import tempfile +import nf_core +import nf_core.create +import nf_core.list +import nf_core.sync +import nf_core.utils + +log = logging.getLogger(__name__) + + class SyncException(Exception): """Exception raised when there was an error with TEMPLATE branch synchronisation """ + pass + class PullRequestException(Exception): """Exception raised when there was an error creating a Pull-Request on GitHub.com """ + pass + class PipelineSync(object): """Object to hold syncing information and results. Args: pipeline_dir (str): The path to the Nextflow pipeline root directory - make_template_branch (bool): Set this to `True` to create a `TEMPLATE` branch if it is not found from_branch (str): The branch to use to fetch config vars. If not set, will use current active branch make_pr (bool): Set this to `True` to create a GitHub pull-request with the changes gh_username (str): GitHub username @@ -39,8 +49,7 @@ class PipelineSync(object): Attributes: pipeline_dir (str): Path to target pipeline directory from_branch (str): Repo branch to use when collecting workflow variables. Default: active branch. - make_template_branch (bool): Whether to try to create TEMPLATE branch if not found - orphan_branch (bool): Whether an orphan branch was made when creating TEMPLATE + original_branch (str): Repo branch that was checked out before we started. made_changes (bool): Whether making the new template pipeline introduced any changes make_pr (bool): Whether to try to automatically make a PR on GitHub.com required_config_vars (list): List of nextflow variables required to make template pipeline @@ -49,23 +58,18 @@ class PipelineSync(object): gh_auth_token (str): Authorisation token used to make PR with GitHub API """ - def __init__(self, pipeline_dir, make_template_branch=False, from_branch=None, make_pr=False, - gh_username=None, gh_repo=None, gh_auth_token=None): + def __init__( + self, pipeline_dir, from_branch=None, make_pr=False, gh_username=None, gh_repo=None, gh_auth_token=None, + ): """ Initialise syncing object """ self.pipeline_dir = os.path.abspath(pipeline_dir) self.from_branch = from_branch - self.make_template_branch = make_template_branch - self.orphan_branch = False + self.original_branch = None self.made_changes = False self.make_pr = make_pr self.gh_pr_returned_data = {} - self.required_config_vars = [ - 'manifest.name', - 'manifest.description', - 'manifest.version', - 'manifest.author' - ] + self.required_config_vars = ["manifest.name", "manifest.description", "manifest.version", "manifest.author"] self.gh_username = gh_username self.gh_repo = gh_repo @@ -78,40 +82,36 @@ def sync(self): config_log_msg = "Pipeline directory: {}".format(self.pipeline_dir) if self.from_branch: config_log_msg += "\n Using branch `{}` to fetch workflow variables".format(self.from_branch) - if self.make_template_branch: - config_log_msg += "\n Will attempt to create `TEMPLATE` branch if not found" if self.make_pr: config_log_msg += "\n Will attempt to automatically create a pull request on GitHub.com" - logging.info(config_log_msg) + log.info(config_log_msg) self.inspect_sync_dir() - self.get_wf_config() - self.checkout_template_branch() - + self.delete_template_branch_files() self.make_template_pipeline() - self.commit_template_changes() # Push and make a pull request if we've been asked to - pr_exception = False - if self.make_pr: + if self.made_changes and self.make_pr: try: self.push_template_branch() self.make_pull_request() except PullRequestException as e: - # Keep going - we want to clean up the target directory still - logging.error(e) - pr_exception = e + self.reset_target_dir() + raise PullRequestException(pr_exception) self.reset_target_dir() - if not self.make_pr: - self.git_merge_help() - elif pr_exception: - raise PullRequestException(pr_exception) - + if not self.made_changes: + log.info("No changes made to TEMPLATE - sync complete") + elif not self.make_pr: + log.info( + "Now try to merge the updates in to your pipeline:\n cd {}\n git merge TEMPLATE".format( + self.pipeline_dir + ) + ) def inspect_sync_dir(self): """Takes a look at the target directory for syncing. Checks that it's a git repo @@ -125,11 +125,13 @@ def inspect_sync_dir(self): # get current branch so we can switch back later self.original_branch = self.repo.active_branch.name - logging.debug("Original pipeline repository branch is '{}'".format(self.original_branch)) + log.debug("Original pipeline repository branch is '{}'".format(self.original_branch)) # Check to see if there are uncommitted changes on current branch if self.repo.is_dirty(untracked_files=True): - raise SyncException("Uncommitted changes found in pipeline directory!\nPlease commit these before running nf-core sync") + raise SyncException( + "Uncommitted changes found in pipeline directory!\nPlease commit these before running nf-core sync" + ) def get_wf_config(self): """Check out the target branch if requested and fetch the nextflow config. @@ -138,7 +140,7 @@ def get_wf_config(self): # Try to check out target branch (eg. `origin/dev`) try: if self.from_branch and self.repo.active_branch.name != self.from_branch: - logging.info("Checking out workflow branch '{}'".format(self.from_branch)) + log.info("Checking out workflow branch '{}'".format(self.from_branch)) self.repo.git.checkout(self.from_branch) except git.exc.GitCommandError: raise SyncException("Branch `{}` not found!".format(self.from_branch)) @@ -148,24 +150,30 @@ def get_wf_config(self): try: self.from_branch = self.repo.active_branch.name except git.exc.GitCommandError as e: - logging.error("Could not find active repo branch: ".format(e)) + log.error("Could not find active repo branch: ".format(e)) # Figure out the GitHub username and repo name from the 'origin' remote if we can try: - origin_url = self.repo.remotes.origin.url.rstrip('.git') - gh_origin_match = re.search(r'github\.com[:\/]([^\/]+)/([^\/]+)$', origin_url) + origin_url = self.repo.remotes.origin.url.rstrip(".git") + gh_origin_match = re.search(r"github\.com[:\/]([^\/]+)/([^\/]+)$", origin_url) if gh_origin_match: self.gh_username = gh_origin_match.group(1) self.gh_repo = gh_origin_match.group(2) else: raise AttributeError except AttributeError as e: - logging.debug("Could not find repository URL for remote called 'origin' from remote: {}".format(self.repo.remotes.origin.url)) + log.debug( + "Could not find repository URL for remote called 'origin' from remote: {}".format(self.repo.remotes) + ) else: - logging.debug("Found username and repo from remote: {}, {} - {}".format(self.gh_username, self.gh_repo, self.repo.remotes.origin.url)) + log.debug( + "Found username and repo from remote: {}, {} - {}".format( + self.gh_username, self.gh_repo, self.repo.remotes.origin.url + ) + ) # Fetch workflow variables - logging.info("Fetching workflow config variables") + log.info("Fetching workflow config variables") self.wf_config = nf_core.utils.fetch_wf_config(self.pipeline_dir) # Check that we have the required variables @@ -174,50 +182,31 @@ def get_wf_config(self): raise SyncException("Workflow config variable `{}` not found!".format(rvar)) def checkout_template_branch(self): - """Try to check out the TEMPLATE branch. If it fails, try origin/TEMPLATE. - If it still fails and --make-template-branch was given, create it as an orphan branch. + """ + Try to check out the origin/TEMPLATE in a new TEMPLATE branch. + If this fails, try to check out an existing local TEMPLATE branch. """ # Try to check out the `TEMPLATE` branch try: self.repo.git.checkout("origin/TEMPLATE", b="TEMPLATE") except git.exc.GitCommandError: - # Try to check out an existing local branch called TEMPLATE try: self.repo.git.checkout("TEMPLATE") except git.exc.GitCommandError: + raise SyncException("Could not check out branch 'origin/TEMPLATE' or 'TEMPLATE'") - # Failed, if we're not making a new branch just die - if not self.make_template_branch: - raise SyncException( - "Could not check out branch 'origin/TEMPLATE'" \ - "\nUse flag --make-template-branch to attempt to create this branch" - ) - - # Branch and force is set, fire function to create `TEMPLATE` branch - else: - logging.debug("Could not check out origin/TEMPLATE!") - logging.info("Creating orphan TEMPLATE branch") - try: - self.repo.git.checkout('--orphan', 'TEMPLATE') - self.orphan_branch = True - if self.make_pr: - self.make_pr = False - logging.warning("Will not attempt to make a PR - orphan branch must be merged manually first") - except git.exc.GitCommandError as e: - raise SyncException("Could not create 'TEMPLATE' branch:\n{}".format(e)) - - def make_template_pipeline(self): - """Delete all files and make a fresh template using the workflow variables + def delete_template_branch_files(self): + """ + Delete all files in the TEMPLATE branch """ - # Delete everything - logging.info("Deleting all files in TEMPLATE branch") + log.info("Deleting all files in TEMPLATE branch") for the_file in os.listdir(self.pipeline_dir): - if the_file == '.git': + if the_file == ".git": continue file_path = os.path.join(self.pipeline_dir, the_file) - logging.debug("Deleting {}".format(file_path)) + log.debug("Deleting {}".format(file_path)) try: if os.path.isfile(file_path): os.unlink(file_path) @@ -226,61 +215,53 @@ def make_template_pipeline(self): except Exception as e: raise SyncException(e) - # Make a new pipeline using nf_core.create - logging.info("Making a new template pipeline using pipeline variables") + def make_template_pipeline(self): + """ + Delete all files and make a fresh template using the workflow variables + """ + log.info("Making a new template pipeline using pipeline variables") - # Suppress log messages from the pipeline creation method - orig_loglevel = logging.getLogger().getEffectiveLevel() - if orig_loglevel == getattr(logging, 'INFO'): - logging.getLogger().setLevel(logging.ERROR) + # Only show error messages from pipeline creation + if log.getEffectiveLevel() == logging.INFO: + logging.getLogger("nf_core.create").setLevel(logging.ERROR) nf_core.create.PipelineCreate( - name = self.wf_config['manifest.name'].strip('\"').strip("\'"), - description = self.wf_config['manifest.description'].strip('\"').strip("\'"), - new_version = self.wf_config['manifest.version'].strip('\"').strip("\'"), - no_git = True, - force = True, - outdir = self.pipeline_dir, - author = self.wf_config['manifest.author'].strip('\"').strip("\'"), + name=self.wf_config["manifest.name"].strip('"').strip("'"), + description=self.wf_config["manifest.description"].strip('"').strip("'"), + new_version=self.wf_config["manifest.version"].strip('"').strip("'"), + no_git=True, + force=True, + outdir=self.pipeline_dir, + author=self.wf_config["manifest.author"].strip('"').strip("'"), ).init_pipeline() - # Reset logging - logging.getLogger().setLevel(orig_loglevel) - def commit_template_changes(self): """If we have any changes with the new template files, make a git commit """ - # Commit changes if we have any + # Check that we have something to commit if not self.repo.is_dirty(untracked_files=True): - logging.info("Template contains no changes - no new commit created") - else: - try: - self.repo.git.add(A=True) - self.repo.index.commit("Template update for nf-core/tools version {}".format(nf_core.__version__)) - self.made_changes = True - logging.info("Committed changes to TEMPLATE branch") - except Exception as e: - raise SyncException("Could not commit changes to TEMPLATE:\n{}".format(e)) + log.info("Template contains no changes - no new commit created") + return False + # Commit changes + try: + self.repo.git.add(A=True) + self.repo.index.commit("Template update for nf-core/tools version {}".format(nf_core.__version__)) + self.made_changes = True + log.info("Committed changes to TEMPLATE branch") + except Exception as e: + raise SyncException("Could not commit changes to TEMPLATE:\n{}".format(e)) + return True def push_template_branch(self): """If we made any changes, push the TEMPLATE branch to the default remote and try to make a PR. If we don't have the auth token, try to figure out a URL for the PR and print this to the console. """ - if self.made_changes: - logging.info("Pushing TEMPLATE branch to remote") - try: - self.repo.git.push() - except git.exc.GitCommandError as e: - if self.make_template_branch: - try: - self.repo.git.push('--set-upstream', 'origin', 'TEMPLATE') - except git.exc.GitCommandError as e: - raise PullRequestException("Could not push new TEMPLATE branch:\n {}".format(e)) - else: - raise PullRequestException("Could not push TEMPLATE branch:\n {}".format(e)) - else: - logging.debug("No changes to TEMPLATE - skipping push to remote") + log.info("Pushing TEMPLATE branch to remote") + try: + self.repo.git.push() + except git.exc.GitCommandError as e: + raise PullRequestException("Could not push TEMPLATE branch:\n {}".format(e)) def make_pull_request(self): """Create a pull request to a base branch (default: dev), @@ -288,9 +269,6 @@ def make_pull_request(self): Returns: An instance of class requests.Response """ - if not self.made_changes: - logging.debug("No changes to TEMPLATE - skipping PR creation") - # Check that we know the github username and repo name try: assert self.gh_username is not None @@ -302,70 +280,68 @@ def make_pull_request(self): try: assert self.gh_auth_token is not None except AssertionError: - logging.info("Make a PR at the following URL:\n https://github.com/{}/{}/compare/{}...TEMPLATE".format(self.gh_username, self.gh_repo, self.original_branch)) + log.info( + "Make a PR at the following URL:\n https://github.com/{}/{}/compare/{}...TEMPLATE".format( + self.gh_username, self.gh_repo, self.original_branch + ) + ) raise PullRequestException("No GitHub authentication token set - cannot make PR") - logging.info("Submitting a pull request via the GitHub API") + log.info("Submitting a pull request via the GitHub API") + + pr_body_text = """ + A new release of the main template in nf-core/tools has just been released. + This automated pull-request attempts to apply the relevant updates to this pipeline. + + Please make sure to merge this pull-request as soon as possible. + Once complete, make a new minor release of your pipeline. + + For instructions on how to merge this PR, please see + [https://nf-co.re/developers/sync](https://nf-co.re/developers/sync#merging-automated-prs). + + For more information about this release of [nf-core/tools](https://github.com/nf-core/tools), + please see the [nf-core/tools v{tag} release page](https://github.com/nf-core/tools/releases/tag/{tag}). + """.format( + tag=nf_core.__version__ + ) + pr_content = { - 'title': "Important! Template update for nf-core/tools v{}".format(nf_core.__version__), - 'body': "Some important changes have been made in the nf-core/tools pipeline template. " \ - "Please make sure to merge this pull-request as soon as possible. " \ - "Once complete, make a new minor release of your pipeline.\n\n" \ - "For instructions on how to merge this PR, please see " \ - "[https://nf-co.re/developers/sync](https://nf-co.re/developers/sync#merging-automated-prs).\n\n" \ - "For more information about this release of [nf-core/tools](https://github.com/nf-core/tools), " \ - "please see the [nf-core/tools v{tag} release page](https://github.com/nf-core/tools/releases/tag/{tag}).".format(tag=nf_core.__version__), - 'head': "TEMPLATE", - 'base': self.from_branch + "title": "Important! Template update for nf-core/tools v{}".format(nf_core.__version__), + "body": pr_body_text, + "maintainer_can_modify": True, + "head": "TEMPLATE", + "base": self.from_branch, } r = requests.post( - url = "https://api.github.com/repos/{}/{}/pulls".format(self.gh_username, self.gh_repo), - data = json.dumps(pr_content), - auth = requests.auth.HTTPBasicAuth(self.gh_username, self.gh_auth_token) + url="https://api.github.com/repos/{}/{}/pulls".format(self.gh_username, self.gh_repo), + data=json.dumps(pr_content), + auth=requests.auth.HTTPBasicAuth(self.gh_username, self.gh_auth_token), ) try: - self.gh_pr_returned_data = json.loads(r.text) + self.gh_pr_returned_data = json.loads(r.content) returned_data_prettyprint = json.dumps(self.gh_pr_returned_data, indent=4) except: - self.gh_pr_returned_data = r.text - returned_data_prettyprint = r.text + self.gh_pr_returned_data = r.content + returned_data_prettyprint = r.content if r.status_code != 201: - raise PullRequestException("GitHub API returned code {}: \n{}".format(r.status_code, returned_data_prettyprint)) + raise PullRequestException( + "GitHub API returned code {}: \n{}".format(r.status_code, returned_data_prettyprint) + ) else: - logging.debug("GitHub API PR worked:\n{}".format(returned_data_prettyprint)) - logging.info("GitHub PR created: {}".format(self.gh_pr_returned_data['html_url'])) + log.debug("GitHub API PR worked:\n{}".format(returned_data_prettyprint)) + log.info("GitHub PR created: {}".format(self.gh_pr_returned_data["html_url"])) def reset_target_dir(self): - """Reset the target pipeline directory. Check out the original branch. """ - - # Reset: Check out original branch again - logging.debug("Checking out original branch: '{}'".format(self.original_branch)) + Reset the target pipeline directory. Check out the original branch. + """ + log.debug("Checking out original branch: '{}'".format(self.original_branch)) try: self.repo.git.checkout(self.original_branch) except git.exc.GitCommandError as e: raise SyncException("Could not reset to original branch `{}`:\n{}".format(self.from_branch, e)) - def git_merge_help(self): - """Print a command line help message with instructions on how to merge changes - """ - if self.made_changes: - git_merge_cmd = 'git merge TEMPLATE' - manual_sync_link = '' - if self.orphan_branch: - git_merge_cmd += ' --allow-unrelated-histories' - manual_sync_link = "\n\nFor more information, please see:\nhttps://nf-co.re/developers/sync#merge-template-into-main-branches" - logging.info( - "Now try to merge the updates in to your pipeline:\n cd {}\n {}{}".format( - self.pipeline_dir, - git_merge_cmd, - manual_sync_link - ) - ) - - - def sync_all_pipelines(gh_username=None, gh_auth_token=None): """Sync all nf-core pipelines @@ -384,54 +360,54 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): # Let's do some updating! for wf in wfs.remote_workflows: - logging.info("Syncing {}".format(wf.full_name)) + log.info("Syncing {}".format(wf.full_name)) # Make a local working directory wf_local_path = os.path.join(tmpdir, wf.name) os.mkdir(wf_local_path) - logging.debug("Sync working directory: {}".format(wf_local_path)) + log.debug("Sync working directory: {}".format(wf_local_path)) # Clone the repo wf_remote_url = "https://{}@github.com/nf-core/{}".format(gh_auth_token, wf.name) repo = git.Repo.clone_from(wf_remote_url, wf_local_path) assert repo - # Suppress log messages from the pipeline creation method - orig_loglevel = logging.getLogger().getEffectiveLevel() - if orig_loglevel == getattr(logging, 'INFO'): - logging.getLogger().setLevel(logging.ERROR) + # Only show error messages from pipeline creation + if log.getEffectiveLevel() == logging.INFO: + logging.getLogger("nf_core.create").setLevel(logging.ERROR) # Sync the repo - logging.debug("Running template sync") + log.debug("Running template sync") sync_obj = nf_core.sync.PipelineSync( pipeline_dir=wf_local_path, - from_branch='dev', + from_branch="dev", make_pr=True, gh_username=gh_username, - gh_auth_token=gh_auth_token + gh_auth_token=gh_auth_token, ) try: sync_obj.sync() except (SyncException, PullRequestException) as e: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.error(click.style("Sync failed for {}:\n{}".format(wf.full_name, e), fg='yellow')) + log.error("Sync failed for {}:\n{}".format(wf.full_name, e)) failed_syncs.append(wf.name) except Exception as e: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.error(click.style("Something went wrong when syncing {}:\n{}".format(wf.full_name, e), fg='yellow')) + log.error("Something went wrong when syncing {}:\n{}".format(wf.full_name, e)) failed_syncs.append(wf.name) else: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.info("Sync successful for {}: {}".format(wf.full_name, click.style(sync_obj.gh_pr_returned_data.get('html_url'), fg='blue'))) + log.info( + "[green]Sync successful for {}:[/] [blue][link={1}]{1}[/link]".format( + wf.full_name, sync_obj.gh_pr_returned_data.get("html_url") + ) + ) successful_syncs.append(wf.name) # Clean up - logging.debug("Removing work directory: {}".format(wf_local_path)) + log.debug("Removing work directory: {}".format(wf_local_path)) shutil.rmtree(wf_local_path) if len(successful_syncs) > 0: - logging.info(click.style("Finished. Successfully synchronised {} pipelines".format(len(successful_syncs)), fg='green')) + log.info("[green]Finished. Successfully synchronised {} pipelines".format(len(successful_syncs))) if len(failed_syncs) > 0: - failed_list = '\n - '.join(failed_syncs) - logging.error(click.style("Errors whilst synchronising {} pipelines:\n - {}".format(len(failed_syncs), failed_list), fg='red')) + failed_list = "\n - ".join(failed_syncs) + log.error("[red]Errors whilst synchronising {} pipelines:\n - {}".format(len(failed_syncs), failed_list)) diff --git a/nf_core/utils.py b/nf_core/utils.py index a333fe4706..87c2a4eba3 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -2,16 +2,48 @@ """ Common utility functions for the nf-core python package. """ - +import nf_core import datetime import errno import json +import hashlib import logging import os +import re +import requests +import requests_cache import subprocess import sys +import time +from distutils import version + +log = logging.getLogger(__name__) + + +def check_if_outdated(current_version=None, remote_version=None, source_url="https://nf-co.re/tools_version"): + """ + Check if the current version of nf-core is outdated + """ + # Exit immediately if disabled via ENV var + if os.environ.get("NFCORE_NO_VERSION_CHECK", False): + return True + # Set and clean up the current version string + if current_version == None: + current_version = nf_core.__version__ + current_version = re.sub("[^0-9\.]", "", current_version) + # Build the URL to check against + source_url = os.environ.get("NFCORE_VERSION_URL", source_url) + source_url = "{}?v={}".format(source_url, current_version) + # Fetch and clean up the remote version + if remote_version == None: + response = requests.get(source_url, timeout=3) + remote_version = re.sub("[^0-9\.]", "", response.text) + # Check if we have an available update + is_outdated = version.StrictVersion(remote_version) > version.StrictVersion(current_version) + return (is_outdated, current_version, remote_version) -def fetch_wf_config(wf_path, wf=None): + +def fetch_wf_config(wf_path): """Uses Nextflow to retrieve the the configuration variables from a Nextflow workflow. @@ -28,26 +60,39 @@ def fetch_wf_config(wf_path, wf=None): cache_path = None # Build a cache directory if we can - if os.path.isdir(os.path.join(os.getenv("HOME"), '.nextflow')): - cache_basedir = os.path.join(os.getenv("HOME"), '.nextflow', 'nf-core') + if os.path.isdir(os.path.join(os.getenv("HOME"), ".nextflow")): + cache_basedir = os.path.join(os.getenv("HOME"), ".nextflow", "nf-core") if not os.path.isdir(cache_basedir): os.mkdir(cache_basedir) # If we're given a workflow object with a commit, see if we have a cached copy - if cache_basedir and wf and wf.full_name and wf.commit_sha: - cache_fn = '{}-{}.json'.format(wf.full_name.replace(os.path.sep, '-'), wf.commit_sha) + cache_fn = None + # Make a filename based on file contents + concat_hash = "" + for fn in ["nextflow.config", "main.nf"]: + try: + with open(os.path.join(wf_path, fn), "rb") as fh: + concat_hash += hashlib.sha256(fh.read()).hexdigest() + except FileNotFoundError as e: + pass + # Hash the hash + if len(concat_hash) > 0: + bighash = hashlib.sha256(concat_hash.encode("utf-8")).hexdigest() + cache_fn = "wf-config-cache-{}.json".format(bighash[:25]) + + if cache_basedir and cache_fn: cache_path = os.path.join(cache_basedir, cache_fn) if os.path.isfile(cache_path): - logging.debug("Found a config cache, loading: {}".format(cache_path)) - with open(cache_path, 'r') as fh: + log.debug("Found a config cache, loading: {}".format(cache_path)) + with open(cache_path, "r") as fh: config = json.load(fh) return config - + log.debug("No config cache found") # Call `nextflow config` and pipe stderr to /dev/null try: - with open(os.devnull, 'w') as devnull: - nfconfig_raw = subprocess.check_output(['nextflow', 'config', '-flat', wf_path], stderr=devnull) + with open(os.devnull, "w") as devnull: + nfconfig_raw = subprocess.check_output(["nextflow", "config", "-flat", wf_path], stderr=devnull) except OSError as e: if e.errno == errno.ENOENT: raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") @@ -55,17 +100,29 @@ def fetch_wf_config(wf_path, wf=None): raise AssertionError("`nextflow config` returned non-zero error code: %s,\n %s", e.returncode, e.output) else: for l in nfconfig_raw.splitlines(): - ul = l.decode('utf-8') + ul = l.decode("utf-8") try: - k, v = ul.split(' = ', 1) + k, v = ul.split(" = ", 1) config[k] = v except ValueError: - logging.debug("Couldn't find key=value config pair:\n {}".format(ul)) + log.debug("Couldn't find key=value config pair:\n {}".format(ul)) + + # Scrape main.nf for additional parameter declarations + # Values in this file are likely to be complex, so don't both trying to capture them. Just get the param name. + try: + main_nf = os.path.join(wf_path, "main.nf") + with open(main_nf, "r") as fh: + for l in fh: + match = re.match(r"^\s*(params\.[a-zA-Z0-9_]+)\s*=", l) + if match: + config[match.group(1)] = "false" + except FileNotFoundError as e: + log.debug("Could not open {} to look for parameter declarations - {}".format(main_nf, e)) # If we can, save a cached copy if cache_path: - logging.debug("Saving config cache: {}".format(cache_path)) - with open(cache_path, 'w') as fh: + log.debug("Saving config cache: {}".format(cache_path)) + with open(cache_path, "w") as fh: json.dump(config, fh, indent=4) return config @@ -80,12 +137,92 @@ def setup_requests_cachedir(): # Only import it if we need it import requests_cache - pyversion = '.'.join(str(v) for v in sys.version_info[0:3]) - cachedir = os.path.join(os.getenv("HOME"), os.path.join('.nfcore', 'cache_'+pyversion)) + pyversion = ".".join(str(v) for v in sys.version_info[0:3]) + cachedir = os.path.join(os.getenv("HOME"), os.path.join(".nfcore", "cache_" + pyversion)) if not os.path.exists(cachedir): os.makedirs(cachedir) requests_cache.install_cache( - os.path.join(cachedir, 'github_info'), - expire_after=datetime.timedelta(hours=1), - backend='sqlite', + os.path.join(cachedir, "github_info"), expire_after=datetime.timedelta(hours=1), backend="sqlite", ) + + +def wait_cli_function(poll_func, poll_every=20): + """ + Display a command-line spinner while calling a function repeatedly. + + Keep waiting until that function returns True + + Arguments: + poll_func (function): Function to call + poll_every (int): How many tenths of a second to wait between function calls. Default: 20. + + Returns: + None. Just sits in an infite loop until the function returns True. + """ + try: + is_finished = False + check_count = 0 + + def spinning_cursor(): + while True: + for cursor in "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏": + yield "{} Use ctrl+c to stop waiting and force exit. ".format(cursor) + + spinner = spinning_cursor() + while not is_finished: + # Write a new loading text + loading_text = next(spinner) + sys.stdout.write(loading_text) + sys.stdout.flush() + # Show the loading spinner every 0.1s + time.sleep(0.1) + # Wipe the previous loading text + sys.stdout.write("\b" * len(loading_text)) + sys.stdout.flush() + # Only check every 2 seconds, but update the spinner every 0.1s + check_count += 1 + if check_count > poll_every: + is_finished = poll_func() + check_count = 0 + except KeyboardInterrupt: + raise AssertionError("Cancelled!") + + +def poll_nfcore_web_api(api_url, post_data=None): + """ + Poll the nf-core website API + + Takes argument api_url for URL + + Expects API reponse to be valid JSON and contain a top-level 'status' key. + """ + # Clear requests_cache so that we get the updated statuses + requests_cache.clear() + try: + if post_data is None: + response = requests.get(api_url, headers={"Cache-Control": "no-cache"}) + else: + response = requests.post(url=api_url, data=post_data) + except (requests.exceptions.Timeout): + raise AssertionError("URL timed out: {}".format(api_url)) + except (requests.exceptions.ConnectionError): + raise AssertionError("Could not connect to URL: {}".format(api_url)) + else: + if response.status_code != 200: + log.debug("Response content:\n{}".format(response.content)) + raise AssertionError( + "Could not access remote API results: {} (HTML {} Error)".format(api_url, response.status_code) + ) + else: + try: + web_response = json.loads(response.content) + assert "status" in web_response + except (json.decoder.JSONDecodeError, AssertionError) as e: + log.debug("Response content:\n{}".format(response.content)) + raise AssertionError( + "nf-core website API results response not recognised: {}\n See verbose log for full response".format( + api_url + ) + ) + else: + return web_response diff --git a/nf_core/workflow/__init__.py b/nf_core/workflow/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/nf_core/workflow/parameters.py b/nf_core/workflow/parameters.py deleted file mode 100644 index 424983e8cc..0000000000 --- a/nf_core/workflow/parameters.py +++ /dev/null @@ -1,286 +0,0 @@ -#!/usr/bin/env python - -import copy -import json -import requests -import requests_cache - -from collections import OrderedDict -from jsonschema import validate - -import nf_core.workflow.validation as vld - -NFCORE_PARAMS_SCHEMA_URI = "https://nf-co.re/parameter-schema/0.1.0/parameters.schema.json" - -class Parameters: - """Contains a static factory method - for :class:`Parameter` object creation. - """ - - @staticmethod - def create_from_json(parameters_json, schema_json=""): - """Creates a list of Parameter objects from - a description in JSON. - - Args: - parameters_json (str): Parameter(s) description in JSON. - schema (str): Parameter schema in JSON. - - Returns: - list: Parameter objects. - - Raises: - ValidationError: When the parameter JSON violates the schema. - LookupError: When the schema cannot be downloaded. - """ - # Load the schema and pipeline parameters - if not schema_json: - schema_json = Parameters.__download_schema_from_nf_core(NFCORE_PARAMS_SCHEMA_URI) - schema = json.loads(schema_json, object_pairs_hook=OrderedDict) - properties = json.loads(parameters_json, object_pairs_hook=OrderedDict) - - # Validate the parameters JSON. Throws a ValidationError when schema is violated - validate(properties, schema) - - # Build the parameters object - parameters = [] - for param in properties.get("parameters"): - parameter = (Parameter.builder() - .name(param.get("name")) - .label(param.get("label")) - .usage(param.get("usage")) - .param_type(param.get("type")) - .choices(param.get("choices")) - .default(param.get("default_value")) - .pattern(param.get("pattern")) - .render(param.get("render")) - .arity(param.get("arity")) - .group(param.get("group")) - .build()) - parameters.append(parameter) - return parameters - - @staticmethod - def in_nextflow_json(parameters, indent=0): - """Converts a list of Parameter objects into JSON, readable by Nextflow. - - Args: - parameters (list): List of :class:`Parameter` objects. - indent (integer): String output indentation. Defaults to 0. - - Returns: - list: JSON formatted parameters. - """ - params = {} - for p in parameters: - if p.value and p.value != p.default_value: - params[p.name] = p.value - return json.dumps(params, indent=indent) - - @staticmethod - def in_full_json(parameters, indent=0): - """Converts a list of Parameter objects into JSON. All attributes - are written. - - Args: - parameters (list): List of :class:`Parameter` objects. - indent (integer): String output indentation. Defaults to 0. - - Returns: - list: JSON formatted parameters. - """ - params_dict = {} - params_dict["parameters"] = [p.as_dict() for p in parameters] - return json.dumps(params_dict, indent=indent) - - @classmethod - def __download_schema_from_nf_core(cls, url): - with requests_cache.disabled(): - result = requests.get(url, headers={'Cache-Control': 'no-cache'}) - if not result.status_code == 200: - raise LookupError("Could not fetch schema from {url}.\n{e}".format( - url, result.text)) - return result.text - - -class Parameter(object): - """Holds information about a workflow parameter. - """ - - def __init__(self, param_builder): - # Make some checks - - # Put content - self.name = param_builder.p_name - self.label = param_builder.p_label - self.usage = param_builder.p_usage - self.type = param_builder.p_type - self.value = param_builder.p_value - self.choices = copy.deepcopy(param_builder.p_choices) - self.default_value = param_builder.p_default_value - self.pattern = param_builder.p_pattern - self.arity = param_builder.p_arity - self.required = param_builder.p_required - self.render = param_builder.p_render - self.group = param_builder.p_group - - @staticmethod - def builder(): - return ParameterBuilder() - - def as_dict(self): - """Describes its attibutes in a dictionary. - - Returns: - dict: Parameter object as key value pairs. - """ - params_dict = {} - for attribute in ['name', 'label', 'usage', 'required', - 'type', 'value', 'choices', 'default_value', 'pattern', 'arity', 'render', 'group']: - if getattr(self, attribute): - params_dict[attribute] = getattr(self, attribute) - params_dict['required'] = getattr(self, 'required') - return params_dict - - def validate(self): - """Validates the parameter's value. If the value is within - the parameter requirements, no exception is thrown. - - Raises: - LookupError: Raised when no matching validator can be determined. - AttributeError: Raised with description, if a parameter value violates - the parameter constrains. - """ - validator = vld.Validators.get_validator_for_param(self) - validator.validate() - - -class ParameterBuilder: - """Parameter builder. - """ - - def __init__(self): - self.p_name = "" - self.p_label = "" - self.p_usage = "" - self.p_type = "" - self.p_value = "" - self.p_choices = [] - self.p_default_value = "" - self.p_pattern = "" - self.p_arity = 0 - self.p_render = "" - self.p_required = False - self.p_group = "" - - def group(self, group): - """Sets the parameter group tag - - Args: - group (str): Parameter group tag. - """ - self.p_group = group - return self - - def name(self, name): - """Sets the parameter name. - - Args: - name (str): Parameter name. - """ - self.p_name = name - return self - - def label(self, label): - """Sets the parameter label. - - Args: - label (str): Parameter label. - """ - self.p_label = label - return self - - def usage(self, usage): - """Sets the parameter usage. - - Args: - usage (str): Parameter usage description. - """ - self.p_usage = usage - return self - - def value(self, value): - """Sets the parameter value. - - Args: - value (str): Parameter value. - """ - self.p_value = value - return self - - def choices(self, choices): - """Sets the parameter value choices. - - Args: - choices (list): Parameter value choices. - """ - self.p_choices = choices - return self - - def param_type(self, param_type): - """Sets the parameter type. - - Args: - param_type (str): Parameter type. - """ - self.p_type = param_type - return self - - def default(self, default): - """Sets the parameter default value. - - Args: - default (str): Parameter default value. - """ - self.p_default_value = default - return self - - def pattern(self, pattern): - """Sets the parameter regex pattern. - - Args: - pattern (str): Parameter regex pattern. - """ - self.p_pattern = pattern - return self - - def arity(self, arity): - """Sets the parameter regex pattern. - - Args: - pattern (str): Parameter regex pattern. - """ - self.p_arity = arity - return self - - def render(self, render): - """Sets the parameter render type. - - Args: - render (str): UI render type. - """ - self.p_render = render - return self - - def required(self, required): - """Sets the required parameter flag.""" - self.p_required = required - return self - - def build(self): - """Builds parameter object. - - Returns: - Parameter: Fresh from the factory. - """ - return Parameter(self) diff --git a/nf_core/workflow/validation.py b/nf_core/workflow/validation.py deleted file mode 100644 index fa8ac8ee3f..0000000000 --- a/nf_core/workflow/validation.py +++ /dev/null @@ -1,182 +0,0 @@ -#!/usr/bin/env python - -import abc -import re -import sys - -if sys.version_info >= (3, 4): - ABC = abc.ABC -else: - ABC = abc.ABCMeta('ABC', (), {}) - -class Validators(object): - """Gives access to a factory method for objects of instance - :class:`Validator` which returns the correct Validator for a - given parameter type. - """ - def __init__(self): - pass - - @staticmethod - def get_validator_for_param(parameter): - """Determines matching :class:`Validator` instance for a given parameter. - - Returns: - Validator: Matching validator for a given :class:`Parameter`. - - Raises: - LookupError: In case no matching validator for a given parameter type - can be determined. - """ - if parameter.type == "integer": - return IntegerValidator(parameter) - elif parameter.type == "string": - return StringValidator(parameter) - elif parameter.type == "boolean": - return BooleanValidator(parameter) - elif parameter.type == "decimal": - return DecimalValidator(parameter) - raise LookupError("Cannot find a matching validator for type '{}'." - .format(parameter.type)) - - -class Validator(ABC): - """Abstract base class for different parameter validators. - """ - __metaclass__ = abc.ABCMeta - - @abc.abstractmethod - def __init__(self, parameter): - self._param = parameter - - @abc.abstractmethod - def validate(self): - raise ValueError - - -class IntegerValidator(Validator): - """Implementation for parameters of type integer. - - Args: - parameter (:class:`Parameter`): A Parameter object. - - Raises: - AttributeError: In case the argument is not of instance integer. - """ - - def __init__(self, parameter): - super(IntegerValidator, self).__init__(parameter) - - def validate(self): - """Validates an parameter integer value against a given range (choices). - If the value is valid, no error is risen. - - Raises: - AtrributeError: Description of the value error. - """ - value = self._param.value - if not isinstance(value, int): - raise AttributeError("The value {} for parameter {} needs to be an Integer, but was a {}" - .format(value, self._param.name, type(value))) - if self._param.choices: - choices = sorted([x for x in self._param.choices]) - if len(choices) < 2: - raise AttributeError("The property 'choices' must have at least two entries.") - if not (value >= choices[0] and value <= choices[-1]): - raise AttributeError("'{}' must be within the range [{},{}]" - .format(self._param.name, choices[0], choices[-1])) - - -class StringValidator(Validator): - """Implementation for parameters of type string. - - Args: - parameter (:class:`Parameter`): A Parameter object. - - Raises: - AttributeError: In case the argument is not of instance string. - """ - - def __init__(self, parameter): - super(StringValidator, self).__init__(parameter) - - def validate(self): - """Validates an parameter integer value against a given range (choices). - If the value is valid, no error is risen. - - Raises: - AtrributeError: Description of the value error. - """ - value = self._param.value - if not isinstance(value, str): - raise AttributeError("The value {} for parameter {} needs to be of type String, but was {}" - .format(value, self._param.name, type(value))) - choices = sorted([x for x in self._param.choices]) if self._param.choices else [] - if not choices: - if not self._param.pattern: - raise AttributeError("Can't validate value for parameter '{}', " \ - "because the value for 'choices' and 'pattern' were empty.".format(self._param.value)) - result = re.match(self._param.pattern, self._param.value) - if not result: - raise AttributeError("'{}' doesn't match the regex pattern '{}'".format( - self._param.value, self._param.pattern - )) - else: - if value not in choices: - raise AttributeError( - "'{}' is not not one of the choices {}".format( - value, str(choices) - ) - ) - - -class BooleanValidator(Validator): - """Implementation for parameters of type boolean. - - Args: - parameter (:class:`Parameter`): A Parameter object. - - Raises: - AttributeError: In case the argument is not of instance boolean. - """ - - def __init__(self, parameter): - super(BooleanValidator, self).__init__(parameter) - - def validate(self): - """Validates an parameter boolean value. - If the value is valid, no error is risen. - - Raises: - AtrributeError: Description of the value error. - """ - value = self._param.value - if not isinstance(self._param.value, bool): - raise AttributeError("The value {} for parameter {} needs to be of type Boolean, but was {}" - .format(value, self._param.name, type(value))) - - -class DecimalValidator(Validator): - """Implementation for parameters of type boolean. - - Args: - parameter (:class:`Parameter`): A Parameter object. - - Raises: - AttributeError: In case the argument is not of instance decimal. - """ - - def __init__(self, parameter): - super(DecimalValidator, self).__init__(parameter) - - def validate(self): - """Validates an parameter boolean value. - If the value is valid, no error is risen. - - Raises: - AtrributeError: Description of the value error. - """ - value = self._param.value - if not isinstance(self._param.value, float): - raise AttributeError("The value {} for parameter {} needs to be of type Decimal, but was {}" - .format(value, self._param.name, type(value))) diff --git a/nf_core/workflow/workflow.py b/nf_core/workflow/workflow.py deleted file mode 100644 index ae59c8958e..0000000000 --- a/nf_core/workflow/workflow.py +++ /dev/null @@ -1,33 +0,0 @@ -#!/usr/bin/env python - -from nf_core.workflow.parameters import Parameters - - -class Workflow(object): - """nf-core workflow object that holds run parameter information. - - Args: - name (str): Workflow name. - parameters_json (str): Workflow parameter data in JSON. - """ - def __init__(self, name, parameters_json): - self.name = name - self.parameters = Parameters.create_from_json(parameters_json) - - def in_nextflow_json(self, indent=0): - """Converts the Parameter list in a workflow readable parameter - JSON file. - - Returns: - str: JSON formatted parameters. - """ - return Parameters.in_nextflow_json(self.parameters, indent) - - def in_full_json(self, indent=0): - """Converts the Parameter list in a complete parameter JSON for - schema validation. - - Returns: - str: JSON formatted parameters. - """ - return Parameters.in_full_json(self.parameters, indent) diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000000..2d9759a7fb --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,3 @@ +[tool.black] +line-length = 120 +target_version = ['py36','py37','py38'] diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000000..10e124429e --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,5 @@ +pytest +pytest-datafiles +pytest-cov +mock +black diff --git a/scripts/nf-core b/scripts/nf-core deleted file mode 100755 index 65e0311114..0000000000 --- a/scripts/nf-core +++ /dev/null @@ -1,353 +0,0 @@ -#!/usr/bin/env python -""" nf-core: Helper tools for use with nf-core Nextflow pipelines. """ - -from __future__ import print_function - -import click -import sys -import os -import re - -import nf_core -import nf_core.bump_version -import nf_core.create -import nf_core.download -import nf_core.launch -import nf_core.licences -import nf_core.lint -import nf_core.list -import nf_core.sync - -import logging - -# Customise the order of subcommands for --help -# https://stackoverflow.com/a/47984810/713980 -class CustomHelpOrder(click.Group): - def __init__(self, *args, **kwargs): - self.help_priorities = {} - super(CustomHelpOrder, self).__init__(*args, **kwargs) - - def get_help(self, ctx): - self.list_commands = self.list_commands_for_help - return super(CustomHelpOrder, self).get_help(ctx) - - def list_commands_for_help(self, ctx): - """reorder the list of commands when listing the help""" - commands = super(CustomHelpOrder, self).list_commands(ctx) - return (c[1] for c in sorted((self.help_priorities.get(command, 1000), command) for command in commands)) - - def command(self, *args, **kwargs): - """Behaves the same as `click.Group.command()` except capture - a priority for listing command names in help. - """ - help_priority = kwargs.pop('help_priority', 1000) - help_priorities = self.help_priorities - def decorator(f): - cmd = super(CustomHelpOrder, self).command(*args, **kwargs)(f) - help_priorities[cmd.name] = help_priority - return cmd - return decorator - -@click.group(cls=CustomHelpOrder) -@click.version_option(nf_core.__version__) -@click.option( - '-v', '--verbose', - is_flag = True, - default = False, - help = "Verbose output (print debug statements)" -) -def nf_core_cli(verbose): - if verbose: - logging.basicConfig(level=logging.DEBUG, format="\n%(levelname)s: %(message)s") - else: - logging.basicConfig(level=logging.INFO, format="\n%(levelname)s: %(message)s") - -# nf-core list -@nf_core_cli.command(help_priority=1) -@click.argument( - 'keywords', - required = False, - nargs = -1, - metavar = "" -) -@click.option( - '-s', '--sort', - type = click.Choice(['release', 'pulled', 'name', 'stars']), - default = 'release', - help = "How to sort listed pipelines" -) -@click.option( - '--json', - is_flag = True, - default = False, - help = "Print full output as JSON" -) -def list(keywords, sort, json): - """ List nf-core pipelines with local info """ - nf_core.list.list_workflows(keywords, sort, json) - -# nf-core launch -@nf_core_cli.command(help_priority=2) -@click.argument( - 'pipeline', - required = True, - metavar = "" -) -@click.option( - '-p', '--params', - type = str, - help = "Local parameter settings file in JSON." -) -@click.option( - '-d', '--direct', - is_flag = True, - help = "Uses given values from the parameter file directly." -) -def launch(pipeline, params, direct): - """ Run pipeline, interactive parameter prompts """ - nf_core.launch.launch_pipeline(pipeline, params, direct) - -# nf-core download -@nf_core_cli.command(help_priority=3) -@click.argument( - 'pipeline', - required = True, - metavar = "" -) -@click.option( - '-r', '--release', - type = str, - help = "Pipeline release" -) -@click.option( - '-s', '--singularity', - is_flag = True, - default = False, - help = "Download singularity containers" -) -@click.option( - '-o', '--outdir', - type = str, - help = "Output directory" -) -@click.option( - '-c', '--compress', - type = click.Choice(['tar.gz', 'tar.bz2', 'zip', 'none']), - default = 'tar.gz', - help = "Compression type" -) -def download(pipeline, release, singularity, outdir, compress): - """ Download a pipeline and singularity container """ - dl = nf_core.download.DownloadWorkflow(pipeline, release, singularity, outdir, compress) - dl.download_workflow() - -# nf-core licences -@nf_core_cli.command(help_priority=4) -@click.argument( - 'pipeline', - required = True, - metavar = "" -) -@click.option( - '--json', - is_flag = True, - default = False, - help = "Print output in JSON" -) -def licences(pipeline, json): - """ List software licences for a given workflow """ - lic = nf_core.licences.WorkflowLicences(pipeline) - lic.fetch_conda_licences() - lic.print_licences(as_json=json) - -# nf-core create -def validate_wf_name_prompt(ctx, opts, value): - """ Force the workflow name to meet the nf-core requirements """ - if not re.match(r'^[a-z]+$', value): - click.echo('Invalid workflow name: must be lowercase without punctuation.') - value = click.prompt(opts.prompt) - return validate_wf_name_prompt(ctx, opts, value) - return value -@nf_core_cli.command(help_priority=5) -@click.option( - '-n', '--name', - prompt = 'Workflow Name', - required = True, - callback = validate_wf_name_prompt, - type = str, - help = 'The name of your new pipeline' -) -@click.option( - '-d', '--description', - prompt = True, - required = True, - type = str, - help = 'A short description of your pipeline' -) -@click.option( - '-a', '--author', - prompt = True, - required = True, - type = str, - help = 'Name of the main author(s)' -) -@click.option( - '--new-version', - type = str, - default = '1.0dev', - help = 'The initial version number to use' -) -@click.option( - '--no-git', - is_flag = True, - default = False, - help = "Do not initialise pipeline as new git repository" -) -@click.option( - '-f', '--force', - is_flag = True, - default = False, - help = "Overwrite output directory if it already exists" -) -@click.option( - '-o', '--outdir', - type = str, - help = "Output directory for new pipeline (default: pipeline name)" -) -def create(name, description, author, new_version, no_git, force, outdir): - """ Create a new pipeline using the template """ - create_obj = nf_core.create.PipelineCreate(name, description, author, new_version, no_git, force, outdir) - create_obj.init_pipeline() - -@nf_core_cli.command(help_priority=6) -@click.argument( - 'pipeline_dir', - type = click.Path(exists=True), - required = True, - metavar = "" -) -@click.option( - '--release', - is_flag = True, - default = os.path.basename(os.path.dirname(os.environ.get('GITHUB_REF','').strip(' \'"'))) == 'master' and os.environ.get('GITHUB_REPOSITORY', '').startswith('nf-core/') and not os.environ.get('GITHUB_REPOSITORY', '') == 'nf-core/tools', - help = "Execute additional checks for release-ready workflows." -) -def lint(pipeline_dir, release): - """ Check pipeline against nf-core guidelines """ - - # Run the lint tests! - lint_obj = nf_core.lint.run_linting(pipeline_dir, release) - if len(lint_obj.failed) > 0: - sys.exit(1) - - -@nf_core_cli.command('bump-version', help_priority=7) -@click.argument( - 'pipeline_dir', - type = click.Path(exists=True), - required = True, - metavar = "" -) -@click.argument( - 'new_version', - required = True, - metavar = "" -) -@click.option( - '-n', '--nextflow', - is_flag = True, - default = False, - help = "Bump required nextflow version instead of pipeline version" -) -def bump_version(pipeline_dir, new_version, nextflow): - """ Update nf-core pipeline version number """ - - # First, lint the pipeline to check everything is in order - logging.info("Running nf-core lint tests") - lint_obj = nf_core.lint.run_linting(pipeline_dir, False) - if len(lint_obj.failed) > 0: - logging.error("Please fix lint errors before bumping versions") - return - - # Bump the pipeline version number - if not nextflow: - nf_core.bump_version.bump_pipeline_version(lint_obj, new_version) - else: - nf_core.bump_version.bump_nextflow_version(lint_obj, new_version) - - -@nf_core_cli.command('sync', help_priority=8) -@click.argument( - 'pipeline_dir', - type = click.Path(exists=True), - nargs = -1, - metavar = "" -) -@click.option( - '-t', '--make-template-branch', - is_flag = True, - default = False, - help = "Create a TEMPLATE branch if none is found." -) -@click.option( - '-b', '--from-branch', - type = str, - help = 'The git branch to use to fetch workflow vars.' -) -@click.option( - '-p', '--pull-request', - is_flag = True, - default = False, - help = "Make a GitHub pull-request with the changes." -) -@click.option( - '-u', '--username', - type = str, - help = 'GitHub username for the PR.' -) -@click.option( - '-r', '--repository', - type = str, - help = 'GitHub repository name for the PR.' -) -@click.option( - '-a', '--auth-token', - type = str, - help = 'GitHub API personal access token.' -) -@click.option( - '--all', - is_flag = True, - default = False, - help = "Sync template for all nf-core pipelines." -) -def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username, repository, auth_token, all): - """ Sync a pipeline TEMPLATE branch with the nf-core template""" - - # Pull and sync all nf-core pipelines - if all: - nf_core.sync.sync_all_pipelines(username, auth_token) - else: - # Manually check for the required parameter - if not pipeline_dir or len(pipeline_dir) != 1: - logging.error("Either use --all or specify one ") - sys.exit(1) - else: - pipeline_dir = pipeline_dir[0] - - # Sync the given pipeline dir - sync_obj = nf_core.sync.PipelineSync(pipeline_dir, make_template_branch, from_branch, pull_request) - try: - sync_obj.sync() - except (nf_core.sync.SyncException, nf_core.sync.PullRequestException) as e: - logging.error(e) - sys.exit(1) - - -if __name__ == '__main__': - click.echo(click.style("\n ,--.", fg='green')+click.style("/",fg='black')+click.style(",-.", fg='green'), err=True) - click.echo(click.style(" ___ __ __ __ ___ ", fg='blue')+click.style("/,-._.--~\\", fg='green'), err=True) - click.echo(click.style(" |\ | |__ __ / ` / \ |__) |__ ", fg='blue')+click.style(" } {", fg='yellow'), err=True) - click.echo(click.style(" | \| | \__, \__/ | \ |___ ", fg='blue')+click.style("\`-._,-`-,", fg='green'), err=True) - click.secho(" `._,._,'\n", fg='green', err=True) - nf_core_cli() diff --git a/setup.py b/setup.py index 54bfb7e1a8..180f97f086 100644 --- a/setup.py +++ b/setup.py @@ -3,38 +3,48 @@ from setuptools import setup, find_packages import sys -version = '1.9' +version = "1.10" -with open('README.md') as f: +with open("README.md") as f: readme = f.read() setup( - name = 'nf-core', - version = version, - description = 'Helper tools for use with nf-core Nextflow pipelines.', - long_description = readme, - long_description_content_type='text/markdown', - keywords = ['nf-core', 'nextflow', 'bioinformatics', 'workflow', 'pipeline', 'biology', 'sequencing', 'NGS', 'next generation sequencing'], - author = 'Phil Ewels', - author_email = 'phil.ewels@scilifelab.se', - url = 'https://github.com/nf-core/tools', - license = 'MIT', - scripts = ['scripts/nf-core'], - install_requires = [ - 'cookiecutter', - 'click', - 'GitPython', - 'jsonschema', - 'pyyaml', - 'requests', - 'requests_cache', - 'tabulate' + name="nf-core", + version=version, + description="Helper tools for use with nf-core Nextflow pipelines.", + long_description=readme, + long_description_content_type="text/markdown", + keywords=[ + "nf-core", + "nextflow", + "bioinformatics", + "workflow", + "pipeline", + "biology", + "sequencing", + "NGS", + "next generation sequencing", ], - setup_requires=[ - 'twine>=1.11.0', - 'setuptools>=38.6.' + author="Phil Ewels", + author_email="phil.ewels@scilifelab.se", + url="https://github.com/nf-core/tools", + license="MIT", + entry_points={"console_scripts": ["nf-core=nf_core.__main__:run_nf_core"]}, + install_requires=[ + "cookiecutter", + "click", + "GitPython", + "jinja2", + "jsonschema", + "PyInquirer==1.0.2", + "pyyaml", + "requests", + "requests_cache", + "rich>=4.0.0", + "tabulate", ], - packages = find_packages(exclude=('docs')), - include_package_data = True, - zip_safe = False + setup_requires=["twine>=1.11.0", "setuptools>=38.6."], + packages=find_packages(exclude=("docs")), + include_package_data=True, + zip_safe=False, ) diff --git a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml new file mode 100644 index 0000000000..0563e646e4 --- /dev/null +++ b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml @@ -0,0 +1,41 @@ +name: nf-core AWS full size tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test_full' on AWS batch + +on: + push: + branches: + - master + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == 'nf-core/tools' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + # TODO nf-core: You can customise AWS full pipeline tests as required + # Add full size test data (but still relatively small datasets for few samples) + # on the `test_full.config` test runs with only one set of parameters + # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-tools \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/failing_example/.github/workflows/awstest.yml b/tests/lint_examples/failing_example/.github/workflows/awstest.yml new file mode 100644 index 0000000000..8e72d862db --- /dev/null +++ b/tests/lint_examples/failing_example/.github/workflows/awstest.yml @@ -0,0 +1,44 @@ +name: nf-core AWS tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test' on AWS batch + +on: + push: + branches: + - master + - dev + pull_request: + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == 'nf-core/tools' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + # TODO nf-core: You can customise CI pipeline run tests as required + # For example: adding multiple test runs with different parameters + # Remember that you can parallelise this by using strategy.matrix + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-tools \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/failing_example/nextflow.config b/tests/lint_examples/failing_example/nextflow.config index 44e35592c1..38dc8ee1b6 100644 --- a/tests/lint_examples/failing_example/nextflow.config +++ b/tests/lint_examples/failing_example/nextflow.config @@ -1,4 +1,4 @@ -manifest.homePage = 'http://nf-co.re/pipelines' +manifest.homePage = 'https://nf-co.re/pipelines' manifest.name = 'pipelines' manifest.nextflowVersion = '0.30.1' manifest.version = '0.4dev' diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml new file mode 100644 index 0000000000..99c9ab9165 --- /dev/null +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml @@ -0,0 +1,36 @@ +name: nf-core AWS full size tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test_full' on AWS batch + +on: + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == 'nf-core/tools' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-tools \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test_full --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml new file mode 100644 index 0000000000..3d39c4505a --- /dev/null +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml @@ -0,0 +1,37 @@ +name: nf-core AWS tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test' on AWS batch + +on: + push: + branches: + - master + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == 'nf-core/tools' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-tools \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml index 306f89dfec..49836c50f2 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml @@ -3,15 +3,14 @@ name: nf-core branch protection # It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev` on: pull_request: - branches: - - master + branches: [master] jobs: test: - runs-on: ubuntu-18.04 + runs-on: ubuntu-latest steps: - - uses: actions/checkout@v1 - # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches - name: Check PRs + if: github.repository == 'nf-core/tools' run: | - { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml index 00d4faad99..43adc25b7f 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml @@ -1,26 +1,85 @@ name: nf-core CI -# This workflow is triggered on pushes and PRs to the repository. +# This workflow is triggered on releases and pull-requests. # It runs the pipeline with the minimal test dataset to check that it completes without any syntax errors -on: [push, pull_request] +on: + push: + branches: + - dev + pull_request: + release: + types: [published] jobs: test: - runs-on: ubuntu-18.04 + name: Run workflow tests + # Only run on push if this is the nf-core dev branch (merged PRs) + if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/tools') }} + runs-on: ubuntu-latest + env: + NXF_VER: ${{ matrix.nxf_ver }} + NXF_ANSI_LOG: false strategy: matrix: # Nextflow versions: check pipeline minimum and current latest nxf_ver: ['19.10.0', ''] steps: - - uses: actions/checkout@v1 + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Check if Dockerfile or Conda environment changed + uses: technote-space/get-diff-action@v1 + with: + PREFIX_FILTER: | + Dockerfile + environment.yml + + - name: Build new docker image + if: env.GIT_DIFF + run: docker build --no-cache . -t nfcore/tools:0.4 + + - name: Pull docker image + if: ${{ !env.GIT_DIFF }} + run: | + docker pull nfcore/tools:dev + docker tag nfcore/tools:dev nfcore/tools:0.4 + - name: Install Nextflow run: | - {% raw %}export NXF_VER=${{ matrix.nxf_ver }}{% endraw %} wget -qO- get.nextflow.io | bash sudo mv nextflow /usr/local/bin/ - - name: Pull container - run: | - docker pull nfcore/tools:dev - docker tag nfcore/tools:dev nfcore/tools:0.4 - - name: Run test + + - name: Run pipeline with test data run: | nextflow run ${GITHUB_WORKSPACE} -profile test,docker + + push_dockerhub: + name: Push new Docker image to Docker Hub + runs-on: ubuntu-latest + # Only run if the tests passed + needs: test + # Only run for the nf-core repo, for releases and merged PRs + if: ${{ github.repository == 'nf-core/tools' && (github.event_name == 'release' || github.event_name == 'push') }} + env: + DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }} + DOCKERHUB_PASS: ${{ secrets.DOCKERHUB_PASS }} + steps: + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Build new docker image + run: docker build --no-cache . -t nfcore/tools:latest + + - name: Push Docker image to DockerHub (dev) + if: ${{ github.event_name == 'push' }} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker tag nfcore/tools:latest nfcore/tools:dev + docker push nfcore/tools:dev + + - name: Push Docker image to DockerHub (release) + if: ${{ github.event_name == 'release' }} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker push nfcore/tools:latest + docker tag nfcore/tools:latest nfcore/tools:${{ github.event.release.tag_name }} + docker push nfcore/tools:${{ github.event.release.tag_name }} diff --git a/tests/lint_examples/minimalworkingexample/Dockerfile b/tests/lint_examples/minimalworkingexample/Dockerfile index 0b3f1d2876..7f033b0331 100644 --- a/tests/lint_examples/minimalworkingexample/Dockerfile +++ b/tests/lint_examples/minimalworkingexample/Dockerfile @@ -4,6 +4,6 @@ LABEL authors="phil.ewels@scilifelab.se" \ description="Docker image containing all requirements for the nf-core tools pipeline" COPY environment.yml / -RUN conda env create -f /environment.yml && conda clean -a +RUN conda env create --quiet -f /environment.yml && conda clean -a RUN conda env export --name nf-core-tools-0.4 > nf-core-tools-0.4.yml ENV PATH /opt/conda/envs/nf-core-tools-0.4/bin:$PATH diff --git a/tests/lint_examples/minimalworkingexample/README.md b/tests/lint_examples/minimalworkingexample/README.md index 838a6faefe..fc732ac055 100644 --- a/tests/lint_examples/minimalworkingexample/README.md +++ b/tests/lint_examples/minimalworkingexample/README.md @@ -2,4 +2,4 @@ [![Nextflow](https://img.shields.io/badge/nextflow-%E2%89%A519.10.0-brightgreen.svg)](https://www.nextflow.io/) -[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/) +[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) diff --git a/tests/lint_examples/minimalworkingexample/environment.yml b/tests/lint_examples/minimalworkingexample/environment.yml index 75bd6b2913..c40b9fce5a 100644 --- a/tests/lint_examples/minimalworkingexample/environment.yml +++ b/tests/lint_examples/minimalworkingexample/environment.yml @@ -2,11 +2,12 @@ # conda env create -f environment.yml name: nf-core-tools-0.4 channels: - - defaults - conda-forge - bioconda + - defaults dependencies: - conda-forge::openjdk=8.0.144 + - conda-forge::markdown=3.1.1=py_0 - fastqc=0.11.7 - pip: - multiqc==1.4 diff --git a/tests/lint_examples/minimalworkingexample/nextflow.config b/tests/lint_examples/minimalworkingexample/nextflow.config index 303d675d2d..cbe2163a94 100644 --- a/tests/lint_examples/minimalworkingexample/nextflow.config +++ b/tests/lint_examples/minimalworkingexample/nextflow.config @@ -1,7 +1,7 @@ params { outdir = './results' - reads = "data/*.fastq" + input = "data/*.fastq" single_end = false custom_config_version = 'master' custom_config_base = "https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}" diff --git a/tests/lint_examples/minimalworkingexample/nextflow_schema.json b/tests/lint_examples/minimalworkingexample/nextflow_schema.json new file mode 100644 index 0000000000..bbf2bbe9eb --- /dev/null +++ b/tests/lint_examples/minimalworkingexample/nextflow_schema.json @@ -0,0 +1,29 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/nf-core/tools/master/nextflow_schema.json", + "title": "nf-core/tools pipeline parameters", + "description": "Minimal working example pipeline", + "type": "object", + "properties": { + "outdir": { + "type": "string", + "default": "'./results'" + }, + "input": { + "type": "string", + "default": "'data/*.fastq'" + }, + "single_end": { + "type": "string", + "default": "false" + }, + "custom_config_version": { + "type": "string", + "default": "'master'" + }, + "custom_config_base": { + "type": "string", + "default": "'https://raw.githubusercontent.com/nf-core/configs/master'" + } + } +} diff --git a/tests/test_bump_version.py b/tests/test_bump_version.py index aa81e8520b..5eca5d239f 100644 --- a/tests/test_bump_version.py +++ b/tests/test_bump_version.py @@ -3,60 +3,62 @@ """ import os import pytest -import shutil -import unittest import nf_core.lint, nf_core.bump_version WD = os.path.dirname(__file__) -PATH_WORKING_EXAMPLE = os.path.join(WD, 'lint_examples/minimalworkingexample') +PATH_WORKING_EXAMPLE = os.path.join(WD, "lint_examples/minimalworkingexample") @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) def test_working_bump_pipeline_version(datafiles): """ Test that making a release with the working example files works """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.files = ['nextflow.config', 'Dockerfile', 'environment.yml'] - nf_core.bump_version.bump_pipeline_version(lint_obj, '1.1') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.files = ["nextflow.config", "Dockerfile", "environment.yml"] + nf_core.bump_version.bump_pipeline_version(lint_obj, "1.1") + @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) def test_dev_bump_pipeline_version(datafiles): """ Test that making a release works with a dev name and a leading v """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.files = ['nextflow.config', 'Dockerfile', 'environment.yml'] - nf_core.bump_version.bump_pipeline_version(lint_obj, 'v1.2dev') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.files = ["nextflow.config", "Dockerfile", "environment.yml"] + nf_core.bump_version.bump_pipeline_version(lint_obj, "v1.2dev") + @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) -@pytest.mark.xfail(raises=SyntaxError) +@pytest.mark.xfail(raises=SyntaxError, strict=True) def test_pattern_not_found(datafiles): """ Test that making a release raises and error if a pattern isn't found """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.5' - lint_obj.files = ['nextflow.config', 'Dockerfile', 'environment.yml'] - nf_core.bump_version.bump_pipeline_version(lint_obj, '1.2dev') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.5" + lint_obj.files = ["nextflow.config", "Dockerfile", "environment.yml"] + nf_core.bump_version.bump_pipeline_version(lint_obj, "1.2dev") + @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) -@pytest.mark.xfail(raises=SyntaxError) +@pytest.mark.xfail(raises=SyntaxError, strict=True) def test_multiple_patterns_found(datafiles): """ Test that making a release raises if a version number is found twice """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - with open(os.path.join(str(datafiles), 'nextflow.config'), "a") as nfcfg: + with open(os.path.join(str(datafiles), "nextflow.config"), "a") as nfcfg: nfcfg.write("manifest.version = '0.4'") - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.files = ['nextflow.config', 'Dockerfile', 'environment.yml'] - nf_core.bump_version.bump_pipeline_version(lint_obj, '1.2dev') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.files = ["nextflow.config", "Dockerfile", "environment.yml"] + nf_core.bump_version.bump_pipeline_version(lint_obj, "1.2dev") + @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) def test_successfull_nextflow_version_bump(datafiles): lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.nextflowVersion'] = '19.10.0' - nf_core.bump_version.bump_nextflow_version(lint_obj, '0.40') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.nextflowVersion"] = "19.10.0" + nf_core.bump_version.bump_nextflow_version(lint_obj, "0.40") lint_obj_new = nf_core.lint.PipelineLint(str(datafiles)) lint_obj_new.check_nextflow_config() - assert lint_obj_new.config['manifest.nextflowVersion'] == "'>=0.40'" + assert lint_obj_new.config["manifest.nextflowVersion"] == "'>=0.40'" diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000000..eb1ab6f9df --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python +""" Tests covering the command-line code. +""" + +import nf_core.__main__ + +from click.testing import CliRunner +import mock +import unittest + + +@mock.patch("nf_core.__main__.nf_core_cli") +def test_header(mock_cli): + """ Just try to execute the header function """ + nf_core.__main__.run_nf_core() + + +def test_cli_help(): + """ Test the main launch function with --help """ + runner = CliRunner() + result = runner.invoke(nf_core.__main__.nf_core_cli, ["--help"]) + assert result.exit_code == 0 + assert "Show the version and exit." in result.output + + +def test_cli_bad_subcommand(): + """ Test the main launch function with verbose flag and an unrecognised argument """ + runner = CliRunner() + result = runner.invoke(nf_core.__main__.nf_core_cli, ["-v", "foo"]) + assert result.exit_code == 2 + # Checks that -v was considered valid + assert "No such command" in result.output diff --git a/tests/test_create.py b/tests/test_create.py index cf03533d23..8d527891d3 100644 --- a/tests/test_create.py +++ b/tests/test_create.py @@ -2,29 +2,29 @@ """Some tests covering the pipeline creation sub command. """ import os -import pytest -import nf_core.lint, nf_core.create +import nf_core.create import tempfile import unittest WD = os.path.dirname(__file__) -PIPELINE_NAME = 'nf-core/test' -PIPELINE_DESCRIPTION = 'just for 4w3s0m3 tests' -PIPELINE_AUTHOR = 'Chuck Norris' -PIPELINE_VERSION = '1.0.0' +PIPELINE_NAME = "nf-core/test" +PIPELINE_DESCRIPTION = "just for 4w3s0m3 tests" +PIPELINE_AUTHOR = "Chuck Norris" +PIPELINE_VERSION = "1.0.0" -class NfcoreCreateTest(unittest.TestCase): +class NfcoreCreateTest(unittest.TestCase): def setUp(self): self.tmppath = tempfile.mkdtemp() - self.pipeline = nf_core.create.PipelineCreate(name=PIPELINE_NAME, - description=PIPELINE_DESCRIPTION, - author=PIPELINE_AUTHOR, - new_version=PIPELINE_VERSION, - no_git=False, - force=True, - outdir=self.tmppath) - + self.pipeline = nf_core.create.PipelineCreate( + name=PIPELINE_NAME, + description=PIPELINE_DESCRIPTION, + author=PIPELINE_AUTHOR, + new_version=PIPELINE_VERSION, + no_git=False, + force=True, + outdir=self.tmppath, + ) def test_pipeline_creation(self): assert self.pipeline.name == PIPELINE_NAME @@ -34,4 +34,4 @@ def test_pipeline_creation(self): def test_pipeline_creation_initiation(self): self.pipeline.init_pipeline() - assert (os.path.isdir(os.path.join(self.pipeline.outdir, '.git'))) + assert os.path.isdir(os.path.join(self.pipeline.outdir, ".git")) diff --git a/tests/test_download.py b/tests/test_download.py index 18bf26c555..fe10592aa6 100644 --- a/tests/test_download.py +++ b/tests/test_download.py @@ -2,54 +2,49 @@ """Tests for the download subcommand of nf-core tools """ -import nf_core.list import nf_core.utils from nf_core.download import DownloadWorkflow import hashlib -import io import mock import os import pytest -import requests import shutil import tempfile import unittest -PATH_WORKING_EXAMPLE = os.path.join(os.path.dirname(__file__), 'lint_examples/minimalworkingexample') +PATH_WORKING_EXAMPLE = os.path.join(os.path.dirname(__file__), "lint_examples/minimalworkingexample") + class DownloadTest(unittest.TestCase): # # Tests for 'fetch_workflow_details()' # - @mock.patch('nf_core.list.RemoteWorkflow') - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.RemoteWorkflow") + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_for_release(self, mock_workflows, mock_workflow): - download_obj = DownloadWorkflow( - pipeline = "dummy", - release="1.0.0" - ) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.0.0") mock_workflow.name = "dummy" mock_workflow.releases = [{"tag_name": "1.0.0", "tag_sha": "n3v3rl4nd"}] mock_workflows.remote_workflows = [mock_workflow] download_obj.fetch_workflow_details(mock_workflows) - @mock.patch('nf_core.list.RemoteWorkflow') - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.RemoteWorkflow") + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_for_dev_version(self, mock_workflows, mock_workflow): - download_obj = DownloadWorkflow(pipeline = "dummy") + download_obj = DownloadWorkflow(pipeline="dummy") mock_workflow.name = "dummy" mock_workflow.releases = [] mock_workflows.remote_workflows = [mock_workflow] download_obj.fetch_workflow_details(mock_workflows) - @mock.patch('nf_core.list.RemoteWorkflow') - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.RemoteWorkflow") + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_and_autoset_release(self, mock_workflows, mock_workflow): - download_obj = DownloadWorkflow(pipeline = "dummy") + download_obj = DownloadWorkflow(pipeline="dummy") mock_workflow.name = "dummy" mock_workflow.releases = [{"tag_name": "1.0.0", "tag_sha": "n3v3rl4nd"}] mock_workflows.remote_workflows = [mock_workflow] @@ -57,47 +52,36 @@ def test_fetch_workflow_details_and_autoset_release(self, mock_workflows, mock_w download_obj.fetch_workflow_details(mock_workflows) assert download_obj.release == "1.0.0" - @mock.patch('nf_core.list.RemoteWorkflow') - @mock.patch('nf_core.list.Workflows') - @pytest.mark.xfail(raises=LookupError) + @mock.patch("nf_core.list.RemoteWorkflow") + @mock.patch("nf_core.list.Workflows") + @pytest.mark.xfail(raises=LookupError, strict=True) def test_fetch_workflow_details_for_unknown_release(self, mock_workflows, mock_workflow): - download_obj = DownloadWorkflow( - pipeline = "dummy", - release = "1.2.0" - ) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0") mock_workflow.name = "dummy" mock_workflow.releases = [{"tag_name": "1.0.0", "tag_sha": "n3v3rl4nd"}] mock_workflows.remote_workflows = [mock_workflow] download_obj.fetch_workflow_details(mock_workflows) - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_for_github_ressource(self, mock_workflows): - download_obj = DownloadWorkflow( - pipeline = "myorg/dummy", - release = "1.2.0" - ) + download_obj = DownloadWorkflow(pipeline="myorg/dummy", release="1.2.0") mock_workflows.remote_workflows = [] download_obj.fetch_workflow_details(mock_workflows) - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_for_github_ressource_take_master(self, mock_workflows): - download_obj = DownloadWorkflow( - pipeline = "myorg/dummy" - ) + download_obj = DownloadWorkflow(pipeline="myorg/dummy") mock_workflows.remote_workflows = [] download_obj.fetch_workflow_details(mock_workflows) assert download_obj.release == "master" - @mock.patch('nf_core.list.Workflows') - @pytest.mark.xfail(raises=LookupError) + @mock.patch("nf_core.list.Workflows") + @pytest.mark.xfail(raises=LookupError, strict=True) def test_fetch_workflow_details_no_search_result(self, mock_workflows): - download_obj = DownloadWorkflow( - pipeline = "http://my-server.org/dummy", - release = "1.2.0" - ) + download_obj = DownloadWorkflow(pipeline="http://my-server.org/dummy", release="1.2.0") mock_workflows.remote_workflows = [] download_obj.fetch_workflow_details(mock_workflows) @@ -106,11 +90,7 @@ def test_fetch_workflow_details_no_search_result(self, mock_workflows): # Tests for 'download_wf_files' # def test_download_wf_files(self): - download_obj = DownloadWorkflow( - pipeline = "dummy", - release = "1.2.0", - outdir = tempfile.mkdtemp() - ) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0", outdir=tempfile.mkdtemp()) download_obj.wf_name = "nf-core/methylseq" download_obj.wf_sha = "1.0" download_obj.wf_download_url = "https://github.com/nf-core/methylseq/archive/1.0.zip" @@ -120,11 +100,7 @@ def test_download_wf_files(self): # Tests for 'download_configs' # def test_download_configs(self): - download_obj = DownloadWorkflow( - pipeline = "dummy", - release = "1.2.0", - outdir = tempfile.mkdtemp() - ) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0", outdir=tempfile.mkdtemp()) download_obj.download_configs() # @@ -133,61 +109,57 @@ def test_download_configs(self): def test_wf_use_local_configs(self): # Get a workflow and configs test_outdir = tempfile.mkdtemp() - download_obj = DownloadWorkflow( - pipeline = "dummy", - release = "1.2.0", - outdir = test_outdir - ) - shutil.copytree(PATH_WORKING_EXAMPLE, os.path.join(test_outdir, 'workflow')) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0", outdir=test_outdir) + shutil.copytree(PATH_WORKING_EXAMPLE, os.path.join(test_outdir, "workflow")) download_obj.download_configs() # Test the function download_obj.wf_use_local_configs() - wf_config = nf_core.utils.fetch_wf_config(os.path.join(test_outdir, 'workflow')) - assert wf_config['params.custom_config_base'] == "'../configs/'" + wf_config = nf_core.utils.fetch_wf_config(os.path.join(test_outdir, "workflow")) + assert wf_config["params.custom_config_base"] == "'../configs/'" # # Tests for 'find_container_images' # - @mock.patch('nf_core.utils.fetch_wf_config') + @mock.patch("nf_core.utils.fetch_wf_config") def test_find_container_images(self, mock_fetch_wf_config): - download_obj = DownloadWorkflow( - pipeline = "dummy", - outdir = tempfile.mkdtemp()) + download_obj = DownloadWorkflow(pipeline="dummy", outdir=tempfile.mkdtemp()) mock_fetch_wf_config.return_value = { - 'process.mapping.container': 'cutting-edge-container', - 'process.nocontainer': 'not-so-cutting-edge' + "process.mapping.container": "cutting-edge-container", + "process.nocontainer": "not-so-cutting-edge", } download_obj.find_container_images() assert len(download_obj.containers) == 1 - assert download_obj.containers[0] == 'cutting-edge-container' + assert download_obj.containers[0] == "cutting-edge-container" # # Tests for 'validate_md5' # def test_matching_md5sums(self): - download_obj = DownloadWorkflow(pipeline = "dummy") + download_obj = DownloadWorkflow(pipeline="dummy") test_hash = hashlib.md5() test_hash.update(b"test") val_hash = test_hash.hexdigest() tmpfilehandle, tmpfile = tempfile.mkstemp() - with open(tmpfile[1], "w") as f: f.write("test") + with open(tmpfile[1], "w") as f: + f.write("test") download_obj.validate_md5(tmpfile[1], val_hash) # Clean up os.remove(tmpfile[1]) - @pytest.mark.xfail(raises=IOError) + @pytest.mark.xfail(raises=IOError, strict=True) def test_mismatching_md5sums(self): - download_obj = DownloadWorkflow(pipeline = "dummy") + download_obj = DownloadWorkflow(pipeline="dummy") test_hash = hashlib.md5() test_hash.update(b"other value") val_hash = test_hash.hexdigest() tmpfilehandle, tmpfile = tempfile.mkstemp() - with open(tmpfile, "w") as f: f.write("test") + with open(tmpfile, "w") as f: + f.write("test") download_obj.validate_md5(tmpfile[1], val_hash) @@ -197,12 +169,12 @@ def test_mismatching_md5sums(self): # # Tests for 'pull_singularity_image' # + # If Singularity is not installed, will log an error and exit + # If Singularity is installed, should raise an OSError due to non-existant image @pytest.mark.xfail(raises=OSError) def test_pull_singularity_image(self): tmp_dir = tempfile.mkdtemp() - download_obj = DownloadWorkflow( - pipeline = "dummy", - outdir = tmp_dir) + download_obj = DownloadWorkflow(pipeline="dummy", outdir=tmp_dir) download_obj.pull_singularity_image("a-container") # Clean up @@ -211,16 +183,14 @@ def test_pull_singularity_image(self): # # Tests for the main entry method 'download_workflow' # - @mock.patch('nf_core.download.DownloadWorkflow.pull_singularity_image') - def test_download_workflow_with_success(self, - mock_download_image): + @mock.patch("nf_core.download.DownloadWorkflow.pull_singularity_image") + def test_download_workflow_with_success(self, mock_download_image): tmp_dir = tempfile.mkdtemp() download_obj = DownloadWorkflow( - pipeline = "nf-core/methylseq", - outdir = os.path.join(tmp_dir, 'new'), - singularity = True) + pipeline="nf-core/methylseq", outdir=os.path.join(tmp_dir, "new"), singularity=True + ) download_obj.download_workflow() diff --git a/tests/test_launch.py b/tests/test_launch.py new file mode 100644 index 0000000000..f33e60043a --- /dev/null +++ b/tests/test_launch.py @@ -0,0 +1,331 @@ +#!/usr/bin/env python +""" Tests covering the pipeline launch code. +""" + +import nf_core.launch + +import json +import mock +import os +import shutil +import tempfile +import unittest + + +class TestLaunch(unittest.TestCase): + """Class for launch tests""" + + def setUp(self): + """ Create a new PipelineSchema and Launch objects """ + # Set up the schema + root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + self.template_dir = os.path.join(root_repo_dir, "nf_core", "pipeline-template", "{{cookiecutter.name_noslash}}") + self.nf_params_fn = os.path.join(tempfile.mkdtemp(), "nf-params.json") + self.launcher = nf_core.launch.Launch(self.template_dir, params_out=self.nf_params_fn) + + @mock.patch.object(nf_core.launch.Launch, "prompt_web_gui", side_effect=[True]) + @mock.patch.object(nf_core.launch.Launch, "launch_web_gui") + def test_launch_pipeline(self, mock_webbrowser, mock_lauch_web_gui): + """ Test the main launch function """ + self.launcher.launch_pipeline() + + @mock.patch.object(nf_core.launch.Confirm, "ask", side_effect=[False]) + def test_launch_file_exists(self, mock_confirm): + """ Test that we detect an existing params file and return """ + # Make an empty params file to be overwritten + open(self.nf_params_fn, "a").close() + # Try and to launch, return with error + assert self.launcher.launch_pipeline() is False + + @mock.patch.object(nf_core.launch.Launch, "prompt_web_gui", side_effect=[True]) + @mock.patch.object(nf_core.launch.Launch, "launch_web_gui") + @mock.patch.object(nf_core.launch.Confirm, "ask", side_effect=[False]) + def test_launch_file_exists_overwrite(self, mock_webbrowser, mock_lauch_web_gui, mock_confirm): + """ Test that we detect an existing params file and we overwrite it """ + # Make an empty params file to be overwritten + open(self.nf_params_fn, "a").close() + # Try and to launch, return with error + self.launcher.launch_pipeline() + + def test_get_pipeline_schema(self): + """ Test loading the params schema from a pipeline """ + self.launcher.get_pipeline_schema() + assert len(self.launcher.schema_obj.schema["definitions"]["input_output_options"]["properties"]) > 2 + + def test_make_pipeline_schema(self): + """ Make a copy of the template workflow, but delete the schema file, then try to load it """ + test_pipeline_dir = os.path.join(tempfile.mkdtemp(), "wf") + shutil.copytree(self.template_dir, test_pipeline_dir) + os.remove(os.path.join(test_pipeline_dir, "nextflow_schema.json")) + self.launcher = nf_core.launch.Launch(test_pipeline_dir, params_out=self.nf_params_fn) + self.launcher.get_pipeline_schema() + assert len(self.launcher.schema_obj.schema["definitions"]["input_output_options"]["properties"]) > 2 + assert self.launcher.schema_obj.schema["definitions"]["input_output_options"]["properties"]["outdir"] == { + "type": "string", + "description": "The output directory where the results will be saved.", + "default": "./results", + "fa_icon": "fas fa-folder-open", + } + + def test_get_pipeline_defaults(self): + """ Test fetching default inputs from the pipeline schema """ + self.launcher.get_pipeline_schema() + self.launcher.set_schema_inputs() + assert len(self.launcher.schema_obj.input_params) > 0 + assert self.launcher.schema_obj.input_params["outdir"] == "./results" + + def test_get_pipeline_defaults_input_params(self): + """ Test fetching default inputs from the pipeline schema with an input params file supplied """ + tmp_filehandle, tmp_filename = tempfile.mkstemp() + with os.fdopen(tmp_filehandle, "w") as fh: + json.dump({"outdir": "fubar"}, fh) + self.launcher.params_in = tmp_filename + self.launcher.get_pipeline_schema() + self.launcher.set_schema_inputs() + assert len(self.launcher.schema_obj.input_params) > 0 + assert self.launcher.schema_obj.input_params["outdir"] == "fubar" + + def test_nf_merge_schema(self): + """ Checking merging the nextflow schema with the pipeline schema """ + self.launcher.get_pipeline_schema() + self.launcher.set_schema_inputs() + self.launcher.merge_nxf_flag_schema() + assert self.launcher.schema_obj.schema["allOf"][0] == {"$ref": "#/definitions/coreNextflow"} + assert "-resume" in self.launcher.schema_obj.schema["definitions"]["coreNextflow"]["properties"] + + def test_ob_to_pyinquirer_string(self): + """ Check converting a python dict to a pyenquirer format - simple strings """ + sc_obj = { + "type": "string", + "default": "data/*{1,2}.fastq.gz", + } + result = self.launcher.single_param_to_pyinquirer("input", sc_obj) + assert result == {"type": "input", "name": "input", "message": "input", "default": "data/*{1,2}.fastq.gz"} + + @mock.patch("PyInquirer.prompt", side_effect=[{"use_web_gui": "Web based"}]) + def test_prompt_web_gui_true(self, mock_prompt): + """ Check the prompt to launch the web schema or use the cli """ + assert self.launcher.prompt_web_gui() == True + + @mock.patch("PyInquirer.prompt", side_effect=[{"use_web_gui": "Command line"}]) + def test_prompt_web_gui_false(self, mock_prompt): + """ Check the prompt to launch the web schema or use the cli """ + assert self.launcher.prompt_web_gui() == False + + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{}]) + def test_launch_web_gui_missing_keys(self, mock_poll_nfcore_web_api): + """ Check the code that opens the web browser """ + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + try: + self.launcher.launch_web_gui() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + assert e.args[0].startswith("Web launch response not recognised:") + + @mock.patch( + "nf_core.utils.poll_nfcore_web_api", side_effect=[{"api_url": "foo", "web_url": "bar", "status": "recieved"}] + ) + @mock.patch("webbrowser.open") + @mock.patch("nf_core.utils.wait_cli_function") + def test_launch_web_gui(self, mock_poll_nfcore_web_api, mock_webbrowser, mock_wait_cli_function): + """ Check the code that opens the web browser """ + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + assert self.launcher.launch_web_gui() == None + + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "error", "message": "foo"}]) + def test_get_web_launch_response_error(self, mock_poll_nfcore_web_api): + """ Test polling the website for a launch response - status error """ + try: + self.launcher.get_web_launch_response() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + assert e.args[0] == "Got error from launch API (foo)" + + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "foo"}]) + def test_get_web_launch_response_unexpected(self, mock_poll_nfcore_web_api): + """ Test polling the website for a launch response - status error """ + try: + self.launcher.get_web_launch_response() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + assert e.args[0].startswith("Web launch GUI returned unexpected status (foo): ") + + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "waiting_for_user"}]) + def test_get_web_launch_response_waiting(self, mock_poll_nfcore_web_api): + """ Test polling the website for a launch response - status waiting_for_user""" + assert self.launcher.get_web_launch_response() == False + + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "launch_params_complete"}]) + def test_get_web_launch_response_missing_keys(self, mock_poll_nfcore_web_api): + """ Test polling the website for a launch response - complete, but missing keys """ + try: + self.launcher.get_web_launch_response() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + assert e.args[0] == "Missing return key from web API: 'nxf_flags'" + + @mock.patch( + "nf_core.utils.poll_nfcore_web_api", + side_effect=[ + { + "status": "launch_params_complete", + "nxf_flags": {"resume", "true"}, + "input_params": {"foo", "bar"}, + "schema": {}, + "cli_launch": True, + "nextflow_cmd": "nextflow run foo", + "pipeline": "foo", + "revision": "bar", + } + ], + ) + @mock.patch.object(nf_core.launch.Launch, "sanitise_web_response") + def test_get_web_launch_response_valid(self, mock_poll_nfcore_web_api, mock_sanitise): + """ Test polling the website for a launch response - complete, valid response """ + self.launcher.get_pipeline_schema() + assert self.launcher.get_web_launch_response() == True + + def test_sanitise_web_response(self): + """ Check that we can properly sanitise results from the web """ + self.launcher.get_pipeline_schema() + self.launcher.nxf_flags["-name"] = "" + self.launcher.schema_obj.input_params["single_end"] = "true" + self.launcher.schema_obj.input_params["max_cpus"] = "12" + self.launcher.sanitise_web_response() + assert "-name" not in self.launcher.nxf_flags + assert self.launcher.schema_obj.input_params["single_end"] == True + assert self.launcher.schema_obj.input_params["max_cpus"] == 12 + + def test_ob_to_pyinquirer_bool(self): + """ Check converting a python dict to a pyenquirer format - booleans """ + sc_obj = { + "type": "boolean", + "default": "True", + } + result = self.launcher.single_param_to_pyinquirer("single_end", sc_obj) + assert result["type"] == "list" + assert result["name"] == "single_end" + assert result["message"] == "single_end" + assert result["choices"] == ["True", "False"] + assert result["default"] == "True" + print(type(True)) + assert result["filter"]("True") == True + assert result["filter"]("true") == True + assert result["filter"](True) == True + assert result["filter"]("False") == False + assert result["filter"]("false") == False + assert result["filter"](False) == False + + def test_ob_to_pyinquirer_number(self): + """ Check converting a python dict to a pyenquirer format - with enum """ + sc_obj = {"type": "number", "default": 0.1} + result = self.launcher.single_param_to_pyinquirer("min_reps_consensus", sc_obj) + assert result["type"] == "input" + assert result["default"] == "0.1" + assert result["validate"]("123") is True + assert result["validate"]("-123.56") is True + assert result["validate"]("") is True + assert result["validate"]("123.56.78") == "Must be a number" + assert result["validate"]("123.56sdkfjb") == "Must be a number" + assert result["filter"]("123.456") == float(123.456) + assert result["filter"]("") == "" + + def test_ob_to_pyinquirer_integer(self): + """ Check converting a python dict to a pyenquirer format - with enum """ + sc_obj = {"type": "integer", "default": 1} + result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj) + assert result["type"] == "input" + assert result["default"] == "1" + assert result["validate"]("123") is True + assert result["validate"]("-123") is True + assert result["validate"]("") is True + assert result["validate"]("123.45") == "Must be an integer" + assert result["validate"]("123.56sdkfjb") == "Must be an integer" + assert result["filter"]("123") == int(123) + assert result["filter"]("") == "" + + def test_ob_to_pyinquirer_range(self): + """ Check converting a python dict to a pyenquirer format - with enum """ + sc_obj = {"type": "range", "minimum": "10", "maximum": "20", "default": 15} + result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj) + assert result["type"] == "input" + assert result["default"] == "15" + assert result["validate"]("20") is True + assert result["validate"]("") is True + assert result["validate"]("123.56sdkfjb") == "Must be a number" + assert result["validate"]("8") == "Must be greater than or equal to 10" + assert result["validate"]("25") == "Must be less than or equal to 20" + assert result["filter"]("20") == float(20) + assert result["filter"]("") == "" + + def test_ob_to_pyinquirer_enum(self): + """ Check converting a python dict to a pyenquirer format - with enum """ + sc_obj = {"type": "string", "default": "copy", "enum": ["symlink", "rellink"]} + result = self.launcher.single_param_to_pyinquirer("publish_dir_mode", sc_obj) + assert result["type"] == "list" + assert result["default"] == "copy" + assert result["choices"] == ["symlink", "rellink"] + assert result["validate"]("symlink") is True + assert result["validate"]("") is True + assert result["validate"]("not_allowed") == "Must be one of: symlink, rellink" + + def test_ob_to_pyinquirer_pattern(self): + """ Check converting a python dict to a pyenquirer format - with pattern """ + sc_obj = {"type": "string", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$"} + result = self.launcher.single_param_to_pyinquirer("email", sc_obj) + assert result["type"] == "input" + assert result["validate"]("test@email.com") is True + assert result["validate"]("") is True + assert ( + result["validate"]("not_an_email") + == "Must match pattern: ^([a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})$" + ) + + def test_strip_default_params(self): + """ Test stripping default parameters """ + self.launcher.get_pipeline_schema() + self.launcher.set_schema_inputs() + self.launcher.schema_obj.input_params.update({"input": "custom_input"}) + assert len(self.launcher.schema_obj.input_params) > 1 + self.launcher.strip_default_params() + assert self.launcher.schema_obj.input_params == {"input": "custom_input"} + + def test_build_command_empty(self): + """ Test the functionality to build a nextflow command - nothing customsied """ + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + self.launcher.build_command() + assert self.launcher.nextflow_cmd == "nextflow run {}".format(self.template_dir) + + def test_build_command_nf(self): + """ Test the functionality to build a nextflow command - core nf customised """ + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + self.launcher.nxf_flags["-name"] = "Test_Workflow" + self.launcher.nxf_flags["-resume"] = True + self.launcher.build_command() + assert self.launcher.nextflow_cmd == 'nextflow run {} -name "Test_Workflow" -resume'.format(self.template_dir) + + def test_build_command_params(self): + """ Test the functionality to build a nextflow command - params supplied """ + self.launcher.get_pipeline_schema() + self.launcher.schema_obj.input_params.update({"input": "custom_input"}) + self.launcher.build_command() + # Check command + assert self.launcher.nextflow_cmd == 'nextflow run {} -params-file "{}"'.format( + self.template_dir, os.path.relpath(self.nf_params_fn) + ) + # Check saved parameters file + with open(self.nf_params_fn, "r") as fh: + saved_json = json.load(fh) + assert saved_json == {"input": "custom_input"} + + def test_build_command_params_cl(self): + """ Test the functionality to build a nextflow command - params on Nextflow command line """ + self.launcher.use_params_file = False + self.launcher.get_pipeline_schema() + self.launcher.schema_obj.input_params.update({"input": "custom_input"}) + self.launcher.build_command() + assert self.launcher.nextflow_cmd == 'nextflow run {} --input "custom_input"'.format(self.template_dir) diff --git a/tests/test_licenses.py b/tests/test_licenses.py index 6b148921d1..70e0ea461a 100644 --- a/tests/test_licenses.py +++ b/tests/test_licenses.py @@ -1,12 +1,15 @@ #!/usr/bin/env python """Some tests covering the pipeline creation sub command. """ +import json +import os import pytest -import nf_core.lint, nf_core.licences +import tempfile import unittest +from rich.console import Console - -PL_WITH_LICENSES = 'nf-core/hlatyping' +import nf_core.create +import nf_core.licences class WorkflowLicensesTest(unittest.TestCase): @@ -14,16 +17,41 @@ class WorkflowLicensesTest(unittest.TestCase): retrieval functionality of nf-core tools.""" def setUp(self): - self.license_obj = nf_core.licences.WorkflowLicences( - pipeline=PL_WITH_LICENSES - ) - - def test_fetch_licenses_successful(self): - self.license_obj.fetch_conda_licences() - self.license_obj.print_licences() - - @pytest.mark.xfail(raises=LookupError) - def test_errorness_pipeline_name(self): - self.license_obj.pipeline = 'notpresent' - self.license_obj.fetch_conda_licences() - self.license_obj.print_licences() + """ Create a new pipeline, then make a Licence object """ + # Set up the schema + self.pipeline_dir = os.path.join(tempfile.mkdtemp(), "test_pipeline") + self.create_obj = nf_core.create.PipelineCreate("testing", "test pipeline", "tester", outdir=self.pipeline_dir) + self.create_obj.init_pipeline() + self.license_obj = nf_core.licences.WorkflowLicences(self.pipeline_dir) + + def test_run_licences_successful(self): + console = Console(record=True) + console.print(self.license_obj.run_licences()) + output = console.export_text() + assert "GPLv3" in output + + def test_run_licences_successful_json(self): + self.license_obj.as_json = True + console = Console(record=True) + console.print(self.license_obj.run_licences()) + output = json.loads(console.export_text()) + for package in output: + if "multiqc" in package: + assert output[package][0] == "GPLv3" + break + else: + raise LookupError("Could not find MultiQC") + + def test_get_environment_file_local(self): + self.license_obj.get_environment_file() + assert any(["multiqc" in k for k in self.license_obj.conda_config["dependencies"]]) + + def test_get_environment_file_remote(self): + self.license_obj = nf_core.licences.WorkflowLicences("rnaseq") + self.license_obj.get_environment_file() + assert any(["multiqc" in k for k in self.license_obj.conda_config["dependencies"]]) + + @pytest.mark.xfail(raises=LookupError, strict=True) + def test_get_environment_file_nonexistent(self): + self.license_obj = nf_core.licences.WorkflowLicences("fubarnotreal") + self.license_obj.get_environment_file() diff --git a/tests/test_lint.py b/tests/test_lint.py index 6cbe69a538..707c730e71 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -11,40 +11,48 @@ | |... |--test_lint.py """ +import json +import mock import os -import yaml -import requests import pytest +import requests +import tempfile import unittest -import mock +import yaml + import nf_core.lint def listfiles(path): files_found = [] - for (_,_,files) in os.walk(path): + for (_, _, files) in os.walk(path): files_found.extend(files) return files_found + def pf(wd, path): return os.path.join(wd, path) + WD = os.path.dirname(__file__) -PATH_CRITICAL_EXAMPLE = pf(WD, 'lint_examples/critical_example') -PATH_FAILING_EXAMPLE = pf(WD, 'lint_examples/failing_example') -PATH_WORKING_EXAMPLE = pf(WD, 'lint_examples/minimalworkingexample') -PATH_MISSING_LICENSE_EXAMPLE = pf(WD, 'lint_examples/missing_license_example') -PATHS_WRONG_LICENSE_EXAMPLE = [pf(WD, 'lint_examples/wrong_license_example'), - pf(WD, 'lint_examples/license_incomplete_example')] +PATH_CRITICAL_EXAMPLE = pf(WD, "lint_examples/critical_example") +PATH_FAILING_EXAMPLE = pf(WD, "lint_examples/failing_example") +PATH_WORKING_EXAMPLE = pf(WD, "lint_examples/minimalworkingexample") +PATH_MISSING_LICENSE_EXAMPLE = pf(WD, "lint_examples/missing_license_example") +PATHS_WRONG_LICENSE_EXAMPLE = [ + pf(WD, "lint_examples/wrong_license_example"), + pf(WD, "lint_examples/license_incomplete_example"), +] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 71 +MAX_PASS_CHECKS = 84 # The additional tests passed for releases ADD_PASS_RELEASE = 1 # The minimal working example expects a development release version -if 'dev' not in nf_core.__version__: - nf_core.__version__ = '{}dev'.format(nf_core.__version__) +if "dev" not in nf_core.__version__: + nf_core.__version__ = "{}dev".format(nf_core.__version__) + class TestLint(unittest.TestCase): """Class for lint tests""" @@ -55,17 +63,23 @@ def assess_lint_status(self, lint_obj, **expected): for list_type, expect in expected.items(): observed = len(getattr(lint_obj, list_type)) oberved_list = yaml.safe_dump(getattr(lint_obj, list_type)) - self.assertEqual(observed, expect, "Expected {} tests in '{}', but found {}.\n{}".format(expect, list_type.upper(), observed, oberved_list)) + self.assertEqual( + observed, + expect, + "Expected {} tests in '{}', but found {}.\n{}".format( + expect, list_type.upper(), observed, oberved_list + ), + ) def test_call_lint_pipeline_pass(self): """Test the main execution function of PipelineLint (pass) This should not result in any exception for the minimal working example""" lint_obj = nf_core.lint.run_linting(PATH_WORKING_EXAMPLE, False) - expectations = {"failed": 0, "warned": 4, "passed": MAX_PASS_CHECKS-1} + expectations = {"failed": 0, "warned": 5, "passed": MAX_PASS_CHECKS - 1} self.assess_lint_status(lint_obj, **expectations) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_call_lint_pipeline_fail(self): """Test the main execution function of PipelineLint (fail) This should fail after the first test and halt execution """ @@ -77,7 +91,7 @@ def test_call_lint_pipeline_release(self): """Test the main execution function of PipelineLint when running with --release""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) lint_obj.lint_pipeline(release_mode=True) - expectations = {"failed": 0, "warned": 3, "passed": MAX_PASS_CHECKS + ADD_PASS_RELEASE} + expectations = {"failed": 0, "warned": 4, "passed": MAX_PASS_CHECKS + ADD_PASS_RELEASE} self.assess_lint_status(lint_obj, **expectations) def test_failing_dockerfile_example(self): @@ -86,16 +100,16 @@ def test_failing_dockerfile_example(self): lint_obj.check_docker() self.assess_lint_status(lint_obj, failed=1) - @pytest.mark.xfail(raises=AssertionError) def test_critical_missingfiles_example(self): """Tests for missing nextflow config and main.nf files""" lint_obj = nf_core.lint.run_linting(PATH_CRITICAL_EXAMPLE, False) + assert len(lint_obj.failed) == 1 def test_failing_missingfiles_example(self): """Tests for missing files like Dockerfile or LICENSE""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) lint_obj.check_files_exist() - expectations = {"failed": 5, "warned": 2, "passed": 9} + expectations = {"failed": 6, "warned": 2, "passed": 12} self.assess_lint_status(lint_obj, **expectations) def test_mit_licence_example_pass(self): @@ -116,26 +130,26 @@ def test_config_variable_example_pass(self): """Tests that config variable existence test works with good pipeline example""" good_lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) good_lint_obj.check_nextflow_config() - expectations = {"failed": 0, "warned": 1, "passed": 35} + expectations = {"failed": 0, "warned": 1, "passed": 34} self.assess_lint_status(good_lint_obj, **expectations) def test_config_variable_example_with_failed(self): """Tests that config variable existence test fails with bad pipeline example""" bad_lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) bad_lint_obj.check_nextflow_config() - expectations = {"failed": 18, "warned": 8, "passed": 10} + expectations = {"failed": 19, "warned": 6, "passed": 10} self.assess_lint_status(bad_lint_obj, **expectations) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_config_variable_error(self): """Tests that config variable existence test falls over nicely with nextflow can't run""" - bad_lint_obj = nf_core.lint.PipelineLint('/non/existant/path') + bad_lint_obj = nf_core.lint.PipelineLint("/non/existant/path") bad_lint_obj.check_nextflow_config() def test_actions_wf_branch_pass(self): """Tests that linting for GitHub Actions workflow for branch protection works for a good example""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.pipeline_name = 'tools' + lint_obj.pipeline_name = "tools" lint_obj.check_actions_branch_protection() expectations = {"failed": 0, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -143,7 +157,7 @@ def test_actions_wf_branch_pass(self): def test_actions_wf_branch_fail(self): """Tests that linting for GitHub Actions workflow for branch protection fails for a bad example""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) - lint_obj.pipeline_name = 'tools' + lint_obj.pipeline_name = "tools" lint_obj.check_actions_branch_protection() expectations = {"failed": 2, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) @@ -151,31 +165,31 @@ def test_actions_wf_branch_fail(self): def test_actions_wf_ci_pass(self): """Tests that linting for GitHub Actions CI workflow works for a good example""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.minNextflowVersion = '19.10.0' - lint_obj.pipeline_name = 'tools' - lint_obj.config['process.container'] = "'nfcore/tools:0.4'" + lint_obj.minNextflowVersion = "19.10.0" + lint_obj.pipeline_name = "tools" + lint_obj.config["process.container"] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() - expectations = {"failed": 0, "warned": 0, "passed": 4} + expectations = {"failed": 0, "warned": 0, "passed": 5} self.assess_lint_status(lint_obj, **expectations) def test_actions_wf_ci_fail(self): """Tests that linting for GitHub Actions CI workflow fails for a bad example""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) - lint_obj.minNextflowVersion = '19.10.0' - lint_obj.pipeline_name = 'tools' - lint_obj.config['process.container'] = "'nfcore/tools:0.4'" + lint_obj.minNextflowVersion = "19.10.0" + lint_obj.pipeline_name = "tools" + lint_obj.config["process.container"] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() - expectations = {"failed": 4, "warned": 0, "passed": 0} + expectations = {"failed": 5, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) def test_actions_wf_ci_fail_wrong_NF_version(self): """Tests that linting for GitHub Actions CI workflow fails for a bad NXF version""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.minNextflowVersion = '0.28.0' - lint_obj.pipeline_name = 'tools' - lint_obj.config['process.container'] = "'nfcore/tools:0.4'" + lint_obj.minNextflowVersion = "0.28.0" + lint_obj.pipeline_name = "tools" + lint_obj.config["process.container"] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() - expectations = {"failed": 1, "warned": 0, "passed": 3} + expectations = {"failed": 1, "warned": 0, "passed": 4} self.assess_lint_status(lint_obj, **expectations) def test_actions_wf_lint_pass(self): @@ -192,6 +206,34 @@ def test_actions_wf_lint_fail(self): expectations = {"failed": 3, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) + def test_actions_wf_awstest_pass(self): + """Tests that linting for GitHub Actions AWS test wf works for a good example""" + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.check_actions_awstest() + expectations = {"failed": 0, "warned": 0, "passed": 2} + self.assess_lint_status(lint_obj, **expectations) + + def test_actions_wf_awstest_fail(self): + """Tests that linting for GitHub Actions AWS test wf fails for a bad example""" + lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) + lint_obj.check_actions_awstest() + expectations = {"failed": 2, "warned": 0, "passed": 0} + self.assess_lint_status(lint_obj, **expectations) + + def test_actions_wf_awsfulltest_pass(self): + """Tests that linting for GitHub Actions AWS full test wf works for a good example""" + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.check_actions_awsfulltest() + expectations = {"failed": 0, "warned": 0, "passed": 2} + self.assess_lint_status(lint_obj, **expectations) + + def test_actions_wf_awsfulltest_fail(self): + """Tests that linting for GitHub Actions AWS full test wf fails for a bad example""" + lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) + lint_obj.check_actions_awsfulltest() + expectations = {"failed": 1, "warned": 1, "passed": 0} + self.assess_lint_status(lint_obj, **expectations) + def test_wrong_license_examples_with_failed(self): """Tests for checking the license test behavior""" for example in PATHS_WRONG_LICENSE_EXAMPLE: @@ -210,8 +252,8 @@ def test_missing_license_example(self): def test_readme_pass(self): """Tests that the pipeline README file checks work with a good example""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.minNextflowVersion = '19.10.0' - lint_obj.files = ['environment.yml'] + lint_obj.minNextflowVersion = "19.10.0" + lint_obj.files = ["environment.yml"] lint_obj.check_readme() expectations = {"failed": 0, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -219,7 +261,7 @@ def test_readme_pass(self): def test_readme_warn(self): """Tests that the pipeline README file checks fail """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.minNextflowVersion = '0.28.0' + lint_obj.minNextflowVersion = "0.28.0" lint_obj.check_readme() expectations = {"failed": 1, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) @@ -227,9 +269,9 @@ def test_readme_warn(self): def test_readme_fail(self): """Tests that the pipeline README file checks give warnings with a bad example""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) - lint_obj.files = ['environment.yml'] + lint_obj.files = ["environment.yml"] lint_obj.check_readme() - expectations = {"failed": 1, "warned": 1, "passed": 0} + expectations = {"failed": 0, "warned": 2, "passed": 0} self.assess_lint_status(lint_obj, **expectations) def test_dockerfile_pass(self): @@ -299,38 +341,38 @@ def test_version_consistency_with_env_pass(self): def test_conda_env_pass(self): """ Tests the conda environment config checks with a working example """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - with open(os.path.join(PATH_WORKING_EXAMPLE, 'environment.yml'), 'r') as fh: + lint_obj.files = ["environment.yml"] + with open(os.path.join(PATH_WORKING_EXAMPLE, "environment.yml"), "r") as fh: lint_obj.conda_config = yaml.safe_load(fh) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" lint_obj.check_conda_env_yaml() - expectations = {"failed": 0, "warned": 3, "passed": 4} + expectations = {"failed": 0, "warned": 4, "passed": 5} self.assess_lint_status(lint_obj, **expectations) def test_conda_env_fail(self): """ Tests the conda environment config fails with a bad example """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - with open(os.path.join(PATH_WORKING_EXAMPLE, 'environment.yml'), 'r') as fh: + lint_obj.files = ["environment.yml"] + with open(os.path.join(PATH_WORKING_EXAMPLE, "environment.yml"), "r") as fh: lint_obj.conda_config = yaml.safe_load(fh) - lint_obj.conda_config['dependencies'] = ['fastqc', 'multiqc=0.9', 'notapackaage=0.4'] - lint_obj.pipeline_name = 'not_tools' - lint_obj.config['manifest.version'] = '0.23' + lint_obj.conda_config["dependencies"] = ["fastqc", "multiqc=0.9", "notapackaage=0.4"] + lint_obj.pipeline_name = "not_tools" + lint_obj.config["manifest.version"] = "0.23" lint_obj.check_conda_env_yaml() expectations = {"failed": 3, "warned": 1, "passed": 2} self.assess_lint_status(lint_obj, **expectations) - @mock.patch('requests.get') - @pytest.mark.xfail(raises=ValueError) + @mock.patch("requests.get") + @pytest.mark.xfail(raises=ValueError, strict=True) def test_conda_env_timeout(self, mock_get): """ Tests the conda environment handles API timeouts """ # Define the behaviour of the request get mock mock_get.side_effect = requests.exceptions.Timeout() # Now do the test lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.conda_config['channels'] = ['bioconda'] - lint_obj.check_anaconda_package('multiqc=1.6') + lint_obj.conda_config["channels"] = ["bioconda"] + lint_obj.check_anaconda_package("multiqc=1.6") def test_conda_env_skip(self): """ Tests the conda environment config is skipped when not needed """ @@ -342,10 +384,10 @@ def test_conda_env_skip(self): def test_conda_dockerfile_pass(self): """ Tests the conda Dockerfile test works with a working example """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - with open(os.path.join(PATH_WORKING_EXAMPLE, 'Dockerfile'), 'r') as fh: + lint_obj.files = ["environment.yml"] + with open(os.path.join(PATH_WORKING_EXAMPLE, "Dockerfile"), "r") as fh: lint_obj.dockerfile = fh.read().splitlines() - lint_obj.conda_config['name'] = 'nf-core-tools-0.4' + lint_obj.conda_config["name"] = "nf-core-tools-0.4" lint_obj.check_conda_dockerfile() expectations = {"failed": 0, "warned": 0, "passed": 1} self.assess_lint_status(lint_obj, **expectations) @@ -353,9 +395,9 @@ def test_conda_dockerfile_pass(self): def test_conda_dockerfile_fail(self): """ Tests the conda Dockerfile test fails with a bad example """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.conda_config['name'] = 'nf-core-tools-0.4' - lint_obj.dockerfile = ['fubar'] + lint_obj.files = ["environment.yml"] + lint_obj.conda_config["name"] = "nf-core-tools-0.4" + lint_obj.dockerfile = ["fubar"] lint_obj.check_conda_dockerfile() expectations = {"failed": 5, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) @@ -370,10 +412,10 @@ def test_conda_dockerfile_skip(self): def test_pip_no_version_fail(self): """ Tests the pip dependency version definition is present """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 1, "warned": 0, "passed": 1} self.assess_lint_status(lint_obj, **expectations) @@ -381,15 +423,15 @@ def test_pip_no_version_fail(self): def test_pip_package_not_latest_warn(self): """ Tests the pip dependency version definition is present """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc==1.4']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc==1.4"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 0, "warned": 1, "passed": 2} self.assess_lint_status(lint_obj, **expectations) - @mock.patch('requests.get') + @mock.patch("requests.get") def test_pypi_timeout_warn(self, mock_get): """ Tests the PyPi connection and simulates a request timeout, which should return in an addiional warning in the linting """ @@ -397,15 +439,15 @@ def test_pypi_timeout_warn(self, mock_get): mock_get.side_effect = requests.exceptions.Timeout() # Now do the test lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc==1.5']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc==1.5"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 0, "warned": 1, "passed": 2} self.assess_lint_status(lint_obj, **expectations) - @mock.patch('requests.get') + @mock.patch("requests.get") def test_pypi_connection_error_warn(self, mock_get): """ Tests the PyPi connection and simulates a connection error, which should result in an additional warning, as we cannot test if dependent module is latest """ @@ -413,10 +455,10 @@ def test_pypi_connection_error_warn(self, mock_get): mock_get.side_effect = requests.exceptions.ConnectionError() # Now do the test lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc==1.5']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc==1.5"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 0, "warned": 1, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -424,10 +466,10 @@ def test_pypi_connection_error_warn(self, mock_get): def test_pip_dependency_fail(self): """ Tests the PyPi API package information query """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['notpresent==1.5']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["notpresent==1.5"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 1, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -437,10 +479,10 @@ def test_conda_dependency_fails(self): package version is not available on Anaconda. """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': ['openjdk=0.0.0']} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": ["openjdk=0.0.0"]} lint_obj.check_conda_env_yaml() expectations = {"failed": 1, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -450,19 +492,19 @@ def test_pip_dependency_fails(self): package version is not available on Anaconda. """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc==0.0']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc==0.0"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 1, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) def test_pipeline_name_pass(self): """Tests pipeline name good pipeline example: lower case, no punctuation""" - #good_lint_obj = nf_core.lint.run_linting(PATH_WORKING_EXAMPLE) + # good_lint_obj = nf_core.lint.run_linting(PATH_WORKING_EXAMPLE) good_lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - good_lint_obj.pipeline_name = 'tools' + good_lint_obj.pipeline_name = "tools" good_lint_obj.check_pipeline_name() expectations = {"failed": 0, "warned": 0, "passed": 1} self.assess_lint_status(good_lint_obj, **expectations) @@ -470,7 +512,101 @@ def test_pipeline_name_pass(self): def test_pipeline_name_critical(self): """Tests that warning is returned for pipeline not adhering to naming convention""" critical_lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - critical_lint_obj.pipeline_name = 'Tools123' + critical_lint_obj.pipeline_name = "Tools123" critical_lint_obj.check_pipeline_name() - expectations = {"failed": 0, "warned": 2, "passed": 0} + expectations = {"failed": 0, "warned": 1, "passed": 0} self.assess_lint_status(critical_lint_obj, **expectations) + + def test_json_output(self): + """ + Test creation of a JSON file with lint results + + Expected JSON output: + { + "nf_core_tools_version": "1.10.dev0", + "date_run": "2020-06-05 10:56:42", + "tests_pass": [ + [ 1, "This test passed"], + [ 2, "This test also passed"] + ], + "tests_warned": [ + [ 2, "This test gave a warning"] + ], + "tests_failed": [], + "num_tests_pass": 2, + "num_tests_warned": 1, + "num_tests_failed": 0, + "has_tests_pass": true, + "has_tests_warned": true, + "has_tests_failed": false + } + """ + # Don't run testing, just fake some testing results + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.passed.append((1, "This test passed")) + lint_obj.passed.append((2, "This test also passed")) + lint_obj.warned.append((2, "This test gave a warning")) + tmpdir = tempfile.mkdtemp() + json_fn = os.path.join(tmpdir, "lint_results.json") + lint_obj.save_json_results(json_fn) + with open(json_fn, "r") as fh: + saved_json = json.load(fh) + assert saved_json["num_tests_pass"] == 2 + assert saved_json["num_tests_warned"] == 1 + assert saved_json["num_tests_failed"] == 0 + assert saved_json["has_tests_pass"] + assert saved_json["has_tests_warned"] + assert not saved_json["has_tests_failed"] + + def mock_gh_get_comments(**kwargs): + """ Helper function to emulate requests responses from the web """ + + class MockResponse: + def __init__(self, url): + self.status_code = 200 + self.url = url + + def json(self): + if self.url == "existing_comment": + return [ + { + "user": {"login": "github-actions[bot]"}, + "body": "\n#### `nf-core lint` overall result", + "url": "https://github.com", + } + ] + else: + return [] + + return MockResponse(kwargs["url"]) + + @mock.patch("requests.get", side_effect=mock_gh_get_comments) + @mock.patch("requests.post") + def test_gh_comment_post(self, mock_get, mock_post): + """ + Test updating a Github comment with the lint results + """ + os.environ["GITHUB_COMMENTS_URL"] = "https://github.com" + os.environ["GITHUB_TOKEN"] = "testing" + os.environ["GITHUB_PR_COMMIT"] = "abcdefg" + # Don't run testing, just fake some testing results + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.failed.append((1, "This test failed")) + lint_obj.passed.append((2, "This test also passed")) + lint_obj.warned.append((2, "This test gave a warning")) + lint_obj.github_comment() + + @mock.patch("requests.get", side_effect=mock_gh_get_comments) + @mock.patch("requests.post") + def test_gh_comment_update(self, mock_get, mock_post): + """ + Test updating a Github comment with the lint results + """ + os.environ["GITHUB_COMMENTS_URL"] = "existing_comment" + os.environ["GITHUB_TOKEN"] = "testing" + # Don't run testing, just fake some testing results + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.failed.append((1, "This test failed")) + lint_obj.passed.append((2, "This test also passed")) + lint_obj.warned.append((2, "This test gave a warning")) + lint_obj.github_comment() diff --git a/tests/test_list.py b/tests/test_list.py index f54816d8e8..1b7920475d 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -4,31 +4,49 @@ import nf_core.list +import json import mock import os -import git import pytest import time import unittest +from rich.console import Console from datetime import datetime + class TestLint(unittest.TestCase): """Class for list tests""" - @mock.patch('json.dumps') - @mock.patch('subprocess.check_output') - @mock.patch('nf_core.list.LocalWorkflow') - def test_working_listcall(self, mock_loc_wf, mock_subprocess, mock_json): + @mock.patch("subprocess.check_output") + def test_working_listcall(self, mock_subprocess): """ Test that listing pipelines works """ - nf_core.list.list_workflows() - - @mock.patch('json.dumps') - @mock.patch('subprocess.check_output') - @mock.patch('nf_core.list.LocalWorkflow') - def test_working_listcall_json(self, mock_loc_wf, mock_subprocess, mock_json): + wf_table = nf_core.list.list_workflows() + console = Console(record=True) + console.print(wf_table) + output = console.export_text() + assert "rnaseq" in output + assert "exoseq" not in output + + @mock.patch("subprocess.check_output") + def test_working_listcall_archived(self, mock_subprocess): + """ Test that listing pipelines works, showing archived pipelines """ + wf_table = nf_core.list.list_workflows(show_archived=True) + console = Console(record=True) + console.print(wf_table) + output = console.export_text() + assert "exoseq" in output + + @mock.patch("subprocess.check_output") + def test_working_listcall_json(self, mock_subprocess): """ Test that listing pipelines with JSON works """ - nf_core.list.list_workflows([], as_json=True) + wf_json_str = nf_core.list.list_workflows(as_json=True) + wf_json = json.loads(wf_json_str) + for wf in wf_json["remote_workflows"]: + if wf["name"] == "ampliseq": + break + else: + raise AssertionError("Could not find ampliseq in JSON") def test_pretty_datetime(self): """ Test that the pretty datetime function works """ @@ -37,7 +55,7 @@ def test_pretty_datetime(self): now_ts = time.mktime(now.timetuple()) nf_core.list.pretty_date(now_ts) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_local_workflows_and_fail(self): """ Test the local workflow class and try to get local Nextflow workflow information """ @@ -49,24 +67,23 @@ def test_local_workflows_compare_and_fail_silently(self): and remote workflows """ wfs = nf_core.list.Workflows() lwf_ex = nf_core.list.LocalWorkflow("myWF") - lwf_ex.full_name = 'my Workflow' + lwf_ex.full_name = "my Workflow" lwf_ex.commit_sha = "aw3s0meh1sh" remote = { - 'name': 'myWF', - 'full_name': 'my Workflow', - 'description': '...', - 'archived': [], - 'stargazers_count': 42, - 'watchers_count': 6, - 'forks_count': 7, - 'releases': [] + "name": "myWF", + "full_name": "my Workflow", + "description": "...", + "archived": False, + "stargazers_count": 42, + "watchers_count": 6, + "forks_count": 7, + "releases": [], } rwf_ex = nf_core.list.RemoteWorkflow(remote) rwf_ex.commit_sha = "aw3s0meh1sh" - rwf_ex.releases = [{'tag_sha': "aw3s0meh1sh"}] - + rwf_ex.releases = [{"tag_sha": "aw3s0meh1sh"}] wfs.local_workflows.append(lwf_ex) wfs.remote_workflows.append(rwf_ex) @@ -75,7 +92,7 @@ def test_local_workflows_compare_and_fail_silently(self): self.assertEqual(rwf_ex.local_wf, lwf_ex) rwf_ex.releases = [] - rwf_ex.releases.append({'tag_sha': "noaw3s0meh1sh"}) + rwf_ex.releases.append({"tag_sha": "noaw3s0meh1sh"}) wfs.compare_remote_local() rwf_ex.full_name = "your Workflow" @@ -83,79 +100,111 @@ def test_local_workflows_compare_and_fail_silently(self): rwf_ex.releases = None - @mock.patch('nf_core.list.LocalWorkflow') + @mock.patch.dict(os.environ, {"NXF_ASSETS": "/tmp/nxf"}) + @mock.patch("nf_core.list.LocalWorkflow") def test_parse_local_workflow_and_succeed(self, mock_local_wf): - test_path = '/tmp/nxf/nf-core' - if not os.path.isdir(test_path): os.makedirs(test_path) - - if not os.environ.get('NXF_ASSETS'): - os.environ['NXF_ASSETS'] = '/tmp/nxf' - assert os.environ['NXF_ASSETS'] == '/tmp/nxf' - with open('/tmp/nxf/nf-core/dummy-wf', 'w') as f: - f.write('dummy') + test_path = "/tmp/nxf/nf-core" + if not os.path.isdir(test_path): + os.makedirs(test_path) + assert os.environ["NXF_ASSETS"] == "/tmp/nxf" + with open("/tmp/nxf/nf-core/dummy-wf", "w") as f: + f.write("dummy") workflows_obj = nf_core.list.Workflows() workflows_obj.get_local_nf_workflows() assert len(workflows_obj.local_workflows) == 1 - @mock.patch('os.environ.get') - @mock.patch('nf_core.list.LocalWorkflow') - @mock.patch('subprocess.check_output') - def test_parse_local_workflow_home(self, mock_subprocess, mock_local_wf, mock_env): - test_path = '/tmp/nxf/nf-core' - if not os.path.isdir(test_path): os.makedirs(test_path) - - mock_env.side_effect = '/tmp/nxf' - - assert os.environ['NXF_ASSETS'] == '/tmp/nxf' - with open('/tmp/nxf/nf-core/dummy-wf', 'w') as f: - f.write('dummy') + @mock.patch.dict(os.environ, {"NXF_ASSETS": "/tmp/nxf"}) + @mock.patch("nf_core.list.LocalWorkflow") + @mock.patch("subprocess.check_output") + def test_parse_local_workflow_home(self, mock_local_wf, mock_subprocess): + test_path = "/tmp/nxf/nf-core" + if not os.path.isdir(test_path): + os.makedirs(test_path) + assert os.environ["NXF_ASSETS"] == "/tmp/nxf" + with open("/tmp/nxf/nf-core/dummy-wf", "w") as f: + f.write("dummy") workflows_obj = nf_core.list.Workflows() workflows_obj.get_local_nf_workflows() - @mock.patch('os.stat') - @mock.patch('git.Repo') + @mock.patch("os.stat") + @mock.patch("git.Repo") def test_local_workflow_investigation(self, mock_repo, mock_stat): - local_wf = nf_core.list.LocalWorkflow('dummy') - local_wf.local_path = '/tmp' - mock_repo.head.commit.hexsha = 'h00r4y' + local_wf = nf_core.list.LocalWorkflow("dummy") + local_wf.local_path = "/tmp" + mock_repo.head.commit.hexsha = "h00r4y" mock_stat.st_mode = 1 local_wf.get_local_nf_workflow_details() - def test_worflow_filter(self): workflows_obj = nf_core.list.Workflows(["rna", "myWF"]) remote = { - 'name': 'myWF', - 'full_name': 'my Workflow', - 'description': 'rna', - 'archived': [], - 'stargazers_count': 42, - 'watchers_count': 6, - 'forks_count': 7, - 'releases': [] + "name": "myWF", + "full_name": "my Workflow", + "description": "rna", + "archived": False, + "stargazers_count": 42, + "watchers_count": 6, + "forks_count": 7, + "releases": [], } rwf_ex = nf_core.list.RemoteWorkflow(remote) rwf_ex.commit_sha = "aw3s0meh1sh" - rwf_ex.releases = [{'tag_sha': "aw3s0meh1sh"}] + rwf_ex.releases = [{"tag_sha": "aw3s0meh1sh"}] remote2 = { - 'name': 'myWF', - 'full_name': 'my Workflow', - 'description': 'dna', - 'archived': [], - 'stargazers_count': 42, - 'watchers_count': 6, - 'forks_count': 7, - 'releases': [] + "name": "myWF", + "full_name": "my Workflow", + "description": "dna", + "archived": False, + "stargazers_count": 42, + "watchers_count": 6, + "forks_count": 7, + "releases": [], } rwf_ex2 = nf_core.list.RemoteWorkflow(remote2) rwf_ex2.commit_sha = "aw3s0meh1sh" - rwf_ex2.releases = [{'tag_sha': "aw3s0meh1sh"}] + rwf_ex2.releases = [{"tag_sha": "aw3s0meh1sh"}] workflows_obj.remote_workflows.append(rwf_ex) workflows_obj.remote_workflows.append(rwf_ex2) assert len(workflows_obj.filtered_workflows()) == 1 + + def test_filter_archived_workflows(self): + """ + Test that archived workflows are not shown by default + """ + workflows_obj = nf_core.list.Workflows() + remote1 = {"name": "myWF", "full_name": "my Workflow", "archived": True, "releases": []} + rwf_ex1 = nf_core.list.RemoteWorkflow(remote1) + remote2 = {"name": "myWF", "full_name": "my Workflow", "archived": False, "releases": []} + rwf_ex2 = nf_core.list.RemoteWorkflow(remote2) + + workflows_obj.remote_workflows.append(rwf_ex1) + workflows_obj.remote_workflows.append(rwf_ex2) + + filtered_workflows = workflows_obj.filtered_workflows() + expected_workflows = [rwf_ex2] + + assert filtered_workflows == expected_workflows + + def test_show_archived_workflows(self): + """ + Test that archived workflows can be shown optionally + """ + workflows_obj = nf_core.list.Workflows(show_archived=True) + remote1 = {"name": "myWF", "full_name": "my Workflow", "archived": True, "releases": []} + rwf_ex1 = nf_core.list.RemoteWorkflow(remote1) + remote2 = {"name": "myWF", "full_name": "my Workflow", "archived": False, "releases": []} + rwf_ex2 = nf_core.list.RemoteWorkflow(remote2) + + workflows_obj.remote_workflows.append(rwf_ex1) + workflows_obj.remote_workflows.append(rwf_ex2) + + filtered_workflows = workflows_obj.filtered_workflows() + expected_workflows = [rwf_ex1, rwf_ex2] + + assert filtered_workflows == expected_workflows diff --git a/tests/test_modules.py b/tests/test_modules.py new file mode 100644 index 0000000000..7643c70fc5 --- /dev/null +++ b/tests/test_modules.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python +""" Tests covering the modules commands +""" + +import nf_core.modules + +import mock +import os +import shutil +import tempfile +import unittest + + +class TestModules(unittest.TestCase): + """Class for modules tests""" + + def setUp(self): + """ Create a new PipelineSchema and Launch objects """ + # Set up the schema + root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + self.template_dir = os.path.join(root_repo_dir, "nf_core", "pipeline-template", "{{cookiecutter.name_noslash}}") + self.pipeline_dir = os.path.join(tempfile.mkdtemp(), "mypipeline") + shutil.copytree(self.template_dir, self.pipeline_dir) + self.mods = nf_core.modules.PipelineModules() + self.mods.pipeline_dir = self.pipeline_dir + + def test_modulesrepo_class(self): + """ Initialise a modules repo object """ + modrepo = nf_core.modules.ModulesRepo() + assert modrepo.name == "nf-core/modules" + assert modrepo.branch == "master" + + def test_modules_list(self): + """ Test listing available modules """ + self.mods.pipeline_dir = None + listed_mods = self.mods.list_modules() + assert "fastqc" in listed_mods + + def test_modules_install_nopipeline(self): + """ Test installing a module - no pipeline given """ + self.mods.pipeline_dir = None + assert self.mods.install("foo") is False + + def test_modules_install_emptypipeline(self): + """ Test installing a module - empty dir given """ + self.mods.pipeline_dir = tempfile.mkdtemp() + assert self.mods.install("foo") is False + + def test_modules_install_nomodule(self): + """ Test installing a module - unrecognised module given """ + assert self.mods.install("foo") is False + + def test_modules_install_fastqc(self): + """ Test installing a module - FastQC """ + assert self.mods.install("fastqc") is not False + + def test_modules_install_fastqc_twice(self): + """ Test installing a module - FastQC already there """ + self.mods.install("fastqc") + assert self.mods.install("fastqc") is False diff --git a/tests/test_schema.py b/tests/test_schema.py new file mode 100644 index 0000000000..a53d946c09 --- /dev/null +++ b/tests/test_schema.py @@ -0,0 +1,451 @@ +#!/usr/bin/env python +""" Tests covering the pipeline schema code. +""" + +import nf_core.schema + +import click +import json +import mock +import os +import pytest +import requests +import shutil +import tempfile +import unittest +import yaml + + +class TestSchema(unittest.TestCase): + """Class for schema tests""" + + def setUp(self): + """ Create a new PipelineSchema object """ + self.schema_obj = nf_core.schema.PipelineSchema() + self.root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + # Copy the template to a temp directory so that we can use that for tests + self.template_dir = os.path.join(tempfile.mkdtemp(), "wf") + template_dir = os.path.join(self.root_repo_dir, "nf_core", "pipeline-template", "{{cookiecutter.name_noslash}}") + shutil.copytree(template_dir, self.template_dir) + self.template_schema = os.path.join(self.template_dir, "nextflow_schema.json") + + def test_load_lint_schema(self): + """ Check linting with the pipeline template directory """ + self.schema_obj.get_schema_path(self.template_dir) + self.schema_obj.load_lint_schema() + + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_load_lint_schema_nofile(self): + """ Check that linting raises properly if a non-existant file is given """ + self.schema_obj.get_schema_path("fake_file") + self.schema_obj.load_lint_schema() + + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_load_lint_schema_notjson(self): + """ Check that linting raises properly if a non-JSON file is given """ + self.schema_obj.get_schema_path(os.path.join(self.template_dir, "nextflow.config")) + self.schema_obj.load_lint_schema() + + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_load_lint_schema_noparams(self): + """ + Check that linting raises properly if a JSON file is given without any params + """ + # Make a temporary file to write schema to + tmp_file = tempfile.NamedTemporaryFile() + with open(tmp_file.name, "w") as fh: + json.dump({"type": "fubar"}, fh) + self.schema_obj.get_schema_path(tmp_file.name) + self.schema_obj.load_lint_schema() + + def test_get_schema_path_dir(self): + """ Get schema file from directory """ + self.schema_obj.get_schema_path(self.template_dir) + + def test_get_schema_path_path(self): + """ Get schema file from a path """ + self.schema_obj.get_schema_path(self.template_schema) + + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_get_schema_path_path_notexist(self): + """ Get schema file from a path """ + self.schema_obj.get_schema_path("fubar", local_only=True) + + def test_get_schema_path_name(self): + """ Get schema file from the name of a remote pipeline """ + self.schema_obj.get_schema_path("atacseq") + + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_get_schema_path_name_notexist(self): + """ + Get schema file from the name of a remote pipeline + that doesn't have a schema file + """ + self.schema_obj.get_schema_path("exoseq") + + def test_load_schema(self): + """ Try to load a schema from a file """ + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + + def test_save_schema(self): + """ Try to save a schema """ + # Load the template schema + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + + # Make a temporary file to write schema to + tmp_file = tempfile.NamedTemporaryFile() + self.schema_obj.schema_filename = tmp_file.name + self.schema_obj.save_schema() + + def test_load_input_params_json(self): + """ Try to load a JSON file with params for a pipeline run """ + # Make a temporary file to write schema to + tmp_file = tempfile.NamedTemporaryFile() + with open(tmp_file.name, "w") as fh: + json.dump({"input": "fubar"}, fh) + self.schema_obj.load_input_params(tmp_file.name) + + def test_load_input_params_yaml(self): + """ Try to load a YAML file with params for a pipeline run """ + # Make a temporary file to write schema to + tmp_file = tempfile.NamedTemporaryFile() + with open(tmp_file.name, "w") as fh: + yaml.dump({"input": "fubar"}, fh) + self.schema_obj.load_input_params(tmp_file.name) + + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_load_input_params_invalid(self): + """ Check failure when a non-existent file params file is loaded """ + self.schema_obj.load_input_params("fubar") + + def test_validate_params_pass(self): + """ Try validating a set of parameters against a schema """ + # Load the template schema + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + self.schema_obj.input_params = {"input": "fubar"} + assert self.schema_obj.validate_params() + + def test_validate_params_fail(self): + """ Check that False is returned if params don't validate against a schema """ + # Load the template schema + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + self.schema_obj.input_params = {"fubar": "input"} + assert not self.schema_obj.validate_params() + + def test_validate_schema_pass(self): + """ Check that the schema validation passes """ + # Load the template schema + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + self.schema_obj.validate_schema(self.schema_obj.schema) + + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_validate_schema_fail_noparams(self): + """ Check that the schema validation fails when no params described """ + self.schema_obj.schema = {"type": "invalidthing"} + self.schema_obj.validate_schema(self.schema_obj.schema) + + def test_validate_schema_fail_duplicate_ids(self): + """ + Check that the schema validation fails when we have duplicate IDs in definition subschema + """ + self.schema_obj.schema = { + "definitions": {"groupOne": {"properties": {"foo": {}}}, "groupTwo": {"properties": {"foo": {}}}}, + "allOf": [{"$ref": "#/definitions/groupOne"}, {"$ref": "#/definitions/groupTwo"}], + } + try: + self.schema_obj.validate_schema(self.schema_obj.schema) + raise UserWarning("Expected AssertionError") + except AssertionError as e: + assert e.args[0] == "Duplicate parameter found in schema `definitions`: `foo`" + + def test_validate_schema_fail_missing_def(self): + """ + Check that the schema validation fails when we a definition in allOf is not in definitions + """ + self.schema_obj.schema = { + "definitions": {"groupOne": {"properties": {"foo": {}}}, "groupTwo": {"properties": {"bar": {}}}}, + "allOf": [{"$ref": "#/definitions/groupOne"}], + } + try: + self.schema_obj.validate_schema(self.schema_obj.schema) + raise UserWarning("Expected AssertionError") + except AssertionError as e: + assert e.args[0] == "Definition subschema `groupTwo` not included in schema `allOf`" + + def test_validate_schema_fail_unexpected_allof(self): + """ + Check that the schema validation fails when we an unrecognised definition is in allOf + """ + self.schema_obj.schema = { + "definitions": {"groupOne": {"properties": {"foo": {}}}, "groupTwo": {"properties": {"bar": {}}}}, + "allOf": [ + {"$ref": "#/definitions/groupOne"}, + {"$ref": "#/definitions/groupTwo"}, + {"$ref": "#/definitions/groupThree"}, + ], + } + try: + self.schema_obj.validate_schema(self.schema_obj.schema) + raise UserWarning("Expected AssertionError") + except AssertionError as e: + assert e.args[0] == "Subschema `groupThree` found in `allOf` but not `definitions`" + + def test_make_skeleton_schema(self): + """ Test making a new schema skeleton """ + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.pipeline_manifest["name"] = "nf-core/test" + self.schema_obj.pipeline_manifest["description"] = "Test pipeline" + self.schema_obj.make_skeleton_schema() + self.schema_obj.validate_schema(self.schema_obj.schema) + + def test_get_wf_params(self): + """ Test getting the workflow parameters from a pipeline """ + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.get_wf_params() + + def test_prompt_remove_schema_notfound_config_returntrue(self): + """ Remove unrecognised params from the schema """ + self.schema_obj.pipeline_params = {"foo": "bar"} + self.schema_obj.no_prompts = True + assert self.schema_obj.prompt_remove_schema_notfound_config("baz") + + def test_prompt_remove_schema_notfound_config_returnfalse(self): + """ Do not remove unrecognised params from the schema """ + self.schema_obj.pipeline_params = {"foo": "bar"} + self.schema_obj.no_prompts = True + assert not self.schema_obj.prompt_remove_schema_notfound_config("foo") + + def test_remove_schema_notfound_configs(self): + """ Remove unrecognised params from the schema """ + self.schema_obj.schema = { + "properties": {"foo": {"type": "string"}, "bar": {"type": "string"}}, + "required": ["foo"], + } + self.schema_obj.pipeline_params = {"bar": True} + self.schema_obj.no_prompts = True + params_removed = self.schema_obj.remove_schema_notfound_configs() + assert len(self.schema_obj.schema["properties"]) == 1 + assert "required" not in self.schema_obj.schema + assert len(params_removed) == 1 + assert "foo" in params_removed + + def test_remove_schema_notfound_configs_childschema(self): + """ + Remove unrecognised params from the schema, + even when they're in a group + """ + self.schema_obj.schema = { + "definitions": { + "subSchemaId": { + "properties": {"foo": {"type": "string"}, "bar": {"type": "string"}}, + "required": ["foo"], + } + } + } + self.schema_obj.pipeline_params = {"bar": True} + self.schema_obj.no_prompts = True + params_removed = self.schema_obj.remove_schema_notfound_configs() + assert len(self.schema_obj.schema["definitions"]["subSchemaId"]["properties"]) == 1 + assert "required" not in self.schema_obj.schema["definitions"]["subSchemaId"] + assert len(params_removed) == 1 + assert "foo" in params_removed + + def test_add_schema_found_configs(self): + """ Try adding a new parameter to the schema from the config """ + self.schema_obj.pipeline_params = {"foo": "bar"} + self.schema_obj.schema = {"properties": {}} + self.schema_obj.no_prompts = True + params_added = self.schema_obj.add_schema_found_configs() + assert len(self.schema_obj.schema["properties"]) == 1 + assert len(params_added) == 1 + assert "foo" in params_added + + def test_build_schema_param_str(self): + """ Build a new schema param from a config value (string) """ + param = self.schema_obj.build_schema_param("foo") + assert param == {"type": "string", "default": "foo"} + + def test_build_schema_param_bool(self): + """ Build a new schema param from a config value (bool) """ + param = self.schema_obj.build_schema_param("True") + print(param) + assert param == {"type": "boolean", "default": True} + + def test_build_schema_param_int(self): + """ Build a new schema param from a config value (int) """ + param = self.schema_obj.build_schema_param("12") + assert param == {"type": "integer", "default": 12} + + def test_build_schema_param_int(self): + """ Build a new schema param from a config value (float) """ + param = self.schema_obj.build_schema_param("12.34") + assert param == {"type": "number", "default": 12.34} + + def test_build_schema(self): + """ + Build a new schema param from a pipeline + Run code to ensure it doesn't crash. Individual functions tested separately. + """ + param = self.schema_obj.build_schema(self.template_dir, True, False, None) + + def test_build_schema_from_scratch(self): + """ + Build a new schema param from a pipeline with no existing file + Run code to ensure it doesn't crash. Individual functions tested separately. + + Pretty much a copy of test_launch.py test_make_pipeline_schema + """ + test_pipeline_dir = os.path.join(tempfile.mkdtemp(), "wf") + shutil.copytree(self.template_dir, test_pipeline_dir) + os.remove(os.path.join(test_pipeline_dir, "nextflow_schema.json")) + + param = self.schema_obj.build_schema(test_pipeline_dir, True, False, None) + + @pytest.mark.xfail(raises=AssertionError, strict=True) + @mock.patch("requests.post") + def test_launch_web_builder_timeout(self, mock_post): + """ Mock launching the web builder, but timeout on the request """ + # Define the behaviour of the request get mock + mock_post.side_effect = requests.exceptions.Timeout() + self.schema_obj.launch_web_builder() + + @pytest.mark.xfail(raises=AssertionError, strict=True) + @mock.patch("requests.post") + def test_launch_web_builder_connection_error(self, mock_post): + """ Mock launching the web builder, but get a connection error """ + # Define the behaviour of the request get mock + mock_post.side_effect = requests.exceptions.ConnectionError() + self.schema_obj.launch_web_builder() + + @pytest.mark.xfail(raises=AssertionError, strict=True) + @mock.patch("requests.post") + def test_get_web_builder_response_timeout(self, mock_post): + """ Mock checking for a web builder response, but timeout on the request """ + # Define the behaviour of the request get mock + mock_post.side_effect = requests.exceptions.Timeout() + self.schema_obj.launch_web_builder() + + @pytest.mark.xfail(raises=AssertionError, strict=True) + @mock.patch("requests.post") + def test_get_web_builder_response_connection_error(self, mock_post): + """ Mock checking for a web builder response, but get a connection error """ + # Define the behaviour of the request get mock + mock_post.side_effect = requests.exceptions.ConnectionError() + self.schema_obj.launch_web_builder() + + def mocked_requests_post(**kwargs): + """ Helper function to emulate POST requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = status_code + self.content = json.dumps(data) + + if kwargs["url"] == "invalid_url": + return MockResponse({}, 404) + + if kwargs["url"] == "valid_url_error": + response_data = {"status": "error", "api_url": "foo", "web_url": "bar"} + return MockResponse(response_data, 200) + + if kwargs["url"] == "valid_url_success": + response_data = {"status": "recieved", "api_url": "https://nf-co.re", "web_url": "https://nf-co.re"} + return MockResponse(response_data, 200) + + @mock.patch("requests.post", side_effect=mocked_requests_post) + def test_launch_web_builder_404(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_url = "invalid_url" + try: + self.schema_obj.launch_web_builder() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + assert e.args[0] == "Could not access remote API results: invalid_url (HTML 404 Error)" + + @mock.patch("requests.post", side_effect=mocked_requests_post) + def test_launch_web_builder_invalid_status(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_url = "valid_url_error" + try: + self.schema_obj.launch_web_builder() + except AssertionError as e: + assert e.args[0].startswith("Pipeline schema builder response not recognised") + + @mock.patch("requests.post", side_effect=mocked_requests_post) + @mock.patch("requests.get") + @mock.patch("webbrowser.open") + def test_launch_web_builder_success(self, mock_post, mock_get, mock_webbrowser): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_url = "valid_url_success" + try: + self.schema_obj.launch_web_builder() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + # Assertion error comes from get_web_builder_response() function + assert e.args[0].startswith("Could not access remote API results: https://nf-co.re") + + def mocked_requests_get(*args, **kwargs): + """ Helper function to emulate GET requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = status_code + self.content = json.dumps(data) + + if args[0] == "invalid_url": + return MockResponse({}, 404) + + if args[0] == "valid_url_error": + response_data = {"status": "error", "message": "testing URL failure"} + return MockResponse(response_data, 200) + + if args[0] == "valid_url_waiting": + response_data = {"status": "waiting_for_user", "message": "testing URL waiting"} + return MockResponse(response_data, 200) + + if args[0] == "valid_url_saved": + response_data = {"status": "web_builder_edited", "message": "testing saved", "schema": {"foo": "bar"}} + return MockResponse(response_data, 200) + + @mock.patch("requests.get", side_effect=mocked_requests_get) + def test_get_web_builder_response_404(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_api_url = "invalid_url" + try: + self.schema_obj.get_web_builder_response() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + assert e.args[0] == "Could not access remote API results: invalid_url (HTML 404 Error)" + + @mock.patch("requests.get", side_effect=mocked_requests_get) + def test_get_web_builder_response_error(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_api_url = "valid_url_error" + try: + self.schema_obj.get_web_builder_response() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + assert e.args[0] == "Got error from schema builder: 'testing URL failure'" + + @mock.patch("requests.get", side_effect=mocked_requests_get) + def test_get_web_builder_response_waiting(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_api_url = "valid_url_waiting" + assert self.schema_obj.get_web_builder_response() is False + + @mock.patch("requests.get", side_effect=mocked_requests_get) + def test_get_web_builder_response_saved(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_api_url = "valid_url_saved" + try: + self.schema_obj.get_web_builder_response() + raise UserWarning("Should have hit an AssertionError") + except AssertionError as e: + # Check that this is the expected AssertionError, as there are several + assert e.args[0].startswith("Response from schema builder did not pass validation") + assert self.schema_obj.schema == {"foo": "bar"} diff --git a/tests/test_sync.py b/tests/test_sync.py new file mode 100644 index 0000000000..01b56ac358 --- /dev/null +++ b/tests/test_sync.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python +""" Tests covering the sync command +""" + +import nf_core.create +import nf_core.sync + +import json +import mock +import os +import shutil +import tempfile +import unittest + + +class TestModules(unittest.TestCase): + """Class for modules tests""" + + def setUp(self): + self.make_new_pipeline() + + def make_new_pipeline(self): + """ Create a new pipeline to test """ + self.pipeline_dir = os.path.join(tempfile.mkdtemp(), "test_pipeline") + self.create_obj = nf_core.create.PipelineCreate("testing", "test pipeline", "tester", outdir=self.pipeline_dir) + self.create_obj.init_pipeline() + + def test_inspect_sync_dir_notgit(self): + """ Try syncing an empty directory """ + psync = nf_core.sync.PipelineSync(tempfile.mkdtemp()) + try: + psync.inspect_sync_dir() + except nf_core.sync.SyncException as e: + assert "does not appear to be a git repository" in e.args[0] + + def test_inspect_sync_dir_dirty(self): + """ Try syncing a pipeline with uncommitted changes """ + # Add an empty file, uncommitted + test_fn = os.path.join(self.pipeline_dir, "uncommitted") + open(test_fn, "a").close() + # Try to sync, check we halt with the right error + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + try: + psync.inspect_sync_dir() + except nf_core.sync.SyncException as e: + os.remove(test_fn) + assert e.args[0].startswith("Uncommitted changes found in pipeline directory!") + except Exception as e: + os.remove(test_fn) + raise e + + def test_get_wf_config_no_branch(self): + """ Try getting a workflow config when the branch doesn't exist """ + # Try to sync, check we halt with the right error + psync = nf_core.sync.PipelineSync(self.pipeline_dir, from_branch="foo") + try: + psync.inspect_sync_dir() + psync.get_wf_config() + except nf_core.sync.SyncException as e: + assert e.args[0] == "Branch `foo` not found!" + + def test_get_wf_config_fetch_origin(self): + """ + Try getting the GitHub username and repo from the git origin + + Also checks the fetched config variables, should pass + """ + # Try to sync, check we halt with the right error + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + # Add a remote to the git repo + psync.repo.create_remote("origin", "https://github.com/nf-core/demo.git") + psync.get_wf_config() + assert psync.gh_username == "nf-core" + assert psync.gh_repo == "demo" + + def test_get_wf_config_missing_required_config(self): + """ Try getting a workflow config, then make it miss a required config option """ + # Try to sync, check we halt with the right error + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.required_config_vars = ["fakethisdoesnotexist"] + try: + psync.inspect_sync_dir() + psync.get_wf_config() + except nf_core.sync.SyncException as e: + # Check that we did actually get some config back + assert psync.wf_config["params.outdir"] == "'./results'" + # Check that we raised because of the missing fake config var + assert e.args[0] == "Workflow config variable `fakethisdoesnotexist` not found!" + + def test_checkout_template_branch(self): + """ Try checking out the TEMPLATE branch of the pipeline """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + + def test_delete_template_branch_files(self): + """ Confirm that we can delete all files in the TEMPLATE branch """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + psync.delete_template_branch_files() + assert os.listdir(self.pipeline_dir) == [".git"] + + def test_create_template_pipeline(self): + """ Confirm that we can delete all files in the TEMPLATE branch """ + # First, delete all the files + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + psync.delete_template_branch_files() + assert os.listdir(self.pipeline_dir) == [".git"] + # Now create the new template + psync.make_template_pipeline() + assert "main.nf" in os.listdir(self.pipeline_dir) + assert "nextflow.config" in os.listdir(self.pipeline_dir) + + def test_commit_template_changes_nochanges(self): + """ Try to commit the TEMPLATE branch, but no changes were made """ + # Check out the TEMPLATE branch but skip making the new template etc. + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + # Function returns False if no changes were made + assert psync.commit_template_changes() is False + + def test_commit_template_changes_changes(self): + """ Try to commit the TEMPLATE branch, but no changes were made """ + # Check out the TEMPLATE branch but skip making the new template etc. + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + # Add an empty file, uncommitted + test_fn = os.path.join(self.pipeline_dir, "uncommitted") + open(test_fn, "a").close() + # Check that we have uncommitted changes + assert psync.repo.is_dirty(untracked_files=True) is True + # Function returns True if no changes were made + assert psync.commit_template_changes() is True + # Check that we don't have any uncommitted changes + assert psync.repo.is_dirty(untracked_files=True) is False + + def raise_git_exception(self): + """ Raise an exception from GitPython""" + raise git.exc.GitCommandError("Test") + + def test_push_template_branch_error(self): + """ Try pushing the changes, but without a remote (should fail) """ + # Check out the TEMPLATE branch but skip making the new template etc. + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + # Add an empty file and commit it + test_fn = os.path.join(self.pipeline_dir, "uncommitted") + open(test_fn, "a").close() + psync.commit_template_changes() + # Try to push changes + try: + psync.push_template_branch() + except nf_core.sync.PullRequestException as e: + assert e.args[0].startswith("Could not push TEMPLATE branch") + + def test_make_pull_request_missing_username(self): + """ Try making a PR without a repo or username """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.gh_username = None + psync.gh_repo = None + try: + psync.make_pull_request() + except nf_core.sync.PullRequestException as e: + assert e.args[0] == "Could not find GitHub username and repo name" + + def test_make_pull_request_missing_auth(self): + """ Try making a PR without any auth """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.gh_username = "foo" + psync.gh_repo = "bar" + psync.gh_auth_token = None + try: + psync.make_pull_request() + except nf_core.sync.PullRequestException as e: + assert e.args[0] == "No GitHub authentication token set - cannot make PR" + + def mocked_requests_post(**kwargs): + """ Helper function to emulate POST requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = status_code + self.content = json.dumps(data) + + if kwargs["url"] == "https://api.github.com/repos/bad/response/pulls": + return MockResponse({}, 404) + + if kwargs["url"] == "https://api.github.com/repos/good/response/pulls": + response_data = {"html_url": "great_success"} + return MockResponse(response_data, 201) + + @mock.patch("requests.post", side_effect=mocked_requests_post) + def test_make_pull_request_bad_response(self, mock_post): + """ Try making a PR without any auth """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.gh_username = "bad" + psync.gh_repo = "response" + psync.gh_auth_token = "test" + try: + psync.make_pull_request() + except nf_core.sync.PullRequestException as e: + assert e.args[0].startswith("GitHub API returned code 404:") + + @mock.patch("requests.post", side_effect=mocked_requests_post) + def test_make_pull_request_bad_response(self, mock_post): + """ Try making a PR without any auth """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.gh_username = "good" + psync.gh_repo = "response" + psync.gh_auth_token = "test" + psync.make_pull_request() + assert psync.gh_pr_returned_data["html_url"] == "great_success" diff --git a/tests/test_utils.py b/tests/test_utils.py new file mode 100644 index 0000000000..9c73919ad3 --- /dev/null +++ b/tests/test_utils.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python +""" Tests covering for utility functions. +""" + +import nf_core.utils + +import unittest + + +class TestUtils(unittest.TestCase): + """Class for utils tests""" + + def test_check_if_outdated_1(self): + current_version = "1.0" + remote_version = "2.0" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert is_outdated + + def test_check_if_outdated_2(self): + current_version = "2.0" + remote_version = "2.0" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert not is_outdated + + def test_check_if_outdated_3(self): + current_version = "2.0.1" + remote_version = "2.0.2" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert is_outdated + + def test_check_if_outdated_4(self): + current_version = "1.10.dev0" + remote_version = "1.7" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert not is_outdated + + def test_check_if_outdated_5(self): + current_version = "1.10.dev0" + remote_version = "1.11" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert is_outdated diff --git a/tests/workflow/example.json b/tests/workflow/example.json deleted file mode 100644 index 003b4cfc6e..0000000000 --- a/tests/workflow/example.json +++ /dev/null @@ -1,37 +0,0 @@ -{ - "parameters": [ - { - "name": "reads", - "label": "WGS single-end fastq file.", - "usage": "Needs to be provided as workflow input data.", - "type": "string", - "render": "file", - "default_value": "path/to/reads.fastq.gz", - "pattern": ".*(\\.fastq$|\\.fastq\\.gz$)", - "group": "inputdata", - "required": false - }, - { - "name": "index", - "label": "Mapper index", - "usage": "Needs to be provided for the mapping.", - "type": "string", - "render": "file", - "default_value": "path/to/index", - "pattern": ".*", - "group": "inputdata", - "required": false - }, - { - "name": "norm_factor", - "label": "Normalization factor ", - "usage": "Integer value that will be applied against input reads.", - "type": "integer", - "render": "range", - "choices": ["1", "150"], - "default_value": "1", - "group": "normalization", - "required": false - } - ] -} \ No newline at end of file diff --git a/tests/workflow/test_parameters.py b/tests/workflow/test_parameters.py deleted file mode 100644 index ef6812a9de..0000000000 --- a/tests/workflow/test_parameters.py +++ /dev/null @@ -1,74 +0,0 @@ -#!/usr/bin/env python -"""Some tests covering the parameters code. -""" -import json -import jsonschema -from jsonschema import ValidationError -import os -import pytest -import requests -import shutil -import unittest -from nf_core.workflow import parameters as pms - -WD = os.path.dirname(__file__) -PATH_WORKING_EXAMPLE = os.path.join(WD, 'example.json') -SCHEMA_URI = "https://nf-co.re/parameter-schema/0.1.0/parameters.schema.json" - -@pytest.fixture(scope="class") -def schema(): - res = requests.get(SCHEMA_URI) - assert res.status_code == 200 - return res.text - -@pytest.fixture(scope="class") -def example_json(): - assert os.path.isfile(PATH_WORKING_EXAMPLE) - with open(PATH_WORKING_EXAMPLE) as fp: - content = fp.read() - return content - -def test_creating_params_from_json(example_json): - """Tests parsing of a parameter json.""" - result = pms.Parameters.create_from_json(example_json) - assert len(result) == 3 - -def test_groups_from_json(example_json): - """Tests group property of a parameter json.""" - result = pms.Parameters.create_from_json(example_json) - group_labels = set([ param.group for param in result ]) - assert len(group_labels) == 2 - -def test_params_as_json_dump(example_json): - """Tests the JSON dump that can be consumed by Nextflow.""" - result = pms.Parameters.create_from_json(example_json) - parameter = result[0] - assert parameter.name == "reads" - expected_output = """ - { - "reads": "path/to/reads.fastq.gz" - }""" - parsed_output = json.loads(expected_output) - assert len(parsed_output.keys()) == 1 - assert parameter.name in parsed_output.keys() - assert parameter.default_value == parsed_output[parameter.name] - -def test_parameter_builder(): - """Tests the parameter builder.""" - parameter = pms.Parameter.builder().name("width").default(2).build() - assert parameter.name == "width" - assert parameter.default_value == 2 - -@pytest.mark.xfail(raises=ValidationError) -def test_validation(schema): - """Tests the parameter objects against the JSON schema.""" - parameter = pms.Parameter.builder().name("width").param_type("unknown").default(2).build() - params_in_json = pms.Parameters.in_full_json([parameter]) - jsonschema.validate(json.loads(pms.Parameters.in_full_json([parameter])), json.loads(schema)) - -def test_validation_with_success(schema): - """Tests the parameter objects against the JSON schema.""" - parameter = pms.Parameter.builder().name("width").param_type("integer") \ - .default("2").label("The width of a table.").render("range").required(False).build() - params_in_json = pms.Parameters.in_full_json([parameter]) - jsonschema.validate(json.loads(pms.Parameters.in_full_json([parameter])), json.loads(schema)) diff --git a/tests/workflow/test_validator.py b/tests/workflow/test_validator.py deleted file mode 100644 index 07192125f8..0000000000 --- a/tests/workflow/test_validator.py +++ /dev/null @@ -1,104 +0,0 @@ -#!/usr/bin/env python -"""Some tests covering the parameters code. -""" -import os -import pytest -import requests -import shutil -import unittest -from nf_core.workflow import parameters as pms -from nf_core.workflow import validation as valid - - -WD = os.path.dirname(__file__) -PATH_WORKING_EXAMPLE = os.path.join(WD, 'example.json') -SCHEMA_URI = "https://nf-co.re/parameter-schema/0.1.0/parameters.schema.json" - -@pytest.fixture(scope="class") -def valid_integer_param(): - param = pms.Parameter.builder().name("Fake Integer Param") \ - .default(0).value(10).choices([0, 10]).param_type("integer").build() - return param - -@pytest.fixture(scope="class") -def invalid_integer_param(): - param = pms.Parameter.builder().name("Fake Integer Param") \ - .default(0).value(20).choices([0, 10]).param_type("integer").build() - return param - -@pytest.fixture(scope="class") -def invalid_string_param_without_pattern_and_choices(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("Whatever").choices(["0", "10"]).param_type("integer").build() - return param - -@pytest.fixture(scope="class") -def param_with_unknown_type(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("Whatever").choices(["0", "10"]).param_type("unknown").build() - return param - -@pytest.fixture(scope="class") -def string_param_not_matching_pattern(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("id.123A") \ - .param_type("string").pattern(r"^id\.[0-9]*$").build() - return param - -@pytest.fixture(scope="class") -def string_param_matching_pattern(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("id.123") \ - .param_type("string").pattern(r"^id\.[0-9]*$").build() - return param - -@pytest.fixture(scope="class") -def string_param_not_matching_choices(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("snail").choices(["horse", "pig"])\ - .param_type("string").build() - return param - -@pytest.fixture(scope="class") -def string_param_matching_choices(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("horse").choices(["horse", "pig"])\ - .param_type("string").build() - return param - -def test_simple_integer_validation(valid_integer_param): - validator = valid.Validators.get_validator_for_param(valid_integer_param) - validator.validate() - -@pytest.mark.xfail(raises=AttributeError) -def test_simple_integer_out_of_range(invalid_integer_param): - validator = valid.Validators.get_validator_for_param(invalid_integer_param) - validator.validate() - -@pytest.mark.xfail(raises=AttributeError) -def test_string_with_empty_pattern_and_choices(invalid_string_param_without_pattern_and_choices): - validator = valid.Validators.get_validator_for_param(invalid_integer_param) - validator.validate() - -@pytest.mark.xfail(raises=LookupError) -def test_param_with_empty_type(param_with_unknown_type): - validator = valid.Validators.get_validator_for_param(param_with_unknown_type) - validator.validate() - -@pytest.mark.xfail(raises=AttributeError) -def test_string_param_not_matching_pattern(string_param_not_matching_pattern): - validator = valid.Validators.get_validator_for_param(string_param_not_matching_pattern) - validator.validate() - -def test_string_param_matching_pattern(string_param_matching_pattern): - validator = valid.Validators.get_validator_for_param(string_param_matching_pattern) - validator.validate() - -@pytest.mark.xfail(raises=AttributeError) -def test_string_param_not_matching_choices(string_param_not_matching_choices): - validator = valid.Validators.get_validator_for_param(string_param_not_matching_choices) - validator.validate() - -def test_string_param_matching_choices(string_param_matching_choices): - validator = valid.Validators.get_validator_for_param(string_param_matching_choices) - validator.validate() \ No newline at end of file